The amount of information available on the internet for human consumption is astounding. However, if this data doesn’t come in the form of a specialized REST API, it can be challenging to access programmatically. The technique of gathering and processing raw data from the internet is known as web scraping. There are several uses for web scraping in software development. Data collected through web scraping can be applied in market research, lead generation, competitive intelligence, product pricing comparison, monitoring consumer sentiment, brand audits, AI and machine learning, creating a job board, and more.
Utilizing an HTML parser that is specifically made for parsing out HTML pages is simpler than parsing with custom written programming logic. There are several tools available for web scraping using HTML parsers. Beautiful Soup is one such library that works with the HTML parser of your choice for web scraping. Some prefer to parse HTML pages with regex, as it is lightweight and comes out of the box with many programming languages—you don’t have to install any separate dependencies. Also, if you deal with simple, well-formatted HTML pages, implementing a regex-based parsing solution is pretty straightforward.
A regular expression (often abbreviated as regex) is a string of letters that designates a text search pattern. Regular expressions are used in lexical analysis, word processors’ search-and-replace dialogues, text editors’ search-and-replace functions, and text processing tools like sed and AWK.
In this article, you will learn how to parse HTML with regex in Python. In the tutorial, you will download the contents of a website, search for required data, and explore some specific use cases of parsing HTML content using regex. Then you’ll review some challenges involved in using regex for parsing arbitrary HTML and learn about alternative solutions.
Parsing With Regex
Regex support is provided by the majority of general-purpose programming languages, including Python, C, C++, Java, Rust, and JavaScript, either natively or through libraries.
The following snippet is a simple regex syntax in Python that can be used to find the string pattern “” in a given text:
import re
re.findall("![]()
", "![]()
: This is an image tag")
Note: All the source code in this tutorial was developed and executed in Python 3.8.10, but it should work on any machine with Python 3+.
In the example above, the module re is used for performing operations related to regular expressions. findall is a function in the re module that is used for returning a list of all matches of a given pattern (first argument value: : This is an image tag
One straightforward way of parsing HTML is using regular expressions to constantly look for and extract substrings that match a specific pattern. When your HTML is well-formatted and predictable, regular expressions function quite effectively. Imagine you have an HTML file, simple.html, with the following code in it:
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple HTML Filetitle>
head>
<body>
<h1>Tutorial on How to Parse HTML with Regexh1>
<p>This is a sample html file used as an examplep>
body>
html>
Say you’d like to extract the information contained within the
import re
matched = re.compile('' ).search
with open("simple.html", "r") as input_file:
for line in input_file:
if matched(line):
print(line.replace("" , "").replace("", ""))
Place this Python script file, simple-html-parser.py, in the same location as that of the simple.html file and run it to get the output, as shown here:
python .simple-html-parser.py
Simple HTML File
In the above Python code, a regular expression pattern provided as a string is converted into a regex pattern object using Python’s re.compile() function. You can utilize this pattern object to look for matches within various target strings. In this case, the target is the line read from the simple.html file. Once the pattern is matched, the
Although regex allows you to quickly build a parser for simple and well-formed HTML sites, a parser application that exclusively uses regular expressions may either overlook some valid links or produce inaccurate data when being used to parse arbitrary HTML pages.
Some websites may use nested tags—an HTML element placed inside another HTML element—in their HTML pages. At times, the nesting can go at multiple levels. If you want to parse for certain information from a particular level, it can be complex to do so and may result in errors.
While using regex for parsing HTML has some limitations, it allows you to build a quick solution and serves well for use cases like parsing simple HTML pages, filtering out a certain tag/groups of tags from an HTML page, or for extracting a particular text pattern within a tag.
Implementing HTML Parsing Using Regex
Now that you understand how regex-based HTML parsing works at a high level, read on to learn how to do it.
Download the Contents of a Website
To start, you’ll download site content onto your local machine. For this tutorial, you can use the PyPI site’s URL. Create a Python file named download-site-content.py and paste in the following code:
import urllib.request
URL = "https://pypi.org/"
HTML_FILE = "file.html"
def write_html_of_given_url_to_file(input_url: str):
urllib.request.urlretrieve(input_url, HTML_FILE)
write_html_of_given_url_to_file(URL)
This is a simple program to understand. The method write_html_of_given_url_to_file accepts the input URL of the PyPI site and uses the urlretrieve method of the urllib.request module to retrieve the site content and write it into the given file name, file.html. The output HTML file will be created in the same directory where you placed the Python script.
Search for Required Data Using Regex
Earlier, you saw an example of how to extract the information within the
As you can see in the screenshot, there are multiple stats published on the PyPI site—the total number of projects, releases, users, and more. Among these, you want to extract only the information about the total number of projects. If you view the source code of the html file you downloaded earlier, file.html, you’ll notice that this stat is included as part of a tag present inside a
<div class="horizontal-section horizontal-section--grey horizontal-section--thin horizontal-section--statistics">
<div class="statistics-bar">
<p class="statistics-bar__statistic">
410,645 projects
p>
<p class="statistics-bar__statistic">
3,892,920 releases
p>
<p class="statistics-bar__statistic">
6,961,696 files
p>
<p class="statistics-bar__statistic">
633,650 users
p>
div>
div>
You’ll also notice that there are multiple instances of the word “projects” in file.html. The challenge is to extract only the required stat—in this case, 410,645 projects. To do so, copy and paste the following code in a Python file named print-total-number-of-projects.py:
import re
import urllib.request
URL = "https://pypi.org/"
def get_html(input_url: str) -> bytes:
html = urllib.request.urlopen(input_url).read()
return html
def print_projects_count(input_url: str) -> None:
html = get_html(input_url)
stats = re.findall(b'\d+,?\d+,?\d+\s+projects', html)
print("n ***** Printing Projects Count ***** n")
for stat in stats:
print(stat.decode())
print_projects_count(URL)
Here, print_projects_count is called with the PyPI site URL as input. This method in turn uses the get_html method to read the site content and returns the html bytes to the caller method. print_projects_count then uses a regex pattern to get the list of all matches for the supplied pattern. The regex pattern, b'\d+,?\d+,?\d+\s+projects', is the key in finding the total number of projects:
bstands for bytes\d+stands for one or more numbers,stands for comma, as the stats published in the PyPI site use commas as a separator?stands for zero or one-time match of the previous token; in this case, the previous token is,\s+stands for one or more spacesprojectsis just the string
Since the return type of findall is a list, an iteration of the list is performed to print the project stats to the terminal output.
When you run the script, you should see an output like this:
python .print-total-number-of-projects.py
***** Printing Projects Count *****
410,645 projects
Now that you understand the basics of parsing HTML with regex, see if you can program to extract links (any https- or http-referenced URLs) from the same site. You have all the help you need from the earlier sections; now all you have to do is find the right regular expression for your goal of extracting the http(s) URLs.
You can consider this as an exercise to carry out, or refer to the source code below for the answer:
import re
import urllib.request
URL = "https://pypi.org/"
def get_html(input_url: str) -> bytes:
html = urllib.request.urlopen(input_url).read()
return html
def get_all_referenced_urls(input_url: str) -> list:
links_list = []
html = get_html(input_url)
referenced_urls = re.findall(b'href="(https?://.*?)"', html)
print("n ***** Printing all http(s) URLs ***** n")
for referenced_url in referenced_urls:
print(referenced_url.decode())
links_list.append(referenced_url.decode())
return links_list
get_all_referenced_urls(URL)
When you copy and paste the above code into a file named extract-link.py and run the script using the command python extract-links.py, you’ll get the list of referenced URL links from the PyPI site.
Filter Empty Tags
HTML supports empty tags—that is, tags that can’t have any nested tags or child nodes. These empty tags are also known as void elements. They usually have a start tag but don’t need an end tag as non-empty tags do. Even if you have specified an end tag in the HTML page for such empty tags, the browser won’t throw an error. You can refer to this article for more information and to see the full list of empty tags in HTML.
Say you want to filter out these empty tags from the PyPI site’s downloaded HTML file. How would you go about it? Copy and paste the code below into a file, filter-empty-tags-from-html.py:
import os
import re
import shutil
import urllib.request
URL = "https://pypi.org/"
HTML_FILE = "file.html"
ALTERED_HTML_FILE = "altered-file.html"
# Source of Empty tags list or Void elements list :
# https://developer.mozilla.org/en-US/docs/Glossary/Void_element
EMPTY_TAGS_LIST = ["area", "base", "br", "col", "embed", "hr", "img", "input", "keygen", "link", "meta", "param",
"source", "track", "wbr"]
def write_html_of_given_url_to_file(input_url: str):
urllib.request.urlretrieve(input_url, HTML_FILE)
def filter_empty_tags(input_file, output_file):
"""
Accepts two files as input. One will be considered as input and the other will be for writing the output which
will store html contents from input file except that of empty tags
:param input_file:
:param output_file:
:return:
"""
shutil.copyfile(input_file, output_file)
for empty_tag in EMPTY_TAGS_LIST:
remove_empty_tag_from_html(output_file, empty_tag)
def remove_empty_tag_from_html(input_file, empty_tag_element: str):
matched = re.compile('<' + empty_tag_element).search
with open(input_file, "r", encoding="utf-8") as file:
with open('temp.html', 'w', encoding="utf-8") as output_file:
for line in file:
if not matched(line): # save lines that do not match
print(line, end='', file=output_file) # this goes to filename due to inplace=1
os.replace('temp.html', input_file)
write_html_of_given_url_to_file(URL)
filter_empty_tags(HTML_FILE, ALTERED_HTML_FILE)
Run the following command:
python filter-empty-tags-from-html.py
This will create two files in the directory where you have placed and executed the above Python script. The original downloaded content of the PyPI site is contained in file.html and the modified HTML content (after filtering out the empty tags) is contained in altered-file.html.
Using a similar method, you can filter the comments from the HTML page. Comments in HTML are represented as follows:
You can try this for yourself based on what you’ve learned so far. Or, if you want the source code for this use case, copy and paste the Python script below in a file named filter-comments-from-html.py:
import os
import re
import urllib.request
URL = "https://pypi.org/"
HTML_FILE = "file.html"
ALTERED_HTML_FILE = "altered-file.html"
def write_html_of_given_url_to_file(input_url: str):
urllib.request.urlretrieve(input_url, HTML_FILE)
def remove_comments_from_html(input_file, output_file):
matched = re.compile('
