Introduction to BeautifulSoup 4 in Python 2

This article will introduce the use of the BeautifulSoup 4 library in Python 2, so a basic knowledge of this language is assumed. Version 4 of the library also supports Python 3, but it may produce various errors, so it is preferable to use version 2 of this language. Knowledge of HTML document structure and the most common tags is also assumed.

What is BeautifulSoup?

BeautifulSoup is a Python library for extracting content from HTML and XML files. It is very useful for obtaining information from web pages in a processable form (in an easy-to-handle tree structure).

Installation

Debian and derivatives

If you do not have Python installed on your system yet, you can install it through apt:

# apt-get install python2.7 python-pip

this command will install the latest currently available version of Python 2 and PyPi, a python module manager that is very useful for installing various libraries for this language.

Finally, to install BeautifulSoup:

# apt-get install python-bs4

PyPi

On systems other than GNU/Linux, the most convenient option is to install BeautifulSoup through this manager. Before doing so, we must have installed Python 2, which can be downloaded from the python website. Make sure you select the option to install _pip_ in the installer. Then, open a command console and run

# pip install beautifulsoup4

Installing the HTML parser

To extract the content of HTML or XML documents, you need to install a parser. BeautifulSoup supports the parser built into Python, but it has some drawbacks. In this article we will work with _lxml_, as it is the fastest, although naturally it has other drawbacks too. To find the parser that best suits your needs, consult the table in the official documentation.

To install _lxml_, run

# apt-get install python-lxml

on Debian or

# pip install lxml

Basic usage

Create a new Python (.py) file in which the first step is to import the BeautifulSoup module:

from bs4 import BeautifulSoup

 along with the parser and _urllib2_ to read web pages:

import lxml, urllib2

Before you start programming, it is good to take a quick look at the structure of the web page in question, so it is useful to open any web browser and use “Inspeccionar elemento” to see each element's _id_, its tags, and any other information that will help speed up the work later.

 In the .py document started earlier, add the following two lines:

url = urllib2.urlopen("https://es.wikipedia.org/wiki/Debian_GNU/Linux") # read the web page with urllib2
soup = BeautifulSoup(url,"lxml") # create the BeautifulSoup object

The BeautifulSoup object, stored in the soup variable, contains the information for the entire HTML document. In this case we will use the Wikipedia page about Debian

Accessing through tags

Tags in BeautifulSoup correspond to the tags in HTML documents, so to access the title we can write:

soup.title # <title>Debian GNU/Linux - Wikipedia, la enciclopedia libre</title>
soup.title.name # title
soup.title.string # Debian GNU/Linux - Wikipedia, la enciclopedia libre

Here you can see the difference between the _name_ and _string_ attributes. As shown in the example, if no attribute is specified, the HTML text is obtained exactly as it is.

If there are several identical tags, such as _a_ tags, typing `soup.a` only accesses the first tag of that type. To access one other than the first, you can create a list with

alst = soup.find_all('a')

and from it you can access the indices of each element with

alist[i]

where y is the index of the _a_ element you want to access, starting from 0.

To access the content of a particular attribute, such as the _id_ of a heading, use the following format:

soup.ETIQUETA['ATRIBUTO']

an example of this last part, using

find_all

and accessing attributes, is:

alist = soup.find_all('a') # [<a id="top"></a>, ..., <a href="//www.mediawiki.org/"><img alt="Powered by MediaWiki" height="31" src="/static/images/poweredby_mediawiki_88x31.png" srcset="/static/images/poweredby_mediawiki_132x47.png 1.5x, /static/images/poweredby_mediawiki_176x62.png 2x" width="88"/></a>]
print alist[3] # element 4 of the list (starts from 0): <a href="/wiki/GNU" title="GNU">GNU</a>
print alist[3]['href'] # href attribute of element 4: /wiki/GNU
print alist[3]['title'] # title attribute of element 4: GNU

Accessing through children

The previous section showed how to access the document through its tags. Now we will traverse that document hierarchically in a tree structure.

To obtain the levels below a tag, for example the _head_ tag, we write the following code:

soup.head.contents

which will return a list with the different tags, each one an element of the list. To access each element of that list, we do it in the usual way in Python, with square brackets:

soup.head.contents[i] # access element i-1 of the list

Since this is a tree structure, you can access the element above a given one. For example, if we are at the _title_ tag, we can “climb” up to the _head_ tag:

soup.title.parent

which shows us all the information contained in _head_. It is also possible to chain _parent_,

soup.title.parent.parent

which in this case would return the complete HTML.

Documentation

All the documentation for this library is available at https://www.crummy.com/software/BeautifulSoup/bs4/doc/, which is very comprehensive and provides a wide variety of examples.

CC-BY Creative Commons.

David Población Criado