We can get data from web pages with Beautiful Soup.
It lets us parse the DOM and extract the data we want.
In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.
Going Up
We can move up the tree with Beautiful Soup.
For example, we can write:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
title_tag = soup.title
print(title_tag)
print(title_tag.parent)
Then the first print call prints:
<title>The Dormouse's story</title>
And the 2nd print call prints:
<head><title>The Dormouse's story</title></head>
So we see the head tag with the parent property.
.parents
We can iterate over all element’s parents with the .parents property.
For example, we can write:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
link = soup.a
print(link)
for parent in link.parents:
print(parent.name)
We get the first a element with soup.a .
So the first print call is:
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
The 2nd print calls prints all the parents of the a element, so we get:
p
body
html
[document]
Going Sideways
We can get siblings elements with Beautiful Soup.
For example, we can write:
from bs4 import BeautifulSoup
sibling_soup = BeautifulSoup(
"<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.prettify())
Then we get:
<a>
<b>
text1
</b>
<c>
text2
</c>
</a>
printed.
.next_sibling and .previous_sibling
We can get the next sibling with the .next_sibling property and the previous sibling with the .previous_sibling property.
For example, we can write:
from bs4 import BeautifulSoup
sibling_soup = BeautifulSoup(
"<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.b.next_sibling)
print(sibling_soup.c.previous_sibling)
We see:
<c>text2</c>
<b>text1</b>
printed from the print calls.
The strings in the tags aren’t siblings since they don’t have the same parent.
.next_siblings and .previous_siblings
We can get multiple siblings with the .next_siblings and .previous_siblings properties.
For example, if we have:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.a.next_siblings:
print(repr(sibling))
Then we see all the siblings next to the first a element:
u',n'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u' andn'
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
u';nand they lived at the bottom of a well.'
printed.
We can do the same with the previous_siblings property:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.find(id="link3").previous_siblings:
print(repr(sibling))
And we see:
u' andn'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u',n'
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
u'Once upon a time there were three little sisters; and their names weren'
printed.
Conclusion
We can get parent and sibling nodes with Beautiful Soup.