We can get data from web pages with Beautiful Soup.
It lets us parse the DOM and extract the data we want.
In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.
.string
We can get the text content from the element if there’s only one child and the child is a NavigableString .
For example, we can write:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
title_tag = soup.head.title
print(title_tag.string)
to get the title from the head tag.
Then we get:
The Dormouse's story
printed.
If the only child has another tag and that tag has a string, then the parent yag is considered to have the same string as the child.
So if we write:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
head_tag = soup.head
print(head_tag.string)
We also get:
The Dormouse's story
printed.
And if we replace tyhe last line with:
print(head_tag.contents)
We see:
[<title>The Dormouse's story</title>]
printed.
If there’s more than one thing inside a tag, we can use the .strings generator to look at all the contents:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for string in soup.strings:
print(repr(string))
Then we see:
u"The Dormouse's story"
u'n'
u'n'
u"The Dormouse's story"
u'n'
u'Once upon a time there were three little sisters; and their names weren'
u'Elsie'
u',n'
u'Lacie'
u' andn'
u'Tillie'
u';nand they lived at the bottom of a well.'
u'n'
u'...'
u'n'
printed.
To get the strings without the extra whitespace, we can use the stripped_strings generator.
For example, we can write:
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for string in soup.stripped_strings:
print(repr(string))
And we get:
u"The Dormouse's story"
u"The Dormouse's story"
u'Once upon a time there were three little sisters; and their names were'
u'Elsie'
u','
u'Lacie'
u'and'
u'Tillie'
u';nand they lived at the bottom of a well.'
u'...'
printed.
Conclusion
We can get strings at various locations with Beautiful Soup.