Course Content

Web Scraping with Python (res)

1. HTML Files and DevTools

Intro to Web Scraping First Steps HTML File Structure Hierarchy in the HTML String Methods Attributes DevTools

2. Beautiful Soup

Work with the Library Work with Soup Find/Find_all Find All!

3. CSS Selectors/XPaths

Intro to XPath Attributes in XPath Selectors CSS Selectors CSS Selectors in BeautifulSoup XPath vs CSS Selector

4. Tables

Intro to Tables Ways to Scrape Table Simple Solution for Scraping Challenge

Work with Soup

Continue exploring BeautifulSoup let’s learn some important functions! We can extract not only tag but also their parts (for example, names or attributes):


              12
            
print(soup.div.name)
print(soup.div.attrs)

In the code, we used the method .name to get the tag’s name and the function .attrs, which returns all tag attributes as a dictionary.

Another useful function is .get_text(), which extracts all the raw text from the website without HTML tags.

The output of the page will contain a lot of extra blank lines. It happened because of newline characters in the initial HTML file.

In a similar way you can also get only text in the extracted HTML tags using the function .get_text() or .string:


              12
            
print(soup.h1.string)
print(soup.h1.get_text())

If a tag contains more than one thing (or nothing), it is unclear what .string should refer to, so the function returns None.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Section 2. Chapter 2

Work with Soup

Continue exploring BeautifulSoup let’s learn some important functions! We can extract not only tag but also their parts (for example, names or attributes):


              12
            
print(soup.div.name)
print(soup.div.attrs)

In the code, we used the method .name to get the tag’s name and the function .attrs, which returns all tag attributes as a dictionary.

Another useful function is .get_text(), which extracts all the raw text from the website without HTML tags.

The output of the page will contain a lot of extra blank lines. It happened because of newline characters in the initial HTML file.

In a similar way you can also get only text in the extracted HTML tags using the function .get_text() or .string:


              12
            
print(soup.h1.string)
print(soup.h1.get_text())

If a tag contains more than one thing (or nothing), it is unclear what .string should refer to, so the function returns None.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Section 2. Chapter 2

Work with Soup

Continue exploring BeautifulSoup let’s learn some important functions! We can extract not only tag but also their parts (for example, names or attributes):


              12
            
print(soup.div.name)
print(soup.div.attrs)

In the code, we used the method .name to get the tag’s name and the function .attrs, which returns all tag attributes as a dictionary.

Another useful function is .get_text(), which extracts all the raw text from the website without HTML tags.

The output of the page will contain a lot of extra blank lines. It happened because of newline characters in the initial HTML file.

In a similar way you can also get only text in the extracted HTML tags using the function .get_text() or .string:


              12
            
print(soup.h1.string)
print(soup.h1.get_text())

If a tag contains more than one thing (or nothing), it is unclear what .string should refer to, so the function returns None.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Continue exploring BeautifulSoup let’s learn some important functions! We can extract not only tag but also their parts (for example, names or attributes):


              12
            
print(soup.div.name)
print(soup.div.attrs)

In the code, we used the method .name to get the tag’s name and the function .attrs, which returns all tag attributes as a dictionary.

Another useful function is .get_text(), which extracts all the raw text from the website without HTML tags.

The output of the page will contain a lot of extra blank lines. It happened because of newline characters in the initial HTML file.

In a similar way you can also get only text in the extracted HTML tags using the function .get_text() or .string:


              12
            
print(soup.h1.string)
print(soup.h1.get_text())

If a tag contains more than one thing (or nothing), it is unclear what .string should refer to, so the function returns None.

Task

Here you will work on the same page about Christ the Redeemer as in the previous task.

Import the BeautifulSoup library.
Print the attributes of the p tag.
Print only the text of the ul tags.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Section 2. Chapter 2

Switch to desktop for real-world practiceContinue from where you are using one of the options below