Course Content

Web Scraping with Python (res)

1. HTML Files and DevTools

Intro to Web Scraping First Steps HTML File Structure Hierarchy in the HTML String Methods Attributes DevTools

2. Beautiful Soup

Work with the Library Work with Soup Find/Find_all Find All!

3. CSS Selectors/XPaths

Intro to XPath Attributes in XPath Selectors CSS Selectors CSS Selectors in BeautifulSoup XPath vs CSS Selector

4. Tables

Intro to Tables Ways to Scrape Table Simple Solution for Scraping Challenge

First Steps

The web page is comprised of HTML.

HTML is the markup language for creating web pages.

Let’s get the HTML file of the web page!

We can work with the data of sites by their URLs. To open website URLs in your Python programs, use the function urlopen() from the module urllib.request and define the URL you want to open as a string variable:


              1234
            
from urllib.request import urlopen

url = "https://codefinity-content-media.s3.eu-west-1.amazonaws.com/18a4e428-1a0f-44c2-a8ad-244cd9c7985e/page.html"
page = urlopen(url)

It looks simple, but when we want to print the variable page to see what is going on in the HTML file, we get:

It returns HTTPResponse object. To parse it, use the .read() method, which returns a sequence of bytes, and then the function decode("utf-8") to decode the data from bytes to string


              1234
            
bytes = page.read()
html = bytes.decode("utf-8")

print(html)

We can also use methods consequentially: page.read().decode("utf-8").

Here we open the URL we will work within this course!

Task

Write the missing code to get the HTML structure from the page which interests you.

Import module urlopen to open URLs from your code.
Open the URL. Assign the result to the variable page.
Get a sequence of bytes using the method .read(). Assign the result to the variable bytes.
Decode bytes to string using the method .decode(). Assign the result to the variable html.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 2

First Steps

The web page is comprised of HTML.

HTML is the markup language for creating web pages.

Let’s get the HTML file of the web page!


              1234
            
from urllib.request import urlopen

url = "https://codefinity-content-media.s3.eu-west-1.amazonaws.com/18a4e428-1a0f-44c2-a8ad-244cd9c7985e/page.html"
page = urlopen(url)

It looks simple, but when we want to print the variable page to see what is going on in the HTML file, we get:

It returns HTTPResponse object. To parse it, use the .read() method, which returns a sequence of bytes, and then the function decode("utf-8") to decode the data from bytes to string


              1234
            
bytes = page.read()
html = bytes.decode("utf-8")

print(html)

We can also use methods consequentially: page.read().decode("utf-8").

Here we open the URL we will work within this course!

Task

Write the missing code to get the HTML structure from the page which interests you.

Import module urlopen to open URLs from your code.
Open the URL. Assign the result to the variable page.
Get a sequence of bytes using the method .read(). Assign the result to the variable bytes.
Decode bytes to string using the method .decode(). Assign the result to the variable html.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 2

First Steps

The web page is comprised of HTML.

HTML is the markup language for creating web pages.

Let’s get the HTML file of the web page!


              1234
            
from urllib.request import urlopen

url = "https://codefinity-content-media.s3.eu-west-1.amazonaws.com/18a4e428-1a0f-44c2-a8ad-244cd9c7985e/page.html"
page = urlopen(url)

It looks simple, but when we want to print the variable page to see what is going on in the HTML file, we get:

It returns HTTPResponse object. To parse it, use the .read() method, which returns a sequence of bytes, and then the function decode("utf-8") to decode the data from bytes to string


              1234
            
bytes = page.read()
html = bytes.decode("utf-8")

print(html)

We can also use methods consequentially: page.read().decode("utf-8").

Here we open the URL we will work within this course!

Task

Write the missing code to get the HTML structure from the page which interests you.

Import module urlopen to open URLs from your code.
Open the URL. Assign the result to the variable page.
Get a sequence of bytes using the method .read(). Assign the result to the variable bytes.
Decode bytes to string using the method .decode(). Assign the result to the variable html.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

The web page is comprised of HTML.

HTML is the markup language for creating web pages.

Let’s get the HTML file of the web page!


              1234
            
from urllib.request import urlopen

url = "https://codefinity-content-media.s3.eu-west-1.amazonaws.com/18a4e428-1a0f-44c2-a8ad-244cd9c7985e/page.html"
page = urlopen(url)

It looks simple, but when we want to print the variable page to see what is going on in the HTML file, we get:

It returns HTTPResponse object. To parse it, use the .read() method, which returns a sequence of bytes, and then the function decode("utf-8") to decode the data from bytes to string


              1234
            
bytes = page.read()
html = bytes.decode("utf-8")

print(html)

We can also use methods consequentially: page.read().decode("utf-8").

Here we open the URL we will work within this course!

Task

Write the missing code to get the HTML structure from the page which interests you.

Import module urlopen to open URLs from your code.
Open the URL. Assign the result to the variable page.
Get a sequence of bytes using the method .read(). Assign the result to the variable bytes.
Decode bytes to string using the method .decode(). Assign the result to the variable html.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Section 1. Chapter 2

Switch to desktop for real-world practiceContinue from where you are using one of the options below