Conteúdo do Curso

Web Scraping with Python (res)

1. HTML Files and DevTools

Intro to Web Scraping First Steps HTML File Structure Hierarchy in the HTML String Methods Attributes DevTools

2. Beautiful Soup

Work with the Library Work with Soup Find/Find_all Find All!

3. CSS Selectors/XPaths

Intro to XPath Attributes in XPath Selectors CSS Selectors CSS Selectors in BeautifulSoup XPath vs CSS Selector

4. Tables

Intro to Tables Ways to Scrape Table Simple Solution for Scraping Challenge

String Methods

We already know the structure of the HTML document and how to get the HTML of the page by its URL. Let’s explore how to find some information in the code by tags. One of the ways to do it - use string methods. For example, function .find(). This method returns the index of the first appearance of the string you want to find.

Let’s find the index of the first occurrence of the tag <head> on the site from the second chapter:


              12
            
index_first = html.find("<head>")
print(index_first)

Hmm, what does it mean? That we have no head in our HTML code? But it can’t be true. We can easily find it in the first lines of the HTML file. The fact is that this page has extra space before the close-angle bracket. It can be any spaces between the end of the tag name and the closing bracket. So here, we can find the index using the following code:


              12
            
index_first = html.find("<head >")
print(index_first)

We can also find first index of closing tag (here we have no extra spaces):

Finally, you can extract the title block by slicing the html string:


              1234567891011
            
# Find the first index of the tag
index_first = html.find("<title>")
print(index_first)

# Find the last index of the tag
index_last = html.find("</title>") + len("</title>")
print(index_last)

# Extract the title
title = html[index_first:index_last]
print(title)

In the code, we added to the index the length of the close tag </title> to see it in the output.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Before using web scraping, you should always check your website’s terms of use to know if accessing the website with tools is a violation of the terms of use or not.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

Seção 1. Capítulo 5

String Methods

Let’s find the index of the first occurrence of the tag <head> on the site from the second chapter:


              12
            
index_first = html.find("<head>")
print(index_first)


              12
            
index_first = html.find("<head >")
print(index_first)

We can also find first index of closing tag (here we have no extra spaces):

Finally, you can extract the title block by slicing the html string:


              1234567891011
            
# Find the first index of the tag
index_first = html.find("<title>")
print(index_first)

# Find the last index of the tag
index_last = html.find("</title>") + len("</title>")
print(index_last)

# Extract the title
title = html[index_first:index_last]
print(title)

In the code, we added to the index the length of the close tag </title> to see it in the output.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Before using web scraping, you should always check your website’s terms of use to know if accessing the website with tools is a violation of the terms of use or not.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

Seção 1. Capítulo 5

String Methods

Let’s find the index of the first occurrence of the tag <head> on the site from the second chapter:


              12
            
index_first = html.find("<head>")
print(index_first)


              12
            
index_first = html.find("<head >")
print(index_first)

We can also find first index of closing tag (here we have no extra spaces):

Finally, you can extract the title block by slicing the html string:


              1234567891011
            
# Find the first index of the tag
index_first = html.find("<title>")
print(index_first)

# Find the last index of the tag
index_last = html.find("</title>") + len("</title>")
print(index_last)

# Extract the title
title = html[index_first:index_last]
print(title)

In the code, we added to the index the length of the close tag </title> to see it in the output.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Before using web scraping, you should always check your website’s terms of use to know if accessing the website with tools is a violation of the terms of use or not.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

Let’s find the index of the first occurrence of the tag <head> on the site from the second chapter:


              12
            
index_first = html.find("<head>")
print(index_first)


              12
            
index_first = html.find("<head >")
print(index_first)

We can also find first index of closing tag (here we have no extra spaces):

Finally, you can extract the title block by slicing the html string:


              1234567891011
            
# Find the first index of the tag
index_first = html.find("<title>")
print(index_first)

# Find the last index of the tag
index_last = html.find("</title>") + len("</title>")
print(index_last)

# Extract the title
title = html[index_first:index_last]
print(title)

In the code, we added to the index the length of the close tag </title> to see it in the output.

Tarefa

Let's find the title of the page!

Find the index of the first occurrence of the tag <title> and assign it to the variable index_first. Print the variable index_first.
Find the index of the first occurrence of the tag </title>. Add the length of the world "</title>" and assign the result to the variable index_last. Print the variable index_last.
Extract the title block by slicing the variable html and assign the result to the variable title. Print the variable title.

Before using web scraping, you should always check your website’s terms of use to know if accessing the website with tools is a violation of the terms of use or not.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Seção 1. Capítulo 5

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo