Webscraping - Reading trends from macrotrends with plotting¶
In the below steps, we will read crude oil historical data, and plot the trend¶
Preparing the enviroment:¶
In [1]:
import pandas as pd
import requests
from bs4 import BeautifulSoup
Now, we will form the url and then parse the text to html using BeautifulSoap¶
In [2]:
url="https://www.macrotrends.net/1369/crude-oil-price-history-chart"
crude_data = requests.get(url).text
soup = BeautifulSoup(crude_data, 'html5lib')
Now we will read the table in the html and place it into pandas dataframe:¶
In [3]:
crude_data = pd.DataFrame(columns=["Year", "Average Closing Price", "Year Open", "Year High", "Year Low", "Year Close","Annual % Change"])
for row in soup.find("tbody").find_all('tr'):
col = row.find_all("td")
year = col[0].text
acp = col[1].text
yo = col[2].text
yh = col[3].text
yl = col[4].text
yc = col[5].text
apc = col[6].text
crude_data= crude_data.append({"Year":year, "Average Closing Price":acp, "Year Open":yo, "Year High":yh, "Year Low":yl, "Year Close":yc, "Annual % Change":apc}, ignore_index=True)
Let's print the dataframe:¶
In [4]:
#we will print the first 5 rows (the past 5 years)
crude_data.head()
Out[4]:
Let's now simply plot the data¶
In [5]:
crude_data.plot(x ='Year', y='Average Closing Price', kind = 'scatter')
Out[5]:
