Data Visualization - Line Plotting¶
Preparing the enviroment¶
In [9]:
import numpy as np
import pandas as pd
import openpyxl # used to read excel files
Enter the url where your dataset is:¶
In [11]:
URL = 'https://github.com/mmouty/energynds/blob/main/energynds/assets/datasets/UN_MigFlow_All_CountryFiles.zip'
fetching the excel file:¶
In [15]:
df_can = pd.read_excel(
URL,
sheet_name='Canada by Citizenship',
skiprows=range(20),
skipfooter=2,
engine='openpyxl')
print('Data downloaded and read into a dataframe!')
In [16]:
df_can.head() #Just checking the data
Out[16]:
Let's get basic information about our dataframe:¶
In [17]:
df_can.info(verbose=False)
Let's get the list of column headers¶
In [18]:
df_can.columns
Out[18]:
Let's get the list of indicies:¶
In [19]:
df_can.index
Out[19]:
Let's check the type of the columns and index:¶
In [20]:
print(type(df_can.columns))
print(type(df_can.index))
The default type of intance variables index and columns are not list, so let's change them into list:¶
In [21]:
df_can.columns.tolist()
Out[21]:
In [22]:
df_can.index.tolist()
Out[22]:
Now, let's check again:¶
In [23]:
print(type(df_can.columns.tolist()))
print(type(df_can.index.tolist()))
Let's check the dimensions of our dataframe:¶
In [24]:
# size of dataframe (rows, columns)
df_can.shape
Out[24]:
We are not going to use all of these columns, so let's drop some:¶
In [25]:
# in pandas axis=0 represents rows (default) and axis=1 represents columns.
df_can.drop(['AREA','REG','DEV','Type','Coverage','Unnamed: 43','Unnamed: 44','Unnamed: 45','Unnamed: 46','Unnamed: 47','Unnamed: 48','Unnamed: 49','Unnamed: 50'], axis=1, inplace=True)
df_can.head(2)
Out[25]:
Let's rename the columns:¶
In [26]:
df_can.rename(columns={'OdName':'Country', 'AreaName':'Continent', 'RegName':'Region'}, inplace=True)
df_can.columns
Out[26]:
Let's add a 'Total' column that sums up the total immigrants by country over the entire period 1980 - 2013¶
In [27]:
df_can['Total'] = df_can.sum(axis=1)
Let's see how many null objects we have in the dataset¶
In [28]:
df_can.isnull().sum()
Out[28]:
Let's view a quick summary of each column in our dataframe:¶
In [29]:
df_can.describe()
Out[29]:
Data Slicing:¶
Let's filter on the list of countries ('Country'):¶
In [30]:
df_can.Country # returns a series
Out[30]:
Some values are NaN, so let's drop them:¶
In [32]:
df_can.dropna(subset=["Country"], axis=0, inplace=True)
In [33]:
df_can.Country
Out[33]:
Let's filter on the list of countries and the data for years: 1980 - 1985:¶
In [34]:
df_can[['Country', 1980, 1981, 1982, 1983, 1984, 1985]]
Out[34]:
Notice that the default index of the dataset is a numeric range from 0 to 195. This makes it very difficult to do a query by a specific country. For example to search for data on Japan, we need to know the corresponding index value, so let's make the Country column as our index:¶
In [35]:
df_can.set_index('Country', inplace=True)
In [36]:
df_can.head(3)
Out[36]:
Let's view the number of immigrants from Japan for the following scenarios:¶
1. The full row data (all columns)¶
In [37]:
df_can.loc['Japan']
# or you can use df_can.iloc[87]
Out[37]:
2. For year 2013¶
In [40]:
df_can.loc['Japan', 2013]
# or you can use df_can.iloc[87, 36]
Out[40]:
3. For years 1980 to 1985¶
In [41]:
df_can.loc['Japan', [1980, 1981, 1982, 1983, 1984, 1984]]
# or you can use df_can.iloc[87, [3, 4, 5, 6, 7, 8]]
Out[41]:
To avoid this ambuigity, let's convert the column names into strings: '1980' to '2013':¶
In [42]:
df_can.columns = list(map(str, df_can.columns))
Now, let's declare a variable that will allow us to easily call upon the full range of years:¶
In [44]:
years = list(map(str, range(1980, 2014)))
years
Out[44]:
Filtering based on criteria:¶
In [45]:
# 1. create the condition boolean series
condition = df_can['Continent'] == 'Asia'
print(condition)
In [46]:
# 2. pass this condition into the dataFrame
df_can[condition]
Out[46]:
We can pass multiple criteria in the same line¶
Let's filter for AreaNAme = Asia and RegName = Southern Asia¶
In [47]:
df_can[(df_can['Continent']=='Asia') & (df_can['Region']=='Southern Asia')]
Out[47]:
Let's review the changes we have made to our dataframe.¶
In [48]:
print('data dimensions:', df_can.shape)
print(df_can.columns)
df_can.head(2)
Out[48]:
Data Visualization:¶
Let's prepare the enviroment:¶
In [49]:
# we are using the inline backend
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
Now let's check if Matplotlib is loaded¶
In [50]:
print('Matplotlib version: ', mpl.__version__) # >= 2.0.0
Let's apply style:¶
In [51]:
print(plt.style.available)
mpl.style.use(['ggplot']) # optional: for ggplot-like style
Line Pots (Series/Dataframe)¶
Let's plot a line graph of immigration from Haiti¶
In [52]:
# First, we will extract the data series for Haiti:
haiti = df_can.loc['Haiti', years] # passing in years 1980 - 2013 to exclude the 'total' column
haiti.head()
Out[52]:
In [53]:
haiti.plot()
Out[53]:
Let's change the index values of Haiti to type integer for plotting, and label the x and y axes¶
In [54]:
haiti.index = haiti.index.map(int) # let's change the index values of Haiti to type integer for plotting
haiti.plot(kind='line')
plt.title('Immigration from Haiti')
plt.ylabel('Number of immigrants')
plt.xlabel('Years')
plt.show() # need this line to show the updates made to the figure
Let's annotate this spike in the plot "year 2010" (it was due to an earthquake in Haiti)¶
In [55]:
haiti.plot(kind='line')
plt.title('Immigration from Haiti')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
# annotate the 2010 Earthquake.
# syntax: plt.text(x, y, label)
plt.text(2000, 6000, '2010 Earthquake')
plt.show()
Let's compare the number of immigrants from India and China from 1980 to 2013:¶
In [56]:
df_CI = df_can.loc[['India', 'China'], years]
df_CI
Out[56]:
In [57]:
# plotting a graph
df_CI.plot(kind='line')
Out[57]:
Not working, we need to transpose the x and y axis:¶
In [58]:
df_CI = df_CI.transpose()
df_CI.head()
Out[58]:
In [59]:
# Now, let's try again
df_CI.index = df_CI.index.map(int) # let's change the index values of df_CI to type integer for plotting
df_CI.plot(kind='line')
plt.title('Immigrants from China and India')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
plt.show()
Let's compare the trend of top 5 countries that contributed the most to immigration to Canada.¶
1.Get the dataset. Recall that we created a Total column that calculates cumulative immigration by country, we will sort on this column to get our top 5 countries using pandas sort_values() method.¶
In [ ]:
inplace = True # paramemter saves the changes to the original df_can dataframe
df_can.sort_values(by='Total', ascending=False, axis=0, inplace=True)
# get the top 5 entries
df_top5 = df_can.head(5)
# transpose the dataframe
df_top5 = df_top5[years].transpose()
print(df_top5)
2. Plot the dataframe. To make the plot more readeable, we will change the size using the figsize parameter.¶
In [61]:
df_top5.index = df_top5.index.map(int) # let's change the index values of df_top5 to type integer for plotting
df_top5.plot(kind='line', figsize=(14, 8)) # pass a tuple (x, y) size
plt.title('Immigration Trend of Top 5 Countries')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
plt.show()
