Car Evaluation Sample Project - Part 3
Car Evaluation Dataset: https://archive.ics.uci.edu/ml/datasets/Car+Evaluation
Third Step: Exploratory Analysis¶
Analyzing Individual Feature Patterns Using Visualization¶
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# list the data types for each column
print(df.dtypes)
calculate the correlation between variables of type "int64" or "float64"¶
df.corr()
df[['bore', 'stroke', 'compression-ratio', 'horsepower']].corr()
Let's find the scatterplot of "engine-size" and "price".¶
# Engine size as potential predictor variable of price
sns.regplot(x="engine-size", y="price", data=df)
plt.ylim(0,)
We can examine the correlation between 'engine-size' and 'price' and see that it's approximately 0.87.¶
df[["engine-size", "price"]].corr()
Highway mpg is a potential predictor variable of price. Let's find the scatterplot of "highway-mpg" and "price".¶
sns.regplot(x="highway-mpg", y="price", data=df)
df[['highway-mpg', 'price']].corr()
Let's see if "peak-rpm" is a predictor variable of "price".¶
sns.regplot(x="peak-rpm", y="price", data=df)
Peak rpm does not seem like a good predictor of the price at all since the regression line is close to horizontal
df[['peak-rpm','price']].corr()
df[["stroke","price"]].corr()
sns.regplot(x="stroke", y="price", data=df)
There is a weak correlation between the variable 'stroke' and 'price'
Categorical Variables¶
#check body-style
sns.boxplot(x="body-style", y="price", data=df)
We see that the distributions of price between the different body-style categories have a significant overlap, so body-style would not be a good predictor of price
sns.boxplot(x="engine-location", y="price", data=df)
Here we see that the distribution of price between these two engine-location categories, front and rear, are distinct enough to take engine-location as a potential good predictor of price.
# check drive-wheels
sns.boxplot(x="drive-wheels", y="price", data=df)
Here we see that the distribution of price between the different drive-wheels categories differs. As such, drive-wheels could potentially be a predictor of price.
Descriptive Statistical Analysis¶
df.describe()
The default setting of "describe" skips variables of type object. We can apply the method "describe" on the variables of type 'object' as follows:¶
df.describe(include=['object'])
Value Counts¶
df['drive-wheels'].value_counts()
We can convert the series to a dataframe as follows:¶
df['drive-wheels'].value_counts().to_frame()
Let's repeat the above steps but save the results to the dataframe "drive_wheels_counts" and rename the column 'drive-wheels' to 'value_counts':¶
drive_wheels_counts = df['drive-wheels'].value_counts().to_frame()
drive_wheels_counts.rename(columns={'drive-wheels': 'value_counts'}, inplace=True)
drive_wheels_counts
Now let's rename the index to 'drive-wheels':¶
drive_wheels_counts.index.name = 'drive-wheels'
drive_wheels_counts
We can repeat the above process for the variable 'engine-location'.¶
# engine-location as variable
engine_loc_counts = df['engine-location'].value_counts().to_frame()
engine_loc_counts.rename(columns={'engine-location': 'value_counts'}, inplace=True)
engine_loc_counts.index.name = 'engine-location'
engine_loc_counts.head(10)
After examining the value counts of the engine location, we see that engine location would not be a good predictor variable for the price. This is because we only have three cars with a rear engine and 198 with an engine in the front, so this result is skewed. Thus, we are not able to draw any conclusions about the engine location.
Correlation and Causation¶
- p-value is < 0.001: we say there is strong evidence that the correlation is significant.
- the p-value is < 0.05: there is moderate evidence that the correlation is significant.
- the p-value is < 0.1: there is weak evidence that the correlation is significant.
- the p-value is > 0.1: there is no evidence that the correlation is significant.
from scipy import stats
Let's calculate the Pearson Correlation Coefficient and P-value of 'wheel-base' and 'price'.¶
pearson_coef, p_value = stats.pearsonr(df['wheel-base'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P =", p_value)
Since the p-value is < 0.001, the correlation between wheel-base and price is statistically significant, although the linear relationship isn't extremely strong (~0.589).
Let's calculate the Pearson Correlation Coefficient and P-value of 'horsepower' and 'price'.¶
pearson_coef, p_value = stats.pearsonr(df['horsepower'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value)
Since the p-value is < 0.001, the correlation between horsepower and price is statistically significant, and the linear relationship is quite strong (~0.809, close to 1).
Let's calculate the Pearson Correlation Coefficient and P-value of 'length' and 'price'.¶
pearson_coef, p_value = stats.pearsonr(df['length'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value)
Since the p-value is < 0.001, the correlation between length and price is statistically significant, and the linear relationship is moderately strong (~0.691).
Let's calculate the Pearson Correlation Coefficient and P-value of 'width' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['width'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P =", p_value )
Since the p-value is < 0.001, the correlation between width and price is statistically significant, and the linear relationship is quite strong (~0.752).
Let's calculate the Pearson Correlation Coefficient and P-value of 'curb-weight' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['curb-weight'], df['price'])
print( "The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value)
Since the p-value is < 0.001, the correlation between curb-weight and price is statistically significant, and the linear relationship is quite strong (~0.834)
Let's calculate the Pearson Correlation Coefficient and P-value of 'engine-size' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['engine-size'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P =", p_value)
Since the p-value is < 0.001, the correlation between engine-size and price is statistically significant, and the linear relationship is very strong (~0.872).
Let's calculate the Pearson Correlation Coefficient and P-value of 'bore' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['bore'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value )
Since the p-value is < 0.001, the correlation between bore and price is statistically significant, but the linear relationship is only moderate (~0.521).
Let's calculate the Pearson Correlation Coefficient and P-value of 'city-mpg' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['city-mpg'], df['price'])
print("The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value)
Since the p-value is < 0.001, the correlation between city-mpg and price is statistically significant, and the coefficient of about -0.687 shows that the relationship is negative and moderately strong.
Let's calculate the Pearson Correlation Coefficient and P-value of 'highway-mpg' and 'price':¶
pearson_coef, p_value = stats.pearsonr(df['highway-mpg'], df['price'])
print( "The Pearson Correlation Coefficient is", pearson_coef, " with a P-value of P = ", p_value )
Since the p-value is < 0.001, the correlation between highway-mpg and price is statistically significant, and the coefficient of about -0.705 shows that the relationship is negative and moderately strong.
Conclusion:¶
Important continous numerical variables:
Length
Width
Curb-weight
Engine-size
Horsepower
City-mpg
Highway-mpg
Wheel-base
Bore
Important categorical variables:
Drive-wheels