Automatidata project

Course 4 - Regression Analysis: Simplify complex data relationships

The data consulting firm Automatidata has recently hired you as the newest member of their data analytics team. Their newest client, the NYC Taxi and Limousine Commission (New York City TLC), wants the Automatidata team to build a multiple linear regression model to predict taxi fares using existing data that was collected over the course of a year. The team is getting closer to completing the project, having completed an initial plan of action, initial Python coding work, EDA, and A/B testing.

The Automatidata team has reviewed the results of the A/B testing. Now it’s time to work on predicting the taxi fare amounts. You’ve impressed your Automatidata colleagues with your hard work and attention to detail. The data team believes that you are ready to build the regression model and update the client New York City TLC about your progress.

A notebook was structured and prepared to help you in this project. Please complete the following questions.

Course 4 End-of-course project: Build a multiple linear regression model

In this activity, you will build a multiple linear regression model. As you've learned, multiple linear regression helps you estimate the linear relationship between one continuous dependent variable and two or more independent variables. For data science professionals, this is a useful skill because it allows you to consider more than one variable against the variable you're measuring against. This opens the door for much more thorough and flexible analysis to be completed.

Completing this activity will help you practice planning out and buidling a multiple linear regression model based on a specific business need. The structure of this activity is designed to emulate the proposals you will likely be assigned in your career as a data professional. Completing this activity will help prepare you for those career moments.

The purpose of this project is to demostrate knowledge of EDA and a multiple linear regression model

The goal is to build a multiple linear regression model and evaluate the model
This activity has three parts:

Part 1: EDA & Checking Model Assumptions

• What are some purposes of EDA before constructing a multiple linear regression model?

Part 2: Model Building and evaluation

• What resources do you find yourself using as you complete this stage?

Part 3: Interpreting Model Results

• What key insights emerged from your model(s)?

• What business recommendations do you propose based on the models built?

Follow the instructions and answer the questions below to complete the activity. Then, you will complete an Executive Summary using the questions listed on the PACE Strategy Document.

Be sure to complete this activity before moving on. The next course item will provide you with a completed exemplar to compare to your own work.

Build a multiple linear regression model

PACE stages

Throughout these project notebooks, you'll see references to the problem-solving framework PACE. The following notebook components are labeled with the respective PACE stage: Plan, Analyze, Construct, and Execute.

PACE: Plan

Consider the questions in your PACE Strategy Document to reflect on the Plan stage.

Task 1. Imports and loading

Import the packages that you've learned are needed for building linear regression models.

# Imports
# Packages for numerics + dataframes
### YOUR CODE HERE ###
import pandas as pd
import numpy as np

# Packages for visualization
### YOUR CODE HERE ###
from matplotlib import pyplot as plt
import seaborn as sns

# Packages for date conversions for calculating trip durations
### YOUR CODE HERE ###
from datetime import datetime, date

# Packages for OLS, MLR, confusion matrix
### YOUR CODE HERE ###
from sklearn.model_selection import train_test_split
from statsmodels.formula.api import ols

Note: Pandas is used to load the NYC TLC dataset. As shown in this cell, the dataset has been automatically loaded in for you. You do not need to download the .csv file, or provide more code, in order to access the dataset and proceed with this lab. Please continue with this activity by completing the following instructions.

# Load dataset into dataframe 
df0=pd.read_csv("2017_Yellow_Taxi_Trip_Data.csv")

PACE: Analyze

In this stage, consider the following question where applicable to complete your code response:

• What are some purposes of EDA before constructing a multiple linear regression model?

==> ENTER YOUR RESPONSE HERE

Task 2a. Explore data with EDA

Analyze and discover data, looking for correlations, missing data, outliers, and duplicates.

Start with .shape and .info().

# Start with `.shape` and `.info()`
### YOUR CODE HERE ###
print(df0.shape)
print(df0.info())

(22699, 18)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 22699 entries, 0 to 22698
Data columns (total 18 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   Unnamed: 0             22699 non-null  int64  
 1   VendorID               22699 non-null  int64  
 2   tpep_pickup_datetime   22699 non-null  object 
 3   tpep_dropoff_datetime  22699 non-null  object 
 4   passenger_count        22699 non-null  int64  
 5   trip_distance          22699 non-null  float64
 6   RatecodeID             22699 non-null  int64  
 7   store_and_fwd_flag     22699 non-null  object 
 8   PULocationID           22699 non-null  int64  
 9   DOLocationID           22699 non-null  int64  
 10  payment_type           22699 non-null  int64  
 11  fare_amount            22699 non-null  float64
 12  extra                  22699 non-null  float64
 13  mta_tax                22699 non-null  float64
 14  tip_amount             22699 non-null  float64
 15  tolls_amount           22699 non-null  float64
 16  improvement_surcharge  22699 non-null  float64
 17  total_amount           22699 non-null  float64
dtypes: float64(8), int64(7), object(3)
memory usage: 3.1+ MB
None

Check for missing data and duplicates using .isna() and .drop_duplicates().

# Check for missing data and duplicates using .isna() and .drop_duplicates()
### YOUR CODE HERE ###
df0.drop_duplicates(inplace=True)

Use .describe().

# Use .describe()
### YOUR CODE HERE ###
df0.describe(include='all')

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount
count	2.269900e+04	22699.000000	22699	22699	22699.000000	22699.000000	22699.000000	22699	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000	22699.000000
unique	NaN	NaN	22687	22688	NaN	NaN	NaN	2	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
top	NaN	NaN	07/03/2017 3:45:19 PM	10/18/2017 8:07:45 PM	NaN	NaN	NaN	N	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
freq	NaN	NaN	2	2	NaN	NaN	NaN	22600	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
mean	5.675849e+07	1.556236	NaN	NaN	1.642319	2.913313	1.043394	NaN	162.412353	161.527997	1.336887	13.026629	0.333275	0.497445	1.835781	0.312542	0.299551	16.310502
std	3.274493e+07	0.496838	NaN	NaN	1.285231	3.653171	0.708391	NaN	66.633373	70.139691	0.496211	13.243791	0.463097	0.039465	2.800626	1.399212	0.015673	16.097295
min	1.212700e+04	1.000000	NaN	NaN	0.000000	0.000000	1.000000	NaN	1.000000	1.000000	1.000000	-120.000000	-1.000000	-0.500000	0.000000	0.000000	-0.300000	-120.300000
25%	2.852056e+07	1.000000	NaN	NaN	1.000000	0.990000	1.000000	NaN	114.000000	112.000000	1.000000	6.500000	0.000000	0.500000	0.000000	0.000000	0.300000	8.750000
50%	5.673150e+07	2.000000	NaN	NaN	1.000000	1.610000	1.000000	NaN	162.000000	162.000000	1.000000	9.500000	0.000000	0.500000	1.350000	0.000000	0.300000	11.800000
75%	8.537452e+07	2.000000	NaN	NaN	2.000000	3.060000	1.000000	NaN	233.000000	233.000000	2.000000	14.500000	0.500000	0.500000	2.450000	0.000000	0.300000	17.800000
max	1.134863e+08	2.000000	NaN	NaN	6.000000	33.960000	99.000000	NaN	265.000000	265.000000	4.000000	999.990000	4.500000	0.500000	200.000000	19.100000	0.300000	1200.290000

Task 2b. Convert pickup & dropoff columns to datetime

# Check the format of the data
### YOUR CODE HERE ###
df0[['tpep_pickup_datetime', 'tpep_dropoff_datetime']].info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 22699 entries, 0 to 22698
Data columns (total 2 columns):
 #   Column                 Non-Null Count  Dtype 
---  ------                 --------------  ----- 
 0   tpep_pickup_datetime   22699 non-null  object
 1   tpep_dropoff_datetime  22699 non-null  object
dtypes: object(2)
memory usage: 532.0+ KB

# Convert datetime columns to datetime
### YOUR CODE HERE ###
df0['tpep_pickup_datetime'] = pd.to_datetime(df0['tpep_pickup_datetime'])
df0['tpep_dropoff_datetime'] = pd.to_datetime(df0['tpep_dropoff_datetime'])

Task 2c. Create duration column

Create a new column called duration that represents the total number of minutes that each taxi ride took.

# Create `duration` column
### YOUR CODE HERE ###
df0['duration'] = df0['tpep_dropoff_datetime'] - df0['tpep_pickup_datetime']
df0['duration'] = df0['duration'].dt.total_seconds() / 60
df0['duration'].describe()

count    22699.000000
mean        17.013777
std         61.996482
min        -16.983333
25%          6.650000
50%         11.183333
75%         18.383333
max       1439.550000
Name: duration, dtype: float64

Outliers

Call df.info() to inspect the columns and decide which ones to check for outliers.

### YOUR CODE HERE ###
df0.info()
# trip_distance, fare_amount, total_amount, duration

<class 'pandas.core.frame.DataFrame'>
Int64Index: 22699 entries, 0 to 22698
Data columns (total 19 columns):
 #   Column                 Non-Null Count  Dtype         
---  ------                 --------------  -----         
 0   Unnamed: 0             22699 non-null  int64         
 1   VendorID               22699 non-null  int64         
 2   tpep_pickup_datetime   22699 non-null  datetime64[ns]
 3   tpep_dropoff_datetime  22699 non-null  datetime64[ns]
 4   passenger_count        22699 non-null  int64         
 5   trip_distance          22699 non-null  float64       
 6   RatecodeID             22699 non-null  int64         
 7   store_and_fwd_flag     22699 non-null  object        
 8   PULocationID           22699 non-null  int64         
 9   DOLocationID           22699 non-null  int64         
 10  payment_type           22699 non-null  int64         
 11  fare_amount            22699 non-null  float64       
 12  extra                  22699 non-null  float64       
 13  mta_tax                22699 non-null  float64       
 14  tip_amount             22699 non-null  float64       
 15  tolls_amount           22699 non-null  float64       
 16  improvement_surcharge  22699 non-null  float64       
 17  total_amount           22699 non-null  float64       
 18  duration               22699 non-null  float64       
dtypes: datetime64[ns](2), float64(9), int64(7), object(1)
memory usage: 3.5+ MB

Keeping in mind that many of the features will not be used to fit your model, the most important columns to check for outliers are likely to be:

• trip_distance
• fare_amount
• duration

Task 2d. Box plots

Plot a box plot for each feature: trip_distance, fare_amount, duration.

### YOUR CODE HERE ###

tdbp = sns.boxplot(df0['trip_distance'])
plt.show()

fabp = sns.boxplot(df0['fare_amount'])
plt.show()

tabp = sns.boxplot(df0['total_amount'])
plt.show()

dbp = sns.boxplot(df0['duration'])
plt.show()

Questions:

1. Which variable(s) contains outliers?

2. Are the values in the trip_distance column unbelievable?

3. What about the lower end? Do distances, fares, and durations of 0 (or negative values) make sense?

==> ENTER YOUR RESPONSE HERE All of the selected variables contain outliers. The trip distances values doesn't contain any unbelievable values, except for the minimum trip distaces value, 0, yet the max although an outlier is within reason, 33. All of the selected variables contain either a 0 or a negative value which doesn't make any sense.

Task 2e. Imputations

trip_distance outliers

You know from the summary statistics that there are trip distances of 0. Are these reflective of erroneous data, or are they very short trips that get rounded down?

To check, sort the column values, eliminate duplicates, and inspect the least 10 values. Are they rounded values or precise values?

# Are trip distances of 0 bad data or very short trips rounded down?
### YOUR CODE HERE ###
td = df0.sort_values(by='trip_distance')
td.drop_duplicates()
td

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration
22026	63642923	1	2017-07-27 07:44:24	2017-07-27 07:44:24	1	0.00	1	N	41	264	2	10.50	0.0	0.5	0.00	0.00	0.3	11.30	0.000000
795	101135030	1	2017-11-30 07:11:34	2017-11-30 07:11:34	1	0.00	1	N	246	264	2	8.00	0.0	0.5	0.00	0.00	0.3	8.80	0.000000
6908	24162045	2	2017-03-26 02:07:08	2017-03-26 02:07:12	1	0.00	5	N	61	61	1	18.00	0.0	0.0	2.00	0.00	0.3	20.30	0.066667
13561	14504365	1	2017-02-23 16:06:31	2017-02-23 16:06:54	2	0.00	5	N	175	175	3	32.00	0.0	0.0	0.00	0.00	0.3	32.30	0.383333
12238	95544923	1	2017-11-11 09:28:13	2017-11-11 09:28:27	2	0.00	1	N	145	145	2	2.50	0.0	0.5	0.00	0.00	0.3	3.30	0.233333
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
29	94052446	2	2017-11-06 20:30:50	2017-11-07 00:00:00	1	30.83	1	N	132	23	1	80.00	0.5	0.5	18.56	11.52	0.3	111.38	209.166667
10291	76319330	2	2017-09-11 11:41:04	2017-09-11 12:18:58	1	31.95	4	N	138	265	2	131.00	0.0	0.5	0.00	0.00	0.3	131.80	37.900000
6064	49894023	2	2017-06-13 12:30:22	2017-06-13 13:37:51	1	32.72	3	N	138	1	1	107.00	0.0	0.0	55.50	16.26	0.3	179.06	67.483333
13861	40523668	2	2017-05-19 08:20:21	2017-05-19 09:20:30	1	33.92	5	N	229	265	1	200.01	0.0	0.5	51.64	5.76	0.3	258.21	60.150000
9280	51810714	2	2017-06-18 23:33:25	2017-06-19 00:12:38	2	33.96	5	N	132	265	2	150.00	0.0	0.0	0.00	0.00	0.3	150.30	39.216667

22699 rows × 19 columns

The distances are captured with a high degree of precision. However, it might be possible for trips to have distances of zero if a passenger summoned a taxi and then changed their mind. Besides, are there enough zero values in the data to pose a problem?

Calculate the count of rides where the trip_distance is zero.

### YOUR CODE HERE ###
df0['trip_distance'][df0['trip_distance']==0].count()

148

fare_amount outliers

### YOUR CODE HERE ###
df0[df0['fare_amount']<=0]

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration
314	105454287	2	2017-12-13 02:02:39	2017-12-13 02:03:08	6	0.12	1	N	161	161	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-3.8	0.483333
1646	57337183	2	2017-07-05 11:02:23	2017-07-05 11:03:00	1	0.04	1	N	79	79	3	-2.5	0.0	-0.5	0.0	0.0	-0.3	-3.3	0.616667
4402	108016954	2	2017-12-20 16:06:53	2017-12-20 16:47:50	1	7.06	1	N	263	169	2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	40.950000
4423	97329905	2	2017-11-16 20:13:30	2017-11-16 20:14:50	2	0.06	1	N	237	237	4	-3.0	-0.5	-0.5	0.0	0.0	-0.3	-4.3	1.333333
5448	28459983	2	2017-04-06 12:50:26	2017-04-06 12:52:39	1	0.25	1	N	90	68	3	-3.5	0.0	-0.5	0.0	0.0	-0.3	-4.3	2.216667
5722	49670364	2	2017-06-12 12:08:55	2017-06-12 12:08:57	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.033333
5758	833948	2	2017-01-03 20:15:23	2017-01-03 20:15:39	1	0.02	1	N	170	170	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-3.8	0.266667
8204	91187947	2	2017-10-28 20:39:36	2017-10-28 20:41:59	1	0.41	1	N	236	237	3	-3.5	-0.5	-0.5	0.0	0.0	-0.3	-4.8	2.383333
10281	55302347	2	2017-06-05 17:34:25	2017-06-05 17:36:29	2	0.00	1	N	238	238	4	-2.5	-1.0	-0.5	0.0	0.0	-0.3	-4.3	2.066667
10506	26005024	2	2017-03-30 03:14:26	2017-03-30 03:14:28	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.033333
11204	58395501	2	2017-07-09 07:20:59	2017-07-09 07:23:50	1	0.64	1	N	50	48	3	-4.5	0.0	-0.5	0.0	0.0	-0.3	-5.3	2.850000
12944	29059760	2	2017-04-08 00:00:16	2017-04-08 23:15:57	1	0.17	5	N	138	138	4	-120.0	0.0	0.0	0.0	0.0	-0.3	-120.3	1395.683333
14714	109276092	2	2017-12-24 22:37:58	2017-12-24 22:41:08	5	0.40	1	N	164	161	4	-4.0	-0.5	-0.5	0.0	0.0	-0.3	-5.3	3.166667
17602	24690146	2	2017-03-24 19:31:13	2017-03-24 19:34:49	1	0.46	1	N	87	45	4	-4.0	-1.0	-0.5	0.0	0.0	-0.3	-5.8	3.600000
18565	43859760	2	2017-05-22 15:51:20	2017-05-22 15:52:22	1	0.10	1	N	230	163	3	-3.0	0.0	-0.5	0.0	0.0	-0.3	-3.8	1.033333
19067	58713019	1	2017-07-10 14:40:09	2017-07-10 14:40:59	1	0.10	5	N	261	13	3	0.0	0.0	0.0	0.0	0.0	0.3	0.3	0.833333
20317	75926915	2	2017-09-09 22:59:51	2017-09-09 23:02:06	1	0.24	1	N	116	116	4	-3.5	-0.5	-0.5	0.0	0.0	-0.3	-4.8	2.250000
20698	14668209	2	2017-02-24 00:38:17	2017-02-24 00:42:05	1	0.70	1	N	65	25	4	-4.5	-0.5	-0.5	0.0	0.0	-0.3	-5.8	3.800000
21842	31708083	1	2017-04-18 16:55:29	2017-04-18 18:29:44	2	20.40	5	N	264	264	3	0.0	0.0	0.0	0.0	12.5	0.3	12.8	94.250000
22566	19022898	2	2017-03-07 02:24:47	2017-03-07 02:24:50	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.050000

Question: What do you notice about the values in the fare_amount column?

Impute values less than $0 with 0.

# Impute values less than $0 with 0
### YOUR CODE HERE ###
df0[(df0['fare_amount']<=0)]

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration
314	105454287	2	2017-12-13 02:02:39	2017-12-13 02:03:08	6	0.12	1	N	161	161	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-3.8	0.483333
1646	57337183	2	2017-07-05 11:02:23	2017-07-05 11:03:00	1	0.04	1	N	79	79	3	-2.5	0.0	-0.5	0.0	0.0	-0.3	-3.3	0.616667
4402	108016954	2	2017-12-20 16:06:53	2017-12-20 16:47:50	1	7.06	1	N	263	169	2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	40.950000
4423	97329905	2	2017-11-16 20:13:30	2017-11-16 20:14:50	2	0.06	1	N	237	237	4	-3.0	-0.5	-0.5	0.0	0.0	-0.3	-4.3	1.333333
5448	28459983	2	2017-04-06 12:50:26	2017-04-06 12:52:39	1	0.25	1	N	90	68	3	-3.5	0.0	-0.5	0.0	0.0	-0.3	-4.3	2.216667
5722	49670364	2	2017-06-12 12:08:55	2017-06-12 12:08:57	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.033333
5758	833948	2	2017-01-03 20:15:23	2017-01-03 20:15:39	1	0.02	1	N	170	170	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-3.8	0.266667
8204	91187947	2	2017-10-28 20:39:36	2017-10-28 20:41:59	1	0.41	1	N	236	237	3	-3.5	-0.5	-0.5	0.0	0.0	-0.3	-4.8	2.383333
10281	55302347	2	2017-06-05 17:34:25	2017-06-05 17:36:29	2	0.00	1	N	238	238	4	-2.5	-1.0	-0.5	0.0	0.0	-0.3	-4.3	2.066667
10506	26005024	2	2017-03-30 03:14:26	2017-03-30 03:14:28	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.033333
11204	58395501	2	2017-07-09 07:20:59	2017-07-09 07:23:50	1	0.64	1	N	50	48	3	-4.5	0.0	-0.5	0.0	0.0	-0.3	-5.3	2.850000
12944	29059760	2	2017-04-08 00:00:16	2017-04-08 23:15:57	1	0.17	5	N	138	138	4	-120.0	0.0	0.0	0.0	0.0	-0.3	-120.3	1395.683333
14714	109276092	2	2017-12-24 22:37:58	2017-12-24 22:41:08	5	0.40	1	N	164	161	4	-4.0	-0.5	-0.5	0.0	0.0	-0.3	-5.3	3.166667
17602	24690146	2	2017-03-24 19:31:13	2017-03-24 19:34:49	1	0.46	1	N	87	45	4	-4.0	-1.0	-0.5	0.0	0.0	-0.3	-5.8	3.600000
18565	43859760	2	2017-05-22 15:51:20	2017-05-22 15:52:22	1	0.10	1	N	230	163	3	-3.0	0.0	-0.5	0.0	0.0	-0.3	-3.8	1.033333
19067	58713019	1	2017-07-10 14:40:09	2017-07-10 14:40:59	1	0.10	5	N	261	13	3	0.0	0.0	0.0	0.0	0.0	0.3	0.3	0.833333
20317	75926915	2	2017-09-09 22:59:51	2017-09-09 23:02:06	1	0.24	1	N	116	116	4	-3.5	-0.5	-0.5	0.0	0.0	-0.3	-4.8	2.250000
20698	14668209	2	2017-02-24 00:38:17	2017-02-24 00:42:05	1	0.70	1	N	65	25	4	-4.5	-0.5	-0.5	0.0	0.0	-0.3	-5.8	3.800000
21842	31708083	1	2017-04-18 16:55:29	2017-04-18 18:29:44	2	20.40	5	N	264	264	3	0.0	0.0	0.0	0.0	12.5	0.3	12.8	94.250000
22566	19022898	2	2017-03-07 02:24:47	2017-03-07 02:24:50	1	0.00	1	N	264	193	1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.050000

Now impute the maximum value as Q3 + (6 * IQR).

### YOUR CODE HERE ###
'''
Impute upper-limit values in specified columns based on their interquartile range.

Arguments:
    column_list: A list of columns to iterate over
    iqr_factor: A number representing x in the formula:
                Q3 + (x * IQR). Used to determine maximum threshold,
                beyond which a point is considered an outlier.

The IQR is computed for each column in column_list and values exceeding
the upper threshold for each column are imputed with the upper threshold value.
'''
### YOUR CODE HERE ###
# Reassign minimum to zero
### YOUR CODE HERE ###
df0['fare_amount'][(df0['fare_amount']<=0)] = 0

# Calculate upper threshold
### YOUR CODE HERE ###
q1 = df0['fare_amount'].quantile(0.25)
q3 = df0['fare_amount'].quantile(0.75)
iqr = q3 - q1
th = q3 + (6*iqr)
# Reassign values > threshold to threshold
### YOUR CODE HERE ###
df0['fare_amount'][(df0['fare_amount']>th)] = th

fabp = sns.boxplot(df0['fare_amount'])
plt.show()

duration outliers

# Call .describe() for duration outliers
### YOUR CODE HERE ###
df0['duration'].describe()

count    22699.000000
mean        17.013777
std         61.996482
min        -16.983333
25%          6.650000
50%         11.183333
75%         18.383333
max       1439.550000
Name: duration, dtype: float64

The duration column has problematic values at both the lower and upper extremities.

• Low values: There should be no values that represent negative time. Impute all negative durations with 0.

• High values: Impute high values the same way you imputed the high-end outliers for fares: Q3 + (6 * IQR).

# Impute a 0 for any negative values
### YOUR CODE HERE ###### YOUR CODE HERE ###
# Reassign minimum to zero
### YOUR CODE HERE ###
df0['duration'][(df0['duration']<=0)] = 0

# Impute the high outliers
### YOUR CODE HERE ###
# Calculate upper threshold
### YOUR CODE HERE ###
q1 = df0['duration'].quantile(0.25)
q3 = df0['duration'].quantile(0.75)
iqr = q3 - q1
th = q3 + (6*iqr)
# Reassign values > threshold to threshold
### YOUR CODE HERE ###
df0['duration'][(df0['duration']>th)] = th

sns.boxplot(df0['duration'])
plt.show()

sns.boxplot(df0['fare_amount'])
plt.show()

Task 3a. Feature engineering

Create mean_distance column

When deployed, the model will not know the duration of a trip until after the trip occurs, so you cannot train a model that uses this feature. However, you can use the statistics of trips you do know to generalize about ones you do not know.

In this step, create a column called mean_distance that captures the mean distance for each group of trips that share pickup and dropoff points.

For example, if your data were:

Trip	Start	End	Distance
1	A	B	1
2	C	D	2
3	A	B	1.5
4	D	C	3

The results should be:

A -> B: 1.25 miles
C -> D: 2 miles
D -> C: 3 miles

Notice that C -> D is not the same as D -> C. All trips that share a unique pair of start and end points get grouped and averaged.

Then, a new column mean_distance will be added where the value at each row is the average for all trips with those pickup and dropoff locations:

Trip	Start	End	Distance	mean_distance
1	A	B	1	1.25
2	C	D	2	2
3	A	B	1.5	1.25
4	D	C	3	3

Begin by creating a helper column called pickup_dropoff, which contains the unique combination of pickup and dropoff location IDs for each row.

One way to do this is to convert the pickup and dropoff location IDs to strings and join them, separated by a space. The space is to ensure that, for example, a trip with pickup/dropoff points of 12 & 151 gets encoded differently than a trip with points 121 & 51.

So, the new column would look like this:

Trip	Start	End	pickup_dropoff
1	A	B	'A B'
2	C	D	'C D'
3	A	B	'A B'
4	D	C	'D C'

# Create `pickup_dropoff` column
### YOUR CODE HERE ###
df0['pickup_dropoff'] = df0['PULocationID'].astype(str) + ' ' + df0['DOLocationID'].astype(str)
df0['pickup_dropoff']

0        100 231
1         186 43
2        262 236
3         188 97
4          4 112
          ...   
22694     48 186
22695    132 164
22696    107 234
22697     68 144
22698    239 236
Name: pickup_dropoff, Length: 22699, dtype: object

Now, use a groupby() statement to group each row by the new pickup_dropoff column, compute the mean, and capture the values only in the trip_distance column. Assign the results to a variable named grouped.

### YOUR CODE HERE ###
grouped = df0.groupby('pickup_dropoff')['trip_distance'].mean()
grouped

pickup_dropoff
1 1         2.433333
10 148     15.700000
100 1      16.890000
100 100     0.253333
100 107     1.180000
             ...    
97 65       0.500000
97 66       1.400000
97 80       3.840000
97 90       4.420000
97 97       1.006667
Name: trip_distance, Length: 4172, dtype: float64

grouped is an object of the DataFrame class.

1. Convert it to a dictionary using the to_dict() method. Assign the results to a variable called grouped_dict. This will result in a dictionary with a key of trip_distance whose values are another dictionary. The inner dictionary's keys are pickup/dropoff points and its values are mean distances. This is the information you want.

Example:
grouped_dict = {'trip_distance': {'A B': 1.25, 'C D': 2, 'D C': 3}

2. Reassign the grouped_dict dictionary so it contains only the inner dictionary. In other words, get rid of trip_distance as a key, so:

Example:
grouped_dict = {'A B': 1.25, 'C D': 2, 'D C': 3}

# 1. Convert `grouped` to a dictionary
### YOUR CODE HERE ###
grouped_dict = grouped.to_dict()
# 2. Reassign to only contain the inner dictionary
### YOUR CODE HERE ###
grouped_dict

{'1 1': 2.433333333333333,
 '10 148': 15.7,
 '100 1': 16.89,
 '100 100': 0.25333333333333335,
 '100 107': 1.18,
 '100 113': 2.024,
 '100 114': 1.94,
 '100 12': 4.55,
 '100 125': 2.84,
 '100 13': 4.201666666666667,
 '100 132': 17.2175,
 '100 137': 1.299,
 '100 138': 10.432857142857143,
 '100 140': 2.746,
 '100 141': 2.11,
 '100 142': 1.6958333333333335,
 '100 143': 1.5825,
 '100 144': 3.0066666666666664,
 '100 148': 4.1066666666666665,
 '100 151': 3.668,
 '100 152': 4.9,
 '100 158': 1.938,
 '100 161': 0.9813888888888889,
 '100 162': 1.2163636363636363,
 '100 163': 1.2656,
 '100 164': 0.841,
 '100 166': 5.199999999999999,
 '100 170': 0.8548,
 '100 177': 12.0,
 '100 181': 9.34,
 '100 186': 0.6404761904761904,
 '100 193': 4.39,
 '100 198': 9.01,
 '100 202': 5.3,
 '100 209': 4.43,
 '100 211': 2.48,
 '100 224': 1.9500000000000002,
 '100 225': 7.5,
 '100 229': 1.7850000000000001,
 '100 230': 0.72975,
 '100 231': 3.5216666666666665,
 '100 232': 3.8449999999999998,
 '100 233': 1.2458333333333333,
 '100 234': 1.2545454545454546,
 '100 236': 3.3375,
 '100 237': 2.5566666666666666,
 '100 238': 3.3560000000000003,
 '100 239': 2.327142857142857,
 '100 243': 8.77,
 '100 244': 7.9,
 '100 246': 1.1746666666666667,
 '100 249': 1.8066666666666666,
 '100 25': 7.36,
 '100 255': 6.35,
 '100 256': 5.859999999999999,
 '100 261': 3.8075,
 '100 262': 3.8200000000000003,
 '100 263': 3.4,
 '100 39': 22.6,
 '100 4': 2.6999999999999997,
 '100 40': 7.23,
 '100 41': 4.6,
 '100 42': 6.779999999999999,
 '100 43': 2.033333333333333,
 '100 45': 3.63,
 '100 48': 0.8522727272727273,
 '100 49': 7.35,
 '100 50': 1.1800000000000002,
 '100 66': 4.7,
 '100 68': 0.9942857142857143,
 '100 7': 4.9,
 '100 74': 4.53,
 '100 75': 4.03,
 '100 79': 2.608571428571428,
 '100 87': 5.03,
 '100 88': 5.495,
 '100 90': 1.1228571428571428,
 '100 95': 9.0,
 '106 106': 0.02,
 '106 181': 1.1,
 '106 228': 1.24,
 '106 231': 3.8,
 '106 40': 0.8,
 '107 1': 15.55,
 '107 100': 1.436,
 '107 107': 0.48814814814814816,
 '107 113': 0.8969230769230769,
 '107 114': 1.207142857142857,
 '107 125': 1.8,
 '107 127': 11.57,
 '107 13': 3.8733333333333335,
 '107 130': 12.43,
 '107 132': 16.755,
 '107 137': 0.6828947368421052,
 '107 138': 10.385,
 '107 140': 2.801428571428571,
 '107 141': 2.981666666666667,
 '107 142': 3.228,
 '107 143': 4.3,
 '107 144': 1.61625,
 '107 145': 3.5300000000000002,
 '107 146': 4.3,
 '107 147': 8.11,
 '107 148': 1.7266666666666666,
 '107 152': 6.62,
 '107 158': 1.7777777777777777,
 '107 161': 1.7091666666666667,
 '107 162': 1.5677272727272729,
 '107 163': 2.4775,
 '107 164': 0.748,
 '107 170': 1.0014285714285716,
 '107 186': 1.4310344827586208,
 '107 196': 7.890000000000001,
 '107 202': 5.86,
 '107 209': 3.496,
 '107 21': 11.5,
 '107 211': 1.7342857142857144,
 '107 223': 5.7,
 '107 224': 0.8533333333333334,
 '107 229': 1.911111111111111,
 '107 23': 17.72,
 '107 230': 2.1075,
 '107 231': 3.278,
 '107 232': 2.1418181818181816,
 '107 233': 1.3471428571428572,
 '107 234': 0.6842424242424242,
 '107 236': 3.165,
 '107 237': 2.2516666666666665,
 '107 238': 4.986666666666667,
 '107 244': 8.84,
 '107 246': 1.577142857142857,
 '107 249': 1.3976470588235295,
 '107 25': 4.5,
 '107 256': 3.8666666666666667,
 '107 257': 9.0,
 '107 26': 10.33,
 '107 261': 2.19,
 '107 262': 3.5725,
 '107 263': 3.66875,
 '107 265': 4.5,
 '107 36': 5.6,
 '107 37': 4.9,
 '107 4': 1.16,
 '107 41': 5.975,
 '107 42': 7.0,
 '107 43': 2.8,
 '107 45': 2.1375,
 '107 48': 2.5460000000000003,
 '107 49': 5.0,
 '107 66': 3.58,
 '107 68': 1.3646666666666667,
 '107 7': 5.95,
 '107 74': 5.085,
 '107 75': 4.930000000000001,
 '107 79': 0.9903225806451613,
 '107 80': 4.7,
 '107 82': 7.32,
 '107 87': 3.5,
 '107 88': 3.501666666666667,
 '107 89': 6.97,
 '107 90': 0.9226666666666666,
 '112 112': 0.7,
 '112 223': 4.05,
 '112 263': 8.0,
 '112 49': 3.1,
 '112 66': 4.57,
 '112 80': 0.41500000000000004,
 '113 100': 1.9775,
 '113 106': 4.69,
 '113 107': 1.0516666666666667,
 '113 112': 4.5,
 '113 113': 0.8346153846153845,
 '113 114': 0.7644444444444445,
 '113 116': 8.55,
 '113 125': 1.2175,
 '113 13': 2.2085714285714286,
 '113 137': 1.3325,
 '113 138': 10.4,
 '113 14': 15.62,
 '113 140': 3.3,
 '113 141': 3.6333333333333333,
 '113 142': 3.6,
 '113 143': 4.39,
 '113 144': 1.1291666666666667,
 '113 146': 5.2,
 '113 148': 1.1885714285714286,
 '113 152': 8.0,
 '113 158': 0.9530769230769232,
 '113 161': 2.2449999999999997,
 '113 162': 2.2125,
 '113 163': 2.515,
 '113 164': 1.4830769230769232,
 '113 17': 4.38,
 '113 170': 1.5743749999999999,
 '113 181': 5.546666666666667,
 '113 186': 1.4091666666666667,
 '113 209': 2.893333333333333,
 '113 211': 1.0183333333333333,
 '113 22': 12.0,
 '113 224': 1.4649999999999999,
 '113 230': 2.11,
 '113 231': 1.7366666666666668,
 '113 232': 2.1375,
 '113 233': 2.1500000000000004,
 '113 234': 0.8635714285714285,
 '113 236': 3.9924999999999997,
 '113 237': 3.412,
 '113 238': 5.324,
 '113 239': 5.054,
 '113 243': 12.4,
 '113 244': 9.4,
 '113 246': 2.21,
 '113 249': 0.7723076923076924,
 '113 255': 3.7,
 '113 256': 3.3,
 '113 261': 2.0033333333333334,
 '113 262': 4.4,
 '113 263': 4.0,
 '113 264': 0.0,
 '113 33': 3.8266666666666667,
 '113 36': 6.3,
 '113 4': 1.09,
 '113 41': 9.2,
 '113 42': 8.34,
 '113 45': 2.02,
 '113 48': 2.2960000000000003,
 '113 50': 2.86,
 '113 66': 3.2,
 '113 68': 1.1981818181818182,
 '113 79': 0.7782352941176471,
 '113 80': 5.31,
 '113 87': 3.135,
 '113 88': 2.5999999999999996,
 '113 90': 0.8544444444444445,
 '113 94': 12.5,
 '114 100': 2.3966666666666665,
 '114 107': 1.2999999999999998,
 '114 112': 5.13,
 '114 113': 0.61,
 '114 114': 0.5481818181818182,
 '114 116': 9.1,
 '114 125': 0.752,
 '114 13': 1.9000000000000001,
 '114 137': 1.8425,
 '114 14': 11.11,
 '114 140': 4.1,
 '114 141': 3.9333333333333336,
 '114 142': 3.8766666666666665,
 '114 143': 4.745,
 '114 144': 0.84875,
 '114 145': 6.53,
 '114 148': 0.9109090909090909,
 '114 151': 7.3,
 '114 158': 1.356,
 '114 161': 2.80375,
 '114 162': 2.7075,
 '114 163': 3.5,
 '114 164': 1.7079999999999997,
 '114 166': 7.55,
 '114 169': 11.6,
 '114 170': 2.0833333333333335,
 '114 181': 3.77,
 '114 186': 1.6784615384615384,
 '114 190': 4.7,
 '114 209': 1.66,
 '114 211': 0.45,
 '114 217': 2.4,
 '114 223': 7.6,
 '114 224': 5.23,
 '114 225': 4.41,
 '114 229': 3.0,
 '114 230': 2.8775,
 '114 231': 1.2021428571428572,
 '114 232': 1.3399999999999999,
 '114 233': 2.3375,
 '114 234': 1.3515384615384616,
 '114 236': 4.5649999999999995,
 '114 237': 3.716666666666667,
 '114 238': 5.89,
 '114 239': 4.99,
 '114 24': 6.673333333333333,
 '114 243': 11.23,
 '114 244': 8.9,
 '114 246': 2.0066666666666664,
 '114 249': 0.8476923076923076,
 '114 255': 3.6879999999999997,
 '114 257': 4.96,
 '114 260': 7.08,
 '114 261': 1.7333333333333334,
 '114 262': 5.505,
 '114 263': 4.75,
 '114 36': 5.98,
 '114 4': 1.35,
 '114 43': 3.6,
 '114 45': 1.0866666666666667,
 '114 48': 3.3966666666666665,
 '114 49': 3.965,
 '114 50': 3.315,
 '114 62': 5.7,
 '114 65': 2.8,
 '114 66': 3.3,
 '114 68': 1.7022222222222223,
 '114 69': 10.05,
 '114 7': 6.4,
 '114 79': 1.0336363636363635,
 '114 87': 2.035,
 '114 90': 1.3199999999999998,
 '114 97': 3.7,
 '116 116': 0.47800000000000004,
 '116 119': 2.9,
 '116 132': 19.05,
 '116 159': 1.64,
 '116 162': 6.1,
 '116 166': 1.3275000000000001,
 '116 186': 6.42,
 '116 230': 6.38,
 '116 238': 3.2975000000000003,
 '116 239': 4.52,
 '116 244': 1.085,
 '116 41': 1.7149999999999999,
 '116 42': 1.5825,
 '116 68': 6.31,
 '116 74': 2.08,
 '116 75': 4.23,
 '116 79': 9.3,
 '118 118': 1.43,
 '12 100': 4.0,
 '12 13': 0.9,
 '12 142': 5.56,
 '12 144': 2.08,
 '12 151': 8.3,
 '12 163': 5.5,
 '12 164': 5.38,
 '12 170': 4.9,
 '12 48': 4.67,
 '123 123': 0.93,
 '125 1': 14.67,
 '125 100': 2.1100000000000003,
 '125 106': 5.0,
 '125 107': 2.1316666666666664,
 '125 113': 0.7,
 '125 114': 0.825,
 '125 129': 8.13,
 '125 13': 1.3,
 '125 132': 19.88,
 '125 137': 2.68,
 '125 138': 10.4575,
 '125 140': 4.955,
 '125 141': 6.5,
 '125 142': 4.8,
 '125 144': 0.6583333333333333,
 '125 148': 1.3885714285714283,
 '125 151': 5.43,
 '125 158': 0.7375,
 '125 161': 2.783333333333333,
 '125 162': 3.245,
 '125 163': 3.4000000000000004,
 '125 164': 2.615,
 '125 170': 4.0275,
 '125 186': 1.8719999999999999,
 '125 188': 5.88,
 '125 211': 0.66,
 '125 227': 9.4,
 '125 230': 2.71,
 '125 231': 1.0077777777777779,
 '125 234': 1.7242857142857144,
 '125 236': 4.8,
 '125 237': 3.47,
 '125 238': 6.09,
 '125 239': 5.05,
 '125 244': 9.07,
 '125 246': 2.265,
 '125 249': 0.6785714285714286,
 '125 255': 4.050000000000001,
 '125 256': 3.1,
 '125 261': 2.01,
 '125 263': 5.949999999999999,
 '125 42': 8.16,
 '125 48': 2.74,
 '125 49': 4.21,
 '125 68': 1.93,
 '125 75': 7.2,
 '125 79': 1.5642857142857143,
 '125 87': 2.0875,
 '125 88': 2.505,
 '125 90': 1.435,
 '125 97': 4.835,
 '127 243': 1.92,
 '128 238': 7.3,
 '129 129': 0.808,
 '129 160': 6.3,
 '129 164': 1.96,
 '129 173': 2.1,
 '129 207': 1.2,
 '129 70': 1.69,
 '13 100': 3.9799999999999995,
 '13 107': 4.665,
 '13 113': 2.75,
 '13 114': 2.0975,
 '13 12': 0.9,
 '13 125': 0.93,
 '13 13': 0.518,
 '13 132': 24.505,
 '13 137': 5.045,
 '13 138': 15.219999999999999,
 '13 14': 7.1,
 '13 140': 6.955,
 '13 141': 7.4399999999999995,
 '13 142': 5.1,
 '13 143': 5.15,
 '13 144': 2.1500000000000004,
 '13 148': 3.3075,
 '13 158': 2.35,
 '13 161': 5.9030000000000005,
 '13 162': 6.3725,
 '13 163': 5.172857142857143,
 '13 164': 6.02,
 '13 166': 7.46,
 '13 17': 5.6,
 '13 170': 6.166666666666667,
 '13 181': 3.9,
 '13 186': 3.7733333333333334,
 '13 209': 1.8,
 '13 211': 1.7325,
 '13 224': 4.8,
 '13 225': 7.51,
 '13 226': 8.3,
 '13 229': 6.343999999999999,
 '13 230': 4.56,
 '13 231': 0.9583333333333334,
 '13 232': 3.13,
 '13 233': 6.2,
 '13 234': 3.8175,
 '13 236': 8.33,
 '13 237': 7.136666666666667,
 '13 238': 6.7,
 '13 239': 5.715,
 '13 244': 10.575,
 '13 246': 2.7916666666666665,
 '13 249': 2.145,
 '13 25': 3.0,
 '13 255': 5.53,
 '13 261': 0.78,
 '13 262': 8.11,
 '13 263': 8.07,
 '13 33': 3.99,
 '13 40': 2.8,
 '13 45': 2.4,
 '13 48': 4.154285714285714,
 '13 49': 5.43,
 '13 50': 4.1,
 '13 54': 4.62,
 '13 55': 13.26,
 '13 65': 3.9600000000000004,
 '13 68': 2.7,
 '13 74': 10.36,
 '13 79': 3.9574999999999996,
 '13 85': 10.99,
 '13 87': 1.2125,
 '13 88': 0.9,
 '13 90': 2.7,
 '13 91': 7.86,
 '130 230': 12.8,
 '130 64': 6.03,
 '131 9': 2.1,
 '132 10': 3.75,
 '132 100': 17.6,
 '132 102': 7.7,
 '132 106': 20.2,
 '132 107': 17.561666666666667,
 '132 11': 17.945,
 '132 112': 15.809999999999999,
 '132 113': 18.302,
 '132 114': 21.73,
 '132 117': 12.2,
 '132 121': 10.47,
 '132 123': 15.65,
 '132 124': 5.67,
 '132 125': 18.736666666666668,
 '132 13': 20.86,
 '132 130': 6.716666666666666,
 '132 132': 2.2558620689655173,
 '132 134': 6.576666666666667,
 '132 137': 16.720000000000002,
 '132 138': 11.68625,
 '132 14': 20.065,
 '132 140': 19.293333333333333,
 '132 141': 19.14,
 '132 142': 20.406666666666666,
 '132 143': 10.905000000000001,
 '132 144': 18.537499999999998,
 '132 145': 15.837142857142856,
 '132 148': 17.994285714285716,
 '132 149': 14.32,
 '132 15': 14.4,
 '132 150': 14.85,
 '132 151': 19.834,
 '132 152': 19.1,
 '132 158': 22.7,
 '132 161': 18.601666666666667,
 '132 162': 17.082857142857144,
 '132 163': 19.229,
 '132 164': 18.7575,
 '132 166': 18.6,
 '132 17': 10.4,
 '132 170': 17.203,
 '132 174': 21.17,
 '132 177': 9.2,
 '132 179': 15.27,
 '132 181': 17.358571428571427,
 '132 186': 18.375,
 '132 188': 12.149999999999999,
 '132 189': 12.2,
 '132 19': 10.5,
 '132 195': 26.54,
 '132 196': 9.35,
 '132 197': 6.59,
 '132 198': 9.9,
 '132 201': 12.94,
 '132 205': 6.0,
 '132 209': 21.2,
 '132 211': 18.91,
 '132 212': 16.85,
 '132 213': 15.2,
 '132 215': 4.9,
 '132 216': 4.487,
 '132 218': 4.5,
 '132 22': 17.9,
 '132 220': 30.5,
 '132 222': 8.21,
 '132 223': 13.25,
 '132 224': 17.59,
 '132 225': 11.8,
 '132 226': 14.959999999999999,
 '132 228': 23.875,
 '132 229': 18.49,
 '132 23': 30.83,
 '132 230': 18.5712,
 '132 231': 20.46,
 '132 232': 18.3,
 '132 233': 17.86,
 '132 234': 17.654,
 '132 236': 19.491666666666667,
 '132 237': 19.540000000000003,
 '132 238': 20.8375,
 '132 239': 20.90125,
 '132 24': 19.14,
 '132 241': 20.5,
 '132 243': 22.1,
 '132 244': 19.9,
 '132 246': 18.515,
 '132 248': 17.22,
 '132 249': 18.7325,
 '132 25': 14.808000000000002,
 '132 252': 11.3,
 '132 255': 16.466666666666665,
 '132 256': 17.224,
 '132 257': 19.81,
 '132 259': 20.96,
 '132 26': 13.92,
 '132 261': 22.115000000000002,
 '132 262': 19.165,
 '132 263': 19.21,
 '132 264': 0.0,
 '132 265': 14.885833333333332,
 '132 28': 6.3133333333333335,
 '132 33': 18.683333333333334,
 '132 36': 15.886666666666665,
 '132 37': 14.899999999999999,
 '132 38': 7.300000000000001,
 '132 39': 9.912857142857144,
 '132 4': 18.59,
 '132 40': 14.1,
 '132 42': 17.95,
 '132 43': 18.744999999999997,
 '132 48': 18.761904761904763,
 '132 49': 11.92,
 '132 50': 18.735,
 '132 51': 19.064999999999998,
 '132 52': 26.86,
 '132 54': 27.2,
 '132 55': 17.3,
 '132 61': 10.69,
 '132 62': 13.229999999999999,
 '132 64': 13.6,
 '132 65': 15.686000000000002,
 '132 66': 19.3,
 '132 68': 18.7975,
 '132 7': 14.783333333333333,
 '132 70': 11.3,
 '132 71': 11.34,
 '132 72': 10.19,
 '132 74': 17.25,
 '132 76': 9.256666666666666,
 '132 77': 9.0,
 '132 79': 19.43166666666667,
 '132 80': 15.606666666666667,
 '132 82': 10.335,
 '132 83': 11.15,
 '132 85': 13.45,
 '132 86': 7.8,
 '132 87': 19.96,
 '132 88': 20.6,
 '132 89': 14.71,
 '132 9': 16.51,
 '132 90': 18.666666666666668,
 '132 91': 13.835,
 '132 92': 10.515,
 '132 93': 10.265,
 '132 95': 8.1,
 '132 97': 16.35,
 '133 133': 4.43,
 '134 197': 2.2,
 '135 75': 12.85,
 '137 100': 1.458,
 '137 107': 0.6836363636363636,
 '137 112': 5.1,
 '137 113': 1.338,
 '137 114': 1.7600000000000002,
 '137 125': 2.33,
 '137 13': 5.48,
 '137 132': 22.26,
 '137 135': 10.36,
 '137 137': 0.4633333333333333,
 '137 138': 8.4,
 '137 14': 12.34,
 '137 140': 2.186666666666667,
 '137 141': 1.83,
 '137 142': 3.08,
 '137 145': 2.7,
 '137 148': 1.4,
 '137 158': 2.66,
 '137 161': 1.46875,
 '137 162': 1.1652173913043478,
 '137 163': 1.99,
 '137 164': 0.6784615384615384,
 '137 170': 0.7204347826086956,
 '137 181': 7.65,
 '137 186': 0.9963636363636365,
 '137 209': 4.09,
 '137 220': 12.3,
 '137 223': 7.03,
 '137 224': 0.9,
 '137 229': 1.264,
 '137 230': 1.5200000000000002,
 '137 231': 3.3175,
 '137 232': 2.5949999999999998,
 '137 233': 0.886875,
 '137 234': 1.0316666666666667,
 '137 236': 3.0,
 '137 237': 2.2125,
 '137 238': 5.4,
 '137 239': 4.43,
 '137 243': 9.69,
 '137 246': 2.11,
 '137 249': 2.1574999999999998,
 '137 255': 4.35,
 '137 261': 5.49,
 '137 262': 3.005,
 '137 263': 3.5325,
 '137 4': 1.6219999999999999,
 '137 41': 5.125,
 '137 42': 6.91,
 '137 43': 2.9,
 '137 45': 2.36,
 '137 48': 2.1533333333333333,
 '137 50': 2.7,
 '137 61': 8.133333333333333,
 '137 68': 1.672,
 '137 7': 4.2,
 '137 74': 4.68,
 '137 79': 1.30875,
 '137 82': 5.8,
 '137 87': 4.34,
 '137 88': 4.449999999999999,
 '137 90': 1.3,
 '138 1': 32.72,
 '138 100': 9.765,
 '138 106': 11.0,
 '138 107': 9.463333333333333,
 '138 112': 7.25,
 '138 113': 11.0875,
 '138 114': 11.45,
 '138 116': 8.015,
 '138 121': 6.7,
 '138 125': 14.567499999999999,
 '138 127': 10.16,
 '138 129': 4.00875,
 '138 13': 14.410000000000002,
 '138 130': 7.21,
 '138 132': 12.577142857142858,
 '138 134': 6.859999999999999,
 '138 137': 8.75,
 '138 138': 0.9528571428571428,
 '138 14': 16.18,
 '138 140': 8.88125,
 '138 141': 9.37,
 '138 142': 10.852307692307694,
 '138 143': 10.466666666666667,
 '138 144': 12.341666666666667,
 '138 145': 7.723333333333333,
 '138 146': 4.134,
 '138 148': 11.54,
 '138 15': 8.100000000000001,
 '138 151': 9.15,
 '138 152': 8.86,
 '138 158': 13.899999999999999,
 '138 160': 6.68,
 '138 161': 10.131388888888889,
 '138 162': 9.673,
 '138 163': 10.425714285714285,
 '138 164': 9.65157894736842,
 '138 166': 8.26,
 '138 17': 9.2425,
 '138 170': 8.977058823529413,
 '138 171': 7.33,
 '138 174': 14.1,
 '138 175': 9.3,
 '138 177': 10.95,
 '138 178': 18.225,
 '138 179': 3.9266666666666663,
 '138 180': 11.2,
 '138 181': 11.776666666666666,
 '138 182': 10.29,
 '138 186': 11.034,
 '138 188': 13.36,
 '138 189': 13.433333333333332,
 '138 192': 4.66,
 '138 196': 5.6066666666666665,
 '138 197': 7.495,
 '138 198': 9.96,
 '138 200': 12.3,
 '138 209': 12.950000000000001,
 '138 210': 20.5,
 '138 211': 12.796666666666667,
 '138 220': 13.39,
 '138 223': 3.0500000000000003,
 '138 224': 9.870000000000001,
 '138 225': 9.817499999999999,
 '138 226': 4.58,
 '138 229': 10.012,
 '138 230': 10.601590909090909,
 '138 231': 13.138333333333334,
 '138 232': 11.485,
 '138 233': 8.775555555555556,
 '138 234': 10.5275,
 '138 236': 8.83,
 '138 237': 9.46625,
 '138 238': 9.202222222222222,
 '138 239': 10.172727272727274,
 '138 243': 10.42,
 '138 244': 10.085,
 '138 246': 10.52,
 '138 249': 11.162,
 '138 25': 10.888333333333334,
 '138 252': 5.23,
 '138 255': 7.432222222222222,
 '138 256': 8.68,
 '138 257': 16.4,
 '138 260': 4.27,
 '138 261': 16.1,
 '138 262': 8.055,
 '138 263': 8.690000000000001,
 '138 265': 20.552,
 '138 29': 21.65,
 '138 33': 11.03,
 '138 36': 7.7299999999999995,
 '138 37': 8.565,
 '138 4': 10.49,
 '138 41': 8.065,
 '138 42': 7.01,
 '138 43': 10.445,
 '138 48': 10.377692307692307,
 '138 49': 9.823333333333332,
 '138 50': 10.78,
 '138 51': 13.8,
 '138 52': 12.3,
 '138 53': 5.3,
 '138 56': 4.1,
 '138 61': 13.31,
 '138 62': 10.11,
 '138 64': 12.36,
 '138 65': 9.932857142857143,
 '138 66': 10.6,
 '138 68': 11.475000000000001,
 '138 69': 7.5,
 '138 7': 3.61125,
 '138 70': 1.4266666666666667,
 '138 74': 6.7875,
 '138 75': 8.1475,
 '138 79': 10.967500000000001,
 '138 80': 6.99,
 '138 81': 14.47,
 '138 82': 3.52,
 '138 83': 3.335,
 '138 87': 13.8125,
 '138 88': 15.393333333333333,
 '138 89': 15.11,
 '138 90': 10.943999999999999,
 '138 92': 3.65,
 '138 93': 4.29,
 '138 95': 5.085,
 '138 97': 11.11,
 '138 98': 8.7,
 '14 14': 0.38,
 '140 107': 3.115,
 '140 113': 4.1975,
 '140 125': 8.28,
 '140 13': 7.64,
 '140 132': 18.7,
 '140 135': 14.48,
 '140 137': 2.58,
 '140 138': 9.305,
 '140 140': 0.5986363636363636,
 '140 141': 0.7527272727272727,
 '140 142': 1.7822222222222222,
 '140 143': 2.32,
 '140 151': 2.83,
 '140 161': 1.8414285714285714,
 '140 162': 1.4966666666666666,
 '140 163': 1.5966666666666667,
 '140 164': 2.4683333333333333,
 '140 166': 4.2,
 '140 170': 2.122,
 '140 179': 4.22,
 '140 186': 3.31625,
 '140 193': 4.15,
 '140 209': 6.17,
 '140 211': 5.53,
 '140 223': 6.4,
 '140 224': 3.1,
 '140 226': 3.62,
 '140 229': 1.1131578947368421,
 '140 230': 2.41,
 '140 231': 7.45,
 '140 232': 5.75,
 '140 233': 1.7408333333333335,
 '140 234': 3.5033333333333334,
 '140 236': 1.2205714285714286,
 '140 237': 0.9748837209302326,
 '140 238': 2.196,
 '140 239': 2.347142857142857,
 '140 24': 4.36,
 '140 243': 6.84,
 '140 244': 7.56,
 '140 246': 4.95,
 '140 249': 5.073333333333333,
 '140 260': 4.82,
 '140 262': 0.8657692307692308,
 '140 263': 0.9194117647058824,
 '140 4': 3.825,
 '140 43': 1.7,
 '140 45': 5.0,
 '140 48': 2.669166666666667,
 '140 50': 3.0,
 '140 52': 7.66,
 '140 65': 7.12,
 '140 66': 7.5,
 '140 68': 4.1025,
 '140 7': 4.96,
 '140 74': 2.9725,
 '140 75': 1.807142857142857,
 '140 79': 4.2,
 '140 83': 5.7,
 '140 85': 11.32,
 '140 87': 5.8933333333333335,
 '140 88': 6.109999999999999,
 '140 90': 3.685,
 '140 95': 7.6,
 '140 97': 9.0,
 '141 100': 2.5149999999999997,
 '141 107': 2.60125,
 '141 112': 4.4,
 '141 113': 3.75,
 '141 114': 3.9,
 '141 116': 6.41,
 '141 13': 7.11,
 '141 130': 13.8,
 '141 132': 19.41,
 '141 133': 11.54,
 '141 137': 2.312307692307692,
 '141 138': 9.285,
 '141 140': 0.9331578947368421,
 '141 141': 0.8211764705882354,
 '141 142': 1.7133333333333332,
 '141 143': 1.99,
 '141 145': 2.9,
 '141 148': 5.65,
 '141 151': 3.125,
 '141 158': 4.8,
 '141 161': 1.5977777777777777,
 '141 162': 1.142121212121212,
 '141 163': 1.2077777777777778,
 '141 164': 2.2333333333333334,
 '141 166': 4.166666666666667,
 '141 170': 1.7866666666666668,
 '141 173': 6.45,
 '141 178': 14.0,
 '141 186': 2.7445454545454546,
 '141 193': 2.72,
 '141 196': 7.63,
 '141 209': 6.66,
 '141 211': 5.2,
 '141 220': 10.23,
 '141 224': 3.0,
 '141 226': 2.48,
 '141 229': 0.9359090909090909,
 '141 230': 1.8555555555555554,
 '141 231': 7.246666666666666,
 '141 233': 1.3258333333333334,
 '141 234': 2.982,
 '141 236': 1.1402083333333333,
 '141 237': 0.6136842105263158,
 '141 238': 2.375,
 '141 239': 2.0327272727272727,
 '141 24': 3.9,
 '141 243': 7.63,
 '141 244': 7.68,
 '141 246': 4.55,
 '141 249': 5.66,
 '141 255': 5.13,
 '141 261': 6.885,
 '141 262': 0.8171428571428571,
 '141 263': 0.9044444444444445,
 '141 4': 4.6675,
 '141 42': 4.046666666666667,
 '141 43': 1.231111111111111,
 '141 48': 2.526666666666667,
 '141 50': 2.6775,
 '141 65': 8.0,
 '141 68': 3.1975,
 '141 7': 3.5333333333333337,
 '141 74': 3.1374999999999997,
 '141 75': 1.90125,
 '141 79': 3.9475,
 '141 80': 5.5649999999999995,
 '141 88': 7.26,
 '141 90': 4.05,
 '142 100': 1.6228571428571428,
 '142 107': 3.2199999999999998,
 '142 113': 3.2,
 '142 114': 3.74,
 '142 116': 4.556666666666667,
 '142 125': 3.99,
 '142 127': 8.965,
 '142 129': 5.68,
 '142 13': 5.058333333333334,
 '142 132': 20.77,
 '142 137': 2.985,
 '142 138': 9.133333333333333,
 '142 140': 2.2916666666666665,
 '142 141': 1.7023076923076923,
 '142 142': 0.628974358974359,
 '142 143': 0.8415384615384615,
 '142 144': 4.48,
 '142 145': 3.6,
 '142 148': 7.87,
 '142 151': 2.0314285714285716,
 '142 158': 2.7,
 '142 161': 1.426875,
 '142 162': 1.6821052631578948,
 '142 163': 0.8309375,
 '142 164': 2.382,
 '142 166': 2.6875,
 '142 17': 8.274999999999999,
 '142 170': 2.3214285714285716,
 '142 174': 12.6,
 '142 181': 8.14,
 '142 186': 1.8606666666666667,
 '142 209': 7.3,
 '142 211': 5.0,
 '142 220': 9.0,
 '142 223': 5.795,
 '142 224': 3.885,
 '142 225': 8.8,
 '142 229': 1.6320000000000001,
 '142 230': 1.051212121212121,
 '142 231': 4.8425,
 '142 233': 2.2944444444444443,
 '142 234': 2.9166666666666665,
 '142 236': 2.0282758620689654,
 '142 237': 1.3573333333333333,
 '142 238': 1.44875,
 '142 239': 0.9964999999999999,
 '142 24': 2.1580000000000004,
 '142 243': 7.75,
 '142 244': 6.058333333333334,
 '142 246': 2.075714285714286,
 '142 249': 2.982,
 '142 261': 6.45,
 '142 262': 2.686666666666667,
 '142 263': 2.29,
 '142 264': 0.4,
 '142 41': 2.9244444444444446,
 '142 42': 3.94,
 '142 43': 1.1046153846153848,
 '142 48': 0.9956756756756756,
 '142 50': 1.0758333333333334,
 '142 68': 1.8776470588235294,
 '142 74': 3.8925,
 ...}

1. Create a mean_distance column that is a copy of the pickup_dropoff helper column.

2. Use the map() method on the mean_distance series. Pass grouped_dict as its argument. Reassign the result back to the mean_distance series. When you pass a dictionary to the Series.map() method, it will replace the data in the series where that data matches the dictionary's keys. The values that get imputed are the values of the dictionary.

Example:
df['mean_distance']

mean_distance
'A B'
'C D'
'A B'
'D C'
'E F'

grouped_dict = {'A B': 1.25, 'C D': 2, 'D C': 3}
df['mean_distance`] = df['mean_distance'].map(grouped_dict)
df['mean_distance']

mean_distance
1.25
2
1.25
3
NaN

When used this way, the map() Series method is very similar to replace(), however, note that map() will impute NaN for any values in the series that do not have a corresponding key in the mapping dictionary, so be careful.

# 1. Create a mean_distance column that is a copy of the pickup_dropoff helper column
### YOUR CODE HERE ###
df0['mean_distance'] = df0['pickup_dropoff']
# 2. Map `grouped_dict` to the `mean_distance` column
### YOUR CODE HERE ###
df0['mean_distance'] = df0['mean_distance'].map(grouped_dict)
# Confirm that it worked
### YOUR CODE HERE ###
df0.head()

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	...	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration	pickup_dropoff	mean_distance
0	24870114	2	2017-03-25 08:55:43	2017-03-25 09:09:47	6	3.34	1	N	100	231	...	13.0	0.0	0.5	2.76	0.0	0.3	16.56	14.066667	100 231	3.521667
1	35634249	1	2017-04-11 14:53:28	2017-04-11 15:19:58	1	1.80	1	N	186	43	...	16.0	0.0	0.5	4.00	0.0	0.3	20.80	26.500000	186 43	3.108889
2	106203690	1	2017-12-15 07:26:56	2017-12-15 07:34:08	1	1.00	1	N	262	236	...	6.5	0.0	0.5	1.45	0.0	0.3	8.75	7.200000	262 236	0.881429
3	38942136	2	2017-05-07 13:17:59	2017-05-07 13:48:14	1	3.70	1	N	188	97	...	20.5	0.0	0.5	6.39	0.0	0.3	27.69	30.250000	188 97	3.700000
4	30841670	2	2017-04-15 23:32:20	2017-04-15 23:49:03	1	4.37	1	N	4	112	...	16.5	0.5	0.5	0.00	0.0	0.3	17.80	16.716667	4 112	4.435000

5 rows × 21 columns

Create mean_duration column

Repeat the process used to create the mean_distance column to create a mean_duration column.

### YOUR CODE HERE ###
grouped_duration = df0.groupby('pickup_dropoff')['duration'].mean()
grouped_duration = grouped_duration.to_dict()

# Create a dictionary where keys are unique pickup_dropoffs and values are
# mean trip duration for all trips with those pickup_dropoff combos
### YOUR CODE HERE ###
df0['mean_duration'] = df0['pickup_dropoff']
df0['mean_duration'] = df0['mean_duration'].map(grouped_duration)
# Confirm that it worked
### YOUR CODE HERE ###
df0.head()

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	...	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration	pickup_dropoff	mean_distance	mean_duration
0	24870114	2	2017-03-25 08:55:43	2017-03-25 09:09:47	6	3.34	1	N	100	231	...	0.0	0.5	2.76	0.0	0.3	16.56	14.066667	100 231	3.521667	22.847222
1	35634249	1	2017-04-11 14:53:28	2017-04-11 15:19:58	1	1.80	1	N	186	43	...	0.0	0.5	4.00	0.0	0.3	20.80	26.500000	186 43	3.108889	24.470370
2	106203690	1	2017-12-15 07:26:56	2017-12-15 07:34:08	1	1.00	1	N	262	236	...	0.0	0.5	1.45	0.0	0.3	8.75	7.200000	262 236	0.881429	7.250000
3	38942136	2	2017-05-07 13:17:59	2017-05-07 13:48:14	1	3.70	1	N	188	97	...	0.0	0.5	6.39	0.0	0.3	27.69	30.250000	188 97	3.700000	30.250000
4	30841670	2	2017-04-15 23:32:20	2017-04-15 23:49:03	1	4.37	1	N	4	112	...	0.5	0.5	0.00	0.0	0.3	17.80	16.716667	4 112	4.435000	14.616667

5 rows × 22 columns

Create day and month columns

Create two new columns, day (name of day) and month (name of month) by extracting the relevant information from the tpep_pickup_datetime column.

# Create 'day' col
### YOUR CODE HERE ###
df0['day'] = df0['tpep_pickup_datetime'].dt.day_name()
# Create 'month' col
### YOUR CODE HERE ###
df0['month'] = df0['tpep_pickup_datetime'].dt.month_name()

df0.head()

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	...	tip_amount	tolls_amount	improvement_surcharge	total_amount	duration	pickup_dropoff	mean_distance	mean_duration	day	month
0	24870114	2	2017-03-25 08:55:43	2017-03-25 09:09:47	6	3.34	1	N	100	231	...	2.76	0.0	0.3	16.56	14.066667	100 231	3.521667	22.847222	Saturday	March
1	35634249	1	2017-04-11 14:53:28	2017-04-11 15:19:58	1	1.80	1	N	186	43	...	4.00	0.0	0.3	20.80	26.500000	186 43	3.108889	24.470370	Tuesday	April
2	106203690	1	2017-12-15 07:26:56	2017-12-15 07:34:08	1	1.00	1	N	262	236	...	1.45	0.0	0.3	8.75	7.200000	262 236	0.881429	7.250000	Friday	December
3	38942136	2	2017-05-07 13:17:59	2017-05-07 13:48:14	1	3.70	1	N	188	97	...	6.39	0.0	0.3	27.69	30.250000	188 97	3.700000	30.250000	Sunday	May
4	30841670	2	2017-04-15 23:32:20	2017-04-15 23:49:03	1	4.37	1	N	4	112	...	0.00	0.0	0.3	17.80	16.716667	4 112	4.435000	14.616667	Saturday	April

5 rows × 24 columns

Create rush_hour column

Define rush hour as:

• Any weekday (not Saturday or Sunday) AND
• Either from 06:00–10:00 or from 16:00–20:00

Create a binary rush_hour column that contains a 1 if the ride was during rush hour and a 0 if it was not.

# Create 'rush_hour' col
### YOUR CODE HERE ###
df0['rush_hour'] = 0
# If day is Saturday or Sunday, impute 0 in `rush_hour` column
### YOUR CODE HERE ###
df0['rush_hour'][((df0['tpep_pickup_datetime'].dt.hour >= 6) & (df0['tpep_pickup_datetime'].dt.hour <= 10)) |
                ((df0['tpep_pickup_datetime'].dt.hour >= 16) & (df0['tpep_pickup_datetime'].dt.hour <= 20))] = 1
df0['rush_hour'][(df0['day']=='Saturday') | (df0['day']=='Sunday')] = 0

df0.head(20)

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	...	tolls_amount	improvement_surcharge	total_amount	duration	pickup_dropoff	mean_distance	mean_duration	day	month	rush_hour
0	24870114	2	2017-03-25 08:55:43	2017-03-25 09:09:47	6	3.34	1	N	100	231	...	0.00	0.3	16.56	14.066667	100 231	3.521667	22.847222	Saturday	March	0
1	35634249	1	2017-04-11 14:53:28	2017-04-11 15:19:58	1	1.80	1	N	186	43	...	0.00	0.3	20.80	26.500000	186 43	3.108889	24.470370	Tuesday	April	0
2	106203690	1	2017-12-15 07:26:56	2017-12-15 07:34:08	1	1.00	1	N	262	236	...	0.00	0.3	8.75	7.200000	262 236	0.881429	7.250000	Friday	December	1
3	38942136	2	2017-05-07 13:17:59	2017-05-07 13:48:14	1	3.70	1	N	188	97	...	0.00	0.3	27.69	30.250000	188 97	3.700000	30.250000	Sunday	May	0
4	30841670	2	2017-04-15 23:32:20	2017-04-15 23:49:03	1	4.37	1	N	4	112	...	0.00	0.3	17.80	16.716667	4 112	4.435000	14.616667	Saturday	April	0
5	23345809	2	2017-03-25 20:34:11	2017-03-25 20:42:11	6	2.30	1	N	161	236	...	0.00	0.3	12.36	8.000000	161 236	2.052258	11.855376	Saturday	March	0
6	37660487	2	2017-05-03 19:04:09	2017-05-03 20:03:47	1	12.83	1	N	79	241	...	0.00	0.3	59.16	59.633333	79 241	12.830000	59.633333	Wednesday	May	1
7	69059411	2	2017-08-15 17:41:06	2017-08-15 18:03:05	1	2.98	1	N	237	114	...	0.00	0.3	19.58	21.983333	237 114	4.022500	26.437500	Tuesday	August	1
8	8433159	2	2017-02-04 16:17:07	2017-02-04 16:29:14	1	1.20	1	N	234	249	...	0.00	0.3	9.80	12.116667	234 249	1.019259	7.873457	Saturday	February	0
9	95294817	1	2017-11-10 15:20:29	2017-11-10 15:40:55	1	1.60	1	N	239	237	...	0.00	0.3	16.55	20.433333	239 237	1.580000	10.541111	Friday	November	0
10	18017909	2	2017-03-04 11:58:00	2017-03-04 12:13:12	1	1.77	1	N	162	142	...	0.00	0.3	14.76	15.200000	162 142	1.641000	14.178333	Saturday	March	0
11	18600059	2	2017-03-05 19:15:30	2017-03-05 19:52:18	2	18.90	2	N	236	132	...	5.54	0.3	72.92	36.800000	236 132	19.211667	40.500000	Sunday	March	0
12	46782248	1	2017-06-09 19:00:26	2017-06-09 19:20:11	1	3.00	1	N	13	148	...	0.00	0.3	20.15	19.750000	13 148	3.307500	15.058333	Friday	June	1
13	94113247	2	2017-11-06 23:35:05	2017-11-06 23:42:57	1	2.39	1	N	209	25	...	0.00	0.3	12.96	7.866667	209 25	2.390000	7.866667	Monday	November	0
14	14168279	1	2017-02-22 15:18:31	2017-02-22 15:42:50	1	3.30	1	N	238	161	...	0.00	0.3	22.85	24.316667	238 161	2.930000	19.555000	Wednesday	February	0
15	47444401	2	2017-06-02 06:41:39	2017-06-02 06:57:47	1	5.93	1	N	239	231	...	0.00	0.3	22.80	16.133333	239 231	5.950000	17.166667	Friday	June	1
16	69088676	1	2017-08-15 19:48:08	2017-08-15 20:00:37	1	3.60	1	N	163	41	...	0.00	0.3	17.15	12.483333	163 41	3.266667	14.733333	Tuesday	August	1
17	58691513	2	2017-07-10 13:36:31	2017-07-10 13:48:43	2	1.71	1	N	142	100	...	0.00	0.3	10.30	12.200000	142 100	1.622857	12.304762	Monday	July	0
18	35388828	2	2017-04-10 18:12:58	2017-04-10 18:17:39	2	0.63	1	N	263	262	...	0.00	0.3	6.80	4.683333	263 262	0.662143	4.577381	Monday	April	1
19	18383214	2	2017-03-05 04:01:07	2017-03-05 04:14:11	2	2.77	1	N	79	68	...	0.00	0.3	16.00	13.066667	79 68	2.138333	13.638889	Sunday	March	0

20 rows × 25 columns

### YOUR CODE HERE ###

# Apply the `rush_hourizer()` function to the new column
### YOUR CODE HERE ###

Task 4. Scatter plot

Create a scatterplot to visualize the relationship between mean_duration and fare_amount.

# Create a scatterplot to visualize the relationship between variables of interest
### YOUR CODE HERE ###
sns.scatterplot(data=df0, x='mean_duration', y='fare_amount')

<matplotlib.axes._subplots.AxesSubplot at 0x7b6829308a50>

The mean_duration variable correlates with the target variable. But what are the horizontal lines around fare amounts of 52 dollars and 63 dollars? What are the values and how many are there?

You know what one of the lines represents. 62 dollars and 50 cents is the maximum that was imputed for outliers, so all former outliers will now have fare amounts of $62.50. What is the other line?

Check the value of the rides in the second horizontal line in the scatter plot.

### YOUR CODE HERE ###
df0[(df0['fare_amount']>51) & (df0['fare_amount']<53)].head(30)

	Unnamed: 0	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	...	tolls_amount	improvement_surcharge	total_amount	duration	pickup_dropoff	mean_distance	mean_duration	day	month	rush_hour
11	18600059	2	2017-03-05 19:15:30	2017-03-05 19:52:18	2	18.90	2	N	236	132	...	5.54	0.3	72.92	36.800000	236 132	19.211667	40.500000	Sunday	March	0
110	47959795	1	2017-06-03 14:24:57	2017-06-03 15:31:48	1	18.00	2	N	132	163	...	0.00	0.3	52.80	66.850000	132 163	19.229000	52.941667	Saturday	June	0
156	104881101	1	2017-12-11 10:21:18	2017-12-11 11:14:57	1	15.60	1	N	138	88	...	5.76	0.3	69.66	53.650000	138 88	15.393333	64.316667	Monday	December	1
161	95729204	2	2017-11-11 20:16:16	2017-11-11 20:17:14	1	0.23	2	N	132	132	...	0.00	0.3	52.80	0.966667	132 132	2.255862	3.021839	Saturday	November	0
247	103404868	2	2017-12-06 23:37:08	2017-12-07 00:06:19	1	18.93	2	N	132	79	...	0.00	0.3	52.80	29.183333	132 79	19.431667	47.275000	Wednesday	December	0
356	108458749	2	2017-12-21 21:31:12	2017-12-21 22:11:58	6	18.17	1	N	132	145	...	0.00	0.3	52.80	40.766667	132 145	15.837143	40.759524	Thursday	December	0
379	80479432	2	2017-09-24 23:45:45	2017-09-25 00:15:14	1	17.99	2	N	132	234	...	5.76	0.3	73.20	29.483333	132 234	17.654000	49.833333	Sunday	September	0
388	16226157	1	2017-02-28 18:30:05	2017-02-28 19:09:55	1	18.40	2	N	132	48	...	5.54	0.3	62.84	39.833333	132 48	18.761905	58.246032	Tuesday	February	1
406	55253442	2	2017-06-05 12:51:58	2017-06-05 13:07:35	1	4.73	2	N	228	88	...	5.76	0.3	58.56	15.616667	228 88	4.730000	15.616667	Monday	June	0
449	65900029	2	2017-08-03 22:47:14	2017-08-03 23:32:41	2	18.21	2	N	132	48	...	5.76	0.3	58.56	45.450000	132 48	18.761905	58.246032	Thursday	August	0
468	80904240	2	2017-09-26 13:48:26	2017-09-26 14:31:17	1	17.27	2	N	186	132	...	5.76	0.3	58.56	42.850000	186 132	17.096000	42.920000	Tuesday	September	0
520	33706214	2	2017-04-23 21:34:48	2017-04-23 22:46:23	6	18.34	2	N	132	148	...	0.00	0.3	57.80	71.583333	132 148	17.994286	46.340476	Sunday	April	0
569	99259872	2	2017-11-22 21:31:32	2017-11-22 22:00:25	1	18.65	2	N	132	144	...	0.00	0.3	63.36	28.883333	132 144	18.537500	37.000000	Wednesday	November	0
572	61050418	2	2017-07-18 13:29:06	2017-07-18 13:29:19	1	0.00	2	N	230	161	...	5.76	0.3	70.27	0.216667	230 161	0.685484	7.965591	Tuesday	July	0
586	54444647	2	2017-06-26 13:39:12	2017-06-26 14:34:54	1	17.76	2	N	211	132	...	5.76	0.3	70.27	55.700000	211 132	16.580000	61.691667	Monday	June	0
692	94424289	2	2017-11-07 22:15:00	2017-11-07 22:45:32	2	16.97	2	N	132	170	...	5.76	0.3	70.27	30.533333	132 170	17.203000	37.113333	Tuesday	November	0
717	103094220	1	2017-12-06 05:19:50	2017-12-06 05:53:52	1	20.80	2	N	132	239	...	5.76	0.3	64.41	34.033333	132 239	20.901250	44.862500	Wednesday	December	0
719	66115834	1	2017-08-04 17:53:34	2017-08-04 18:50:56	1	21.60	2	N	264	264	...	5.76	0.3	75.66	57.366667	264 264	3.191516	15.618773	Friday	August	1
782	55934137	2	2017-06-09 09:31:25	2017-06-09 10:24:10	2	18.81	2	N	163	132	...	0.00	0.3	66.00	52.750000	163 132	17.275833	52.338889	Friday	June	1
816	13731926	2	2017-02-21 06:11:03	2017-02-21 06:59:39	5	16.94	2	N	132	170	...	5.54	0.3	60.34	48.600000	132 170	17.203000	37.113333	Tuesday	February	1
818	52277743	2	2017-06-20 08:15:18	2017-06-20 10:24:37	1	17.77	2	N	132	246	...	5.76	0.3	70.27	88.783333	132 246	18.515000	66.316667	Tuesday	June	1
835	2684305	2	2017-01-10 22:29:47	2017-01-10 23:06:46	1	18.57	2	N	132	48	...	0.00	0.3	66.00	36.983333	132 48	18.761905	58.246032	Tuesday	January	0
840	90860814	2	2017-10-27 21:50:00	2017-10-27 22:35:04	1	22.43	2	N	132	163	...	5.76	0.3	58.56	45.066667	132 163	19.229000	52.941667	Friday	October	0
861	106575186	1	2017-12-16 06:39:59	2017-12-16 07:07:59	2	17.80	2	N	75	132	...	5.76	0.3	64.56	28.000000	75 132	18.442500	36.204167	Saturday	December	0
881	110495611	2	2017-12-30 05:25:29	2017-12-30 06:01:29	6	18.23	2	N	68	132	...	0.00	0.3	52.80	36.000000	68 132	18.785000	58.041667	Saturday	December	0
958	87017503	1	2017-10-15 22:39:12	2017-10-15 23:14:22	1	21.80	2	N	132	261	...	0.00	0.3	52.80	35.166667	132 261	22.115000	51.493750	Sunday	October	0
970	12762608	2	2017-02-17 20:39:42	2017-02-17 21:13:29	1	19.57	2	N	132	140	...	5.54	0.3	70.01	33.783333	132 140	19.293333	36.791667	Friday	February	1
984	71264442	1	2017-08-23 18:23:26	2017-08-23 19:18:29	1	16.70	2	N	132	230	...	0.00	0.3	99.59	55.050000	132 230	18.571200	59.598000	Wednesday	August	1
1082	11006300	2	2017-02-07 17:20:19	2017-02-07 17:34:41	1	1.09	2	N	170	48	...	5.54	0.3	62.84	14.366667	170 48	1.265789	14.135965	Tuesday	February	1
1097	68882036	2	2017-08-14 23:01:15	2017-08-14 23:03:35	5	2.12	2	N	265	265	...	0.00	0.3	52.80	2.333333	265 265	0.753077	3.411538	Monday	August	0

30 rows × 25 columns

Examine the first 30 of these trips.

# Set pandas to display all columns
### YOUR CODE HERE ###

Question: What do you notice about the first 30 trips?

==> ENTER YOUR RESPONSE HERE I noticed all of them have the same fare of 52, that seems to be a default amount for some unknonw reason.

Task 5. Isolate modeling variables

Drop features that are redundant, irrelevant, or that will not be available in a deployed environment.

### YOUR CODE HERE ###
df0.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 22699 entries, 0 to 22698
Data columns (total 25 columns):
 #   Column                 Non-Null Count  Dtype         
---  ------                 --------------  -----         
 0   Unnamed: 0             22699 non-null  int64         
 1   VendorID               22699 non-null  int64         
 2   tpep_pickup_datetime   22699 non-null  datetime64[ns]
 3   tpep_dropoff_datetime  22699 non-null  datetime64[ns]
 4   passenger_count        22699 non-null  int64         
 5   trip_distance          22699 non-null  float64       
 6   RatecodeID             22699 non-null  int64         
 7   store_and_fwd_flag     22699 non-null  object        
 8   PULocationID           22699 non-null  int64         
 9   DOLocationID           22699 non-null  int64         
 10  payment_type           22699 non-null  int64         
 11  fare_amount            22699 non-null  float64       
 12  extra                  22699 non-null  float64       
 13  mta_tax                22699 non-null  float64       
 14  tip_amount             22699 non-null  float64       
 15  tolls_amount           22699 non-null  float64       
 16  improvement_surcharge  22699 non-null  float64       
 17  total_amount           22699 non-null  float64       
 18  duration               22699 non-null  float64       
 19  pickup_dropoff         22699 non-null  object        
 20  mean_distance          22699 non-null  float64       
 21  mean_duration          22699 non-null  float64       
 22  day                    22699 non-null  object        
 23  month                  22699 non-null  object        
 24  rush_hour              22699 non-null  int64         
dtypes: datetime64[ns](2), float64(11), int64(8), object(4)
memory usage: 4.5+ MB

### YOUR CODE HERE ###
df = df0.drop(columns=["Unnamed: 0", "tpep_pickup_datetime", "tpep_dropoff_datetime",
                        "trip_distance", "store_and_fwd_flag", "PULocationID", "DOLocationID",
                      "extra", "mta_tax", "tip_amount", "improvement_surcharge", "total_amount",
                       "duration", "pickup_dropoff", "day", "month"])
df.head(10)

	VendorID	passenger_count	RatecodeID	payment_type	fare_amount	tolls_amount	mean_distance	mean_duration	rush_hour
0	2	6	1	1	13.0	0.0	3.521667	22.847222	0
1	1	1	1	1	16.0	0.0	3.108889	24.470370	0
2	1	1	1	1	6.5	0.0	0.881429	7.250000	1
3	2	1	1	1	20.5	0.0	3.700000	30.250000	0
4	2	1	1	2	16.5	0.0	4.435000	14.616667	0
5	2	6	1	1	9.0	0.0	2.052258	11.855376	0
6	2	1	1	1	47.5	0.0	12.830000	59.633333	1
7	2	1	1	1	16.0	0.0	4.022500	26.437500	1
8	2	1	1	2	9.0	0.0	1.019259	7.873457	0
9	1	1	1	1	13.0	0.0	1.580000	10.541111	0

Task 6. Pair plot

Create a pairplot to visualize pairwise relationships between fare_amount, mean_duration, and mean_distance.

# Create a pairplot to visualize pairwise relationships between variables in the data
### YOUR CODE HERE ###
sns.pairplot(df)

<seaborn.axisgrid.PairGrid at 0x7b68292fe1d0>

These variables all show linear correlation with each other. Investigate this further.

Task 7. Identify correlations

Next, code a correlation matrix to help determine most correlated variables.

# Correlation matrix to help determine most correlated variables
### YOUR CODE HERE ###
corr_matrix = df.corr()
corr_matrix

	VendorID	passenger_count	RatecodeID	payment_type	fare_amount	tolls_amount	mean_distance	mean_duration	rush_hour
VendorID	1.000000	0.266463	-0.002991	-0.017787	0.001045	0.011122	0.004741	0.001876	-0.000752
passenger_count	0.266463	1.000000	-0.005743	0.016178	0.014942	0.009532	0.013428	0.015852	-0.024283
RatecodeID	-0.002991	-0.005743	1.000000	-0.000982	0.222102	0.175860	0.159353	0.111667	0.004145
payment_type	-0.017787	0.016178	-0.000982	1.000000	-0.049516	-0.041217	-0.044495	-0.054298	-0.049030
fare_amount	0.001045	0.014942	0.222102	-0.049516	1.000000	0.616719	0.910185	0.859105	-0.025901
tolls_amount	0.011122	0.009532	0.175860	-0.041217	0.616719	1.000000	0.621229	0.512261	-0.000694
mean_distance	0.004741	0.013428	0.159353	-0.044495	0.910185	0.621229	1.000000	0.874864	-0.046794
mean_duration	0.001876	0.015852	0.111667	-0.054298	0.859105	0.512261	0.874864	1.000000	-0.027499
rush_hour	-0.000752	-0.024283	0.004145	-0.049030	-0.025901	-0.000694	-0.046794	-0.027499	1.000000

Visualize a correlation heatmap of the data.

# Create correlation heatmap
### YOUR CODE HERE ###
sns.heatmap(corr_matrix)

<matplotlib.axes._subplots.AxesSubplot at 0x7b68241c4450>

Question: Which variable(s) are correlated with the target variable of fare_amount?

Try modeling with both variables even though they are correlated.

PACE: Construct

After analysis and deriving variables with close relationships, it is time to begin constructing the model. Consider the questions in your PACE Strategy Document to reflect on the Construct stage.

Task 8a. Split data into outcome variable and features

### YOUR CODE HERE ###
y = df[['fare_amount']]
X = df.drop(columns='fare_amount')
y.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 22699 entries, 0 to 22698
Data columns (total 1 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   fare_amount  22699 non-null  float64
dtypes: float64(1)
memory usage: 870.7 KB

Set your X and y variables. X represents the features and y represents the outcome (target) variable.

# Remove the target column from the features
# X = df2.drop(columns='fare_amount')
### YOUR CODE HERE ###
# Set y variable
### YOUR CODE HERE ###
# Display first few rows
### YOUR CODE HERE ###
X.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 22699 entries, 0 to 22698
Data columns (total 8 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   VendorID         22699 non-null  int64  
 1   passenger_count  22699 non-null  int64  
 2   RatecodeID       22699 non-null  int64  
 3   payment_type     22699 non-null  int64  
 4   tolls_amount     22699 non-null  float64
 5   mean_distance    22699 non-null  float64
 6   mean_duration    22699 non-null  float64
 7   rush_hour        22699 non-null  int64  
dtypes: float64(3), int64(5)
memory usage: 2.1 MB

Task 8b. Pre-process data

Dummy encode categorical variables

# Convert VendorID to string
### YOUR CODE HERE ###
X['VendorID'] = X['VendorID'].astype(str)
# Get dummies
### YOUR CODE HERE ###
Xd = pd.get_dummies(X, columns=['VendorID'])
Xd

	passenger_count	RatecodeID	payment_type	tolls_amount	mean_distance	mean_duration	rush_hour	VendorID_1	VendorID_2
0	6	1	1	0.00	3.521667	22.847222	0	0	1
1	1	1	1	0.00	3.108889	24.470370	0	1	0
2	1	1	1	0.00	0.881429	7.250000	1	1	0
3	1	1	1	0.00	3.700000	30.250000	0	0	1
4	1	1	2	0.00	4.435000	14.616667	0	0	1
...	...	...	...	...	...	...	...	...	...
22694	3	1	2	0.00	1.098214	8.594643	1	0	1
22695	1	2	1	5.76	18.757500	59.560417	0	0	1
22696	1	1	2	0.00	0.684242	6.609091	0	0	1
22697	1	1	1	0.00	2.077500	16.650000	0	0	1
22698	1	1	1	0.00	1.476970	9.405556	0	1	0

22699 rows × 9 columns

Split data into training and test sets

Create training and testing sets. The test set should contain 20% of the total samples. Set random_state=0.

# Create training and testing sets
#### YOUR CODE HERE ####
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=42)

Standardize the data

Use StandardScaler(), fit(), and transform() to standardize the X_train variables. Assign the results to a variable called X_train_scaled.

# Standardize the X variables
### YOUR CODE HERE ###
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

# 1. Fit & transform training set in one step
X_train_scaled = scaler.fit_transform(X_train)

# 2. ONLY transform test set (uses training mean/std)
X_test_scaled = scaler.transform(X_test)

X_train_scaled.head(30)

	VendorID	passenger_count	RatecodeID	payment_type	tolls_amount	mean_distance	mean_duration	rush_hour
0	-1.116436	-0.497569	-0.056136	1.329293	-0.225058	-0.407526	-0.516240	1.281921
1	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.513546	-0.514487	-0.780079
2	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.378806	0.031886	1.281921
3	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.436989	-0.709230	-0.780079
4	0.895707	-0.497569	-0.056136	1.329293	-0.225058	0.095566	0.048359	-0.780079
5	0.895707	-0.497569	-0.056136	1.329293	-0.225058	-0.515608	-0.632229	1.281921
6	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.551298	-0.684252	-0.780079
7	-1.116436	0.286433	-0.056136	1.329293	-0.225058	-0.089131	0.030389	1.281921
8	0.895707	2.638436	-0.056136	-0.681095	-0.225058	0.372284	0.240797	-0.780079
9	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.401395	-0.101861	1.281921
10	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	0.072261	0.326170	-0.780079
11	0.895707	1.070434	-0.056136	-0.681095	-0.225058	-0.539575	-0.814177	-0.780079
12	-1.116436	-0.497569	-0.056136	5.350069	-0.225058	-0.616428	-0.759116	1.281921
13	-1.116436	0.286433	-0.056136	1.329293	-0.225058	0.253412	0.857467	1.281921
14	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	0.071490	0.102894	-0.780079
15	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.364229	-0.255531	-0.780079
16	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.616074	-0.907399	-0.780079
17	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.676734	-1.037558	-0.780079
18	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	0.023742	0.273280	-0.780079
19	0.895707	0.286433	-0.056136	1.329293	-0.225058	-0.351524	-0.196280	-0.780079
20	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.315891	-0.693685	-0.780079
21	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	0.363375	0.789421	1.281921
22	0.895707	3.422437	-0.056136	-0.681095	-0.225058	-0.270421	0.274398	-0.780079
23	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	0.360592	1.177789	-0.780079
24	0.895707	0.286433	-0.056136	1.329293	-0.225058	-0.584674	-0.749212	-0.780079
25	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.538203	-0.642100	1.281921
26	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.230856	-0.155712	1.281921
27	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	-0.113595	-0.104588	1.281921
28	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	3.584321	2.245118	-0.780079
29	-1.116436	0.286433	-0.056136	-0.681095	-0.225058	0.604645	0.436509	-0.780079

X_train_scaled = pd.DataFrame(X_train_scaled, columns=["VendorID","passenger_count","RatecodeID","payment_type","tolls_amount",
                                      "mean_distance","mean_duration","rush_hour"])
X_test_scaled = pd.DataFrame(X_test_scaled, columns=["VendorID","passenger_count","RatecodeID","payment_type","tolls_amount",
                                      "mean_distance","mean_duration","rush_hour"])

y_train=y_train.reset_index(drop=True)
y_test = y_test.reset_index(drop=True)

Fit the model

Instantiate your model and fit it to the training data.

# Fit your model to the training data
### YOUR CODE HERE ###
ols_formula = "y_train ~ mean_distance + mean_duration + C(rush_hour) + C(RatecodeID) + C(VendorID)"
#ols_formula = "y_train ~ mean_distance + mean_duration + tolls_amount + C(RatecodeID)"
#ols_formula = "y_train ~ mean_distance + mean_duration + C(rush_hour)"
#ols_formula = "y_train ~ mean_distance + C(rush_hour)"
ols_data = pd.concat([X_train_scaled, y_train], axis=1)
OLS = ols(formula=ols_formula, data=ols_data)
model = OLS.fit()
result = model.summary()
result

OLS Regression Results

Dep. Variable:	y_train	R-squared:	0.870
Model:	OLS	Adj. R-squared:	0.870
Method:	Least Squares	F-statistic:	1.178e+04
Date:	Sun, 27 Sep 2026	Prob (F-statistic):	0.00
Time:	20:31:20	Log-Likelihood:	-43896.
No. Observations:	15889	AIC:	8.781e+04
Df Residuals:	15879	BIC:	8.789e+04
Df Model:	9
Covariance Type:	nonrobust

	coef	std err	t	P>|t|	[0.025	0.975]
Intercept	12.6749	0.052	244.336	0.000	12.573	12.777
C(rush_hour)[T.1.281920660157489]	0.1835	0.063	2.918	0.004	0.060	0.307
C(RatecodeID)[T.1.1524551660464013]	7.0621	0.255	27.655	0.000	6.562	7.563
C(RatecodeID)[T.2.3610460269342153]	15.1546	0.710	21.343	0.000	13.763	16.546
C(RatecodeID)[T.3.569636887822029]	9.6482	1.464	6.590	0.000	6.779	12.518
C(RatecodeID)[T.4.778227748709843]	23.1248	0.563	41.058	0.000	22.021	24.229
C(RatecodeID)[T.118.38576867216435]	48.8707	3.835	12.743	0.000	41.354	56.388
C(VendorID)[T.0.8957071444206769]	-0.0734	0.061	-1.198	0.231	-0.193	0.047
mean_distance	5.9189	0.072	82.357	0.000	5.778	6.060
mean_duration	3.3798	0.064	52.930	0.000	3.255	3.505

Omnibus:	12423.203	Durbin-Watson:	2.010
Prob(Omnibus):	0.000	Jarque-Bera (JB):	878888.645
Skew:	3.212	Prob(JB):	0.00
Kurtosis:	38.865	Cond. No.	173.

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Task 8c. Evaluate model

Train data

Evaluate your model performance by calculating the residual sum of squares and the explained variance score (R^2). Calculate the Mean Absolute Error, Mean Squared Error, and the Root Mean Squared Error.

# Evaluate the model performance on the training data
### YOUR CODE HERE ###
resid = model.resid

rss = model.ssr
r2 = model.rsquared
mae = np.mean(np.abs(resid))
mse = model.mse_resid
rmse = np.sqrt(mse)

print(rss, r2, mae, mse, rmse)

233481.58710770003 0.8697544929332063 2.231021348224999 14.703796656445622 3.834552993041773

Test data

Calculate the same metrics on the test data. Remember to scale the X_test data using the scaler that was fit to the training data. Do not refit the scaler to the testing data, just transform it. Call the results X_test_scaled.

# Scale the X_test data
### YOUR CODE HERE ###
X_test_scaled.head()

	VendorID	passenger_count	RatecodeID	payment_type	tolls_amount	mean_distance	mean_duration	rush_hour
0	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.408106	-0.111065	-0.780079
1	0.895707	-0.497569	-0.056136	-0.681095	-0.225058	-0.471788	-0.398782	1.281921
2	0.895707	2.638436	-0.056136	-0.681095	-0.225058	-0.474121	-0.102173	-0.780079
3	0.895707	2.638436	-0.056136	-0.681095	-0.225058	0.793484	0.717148	-0.780079
4	-1.116436	-0.497569	-0.056136	-0.681095	-0.225058	0.421837	-0.216407	-0.780079

# Evaluate the model performance on the testing data
### YOUR CODE HERE ###
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

y_pred = model.predict(X_test_scaled)
y_pred =pd.DataFrame(y_pred, columns=['fare_amount'])

y_true_arr = y_test.values.ravel()
y_pred_arr = y_pred.values.ravel()

resid_pred = y_true_arr - y_pred_arr

rss_pred = np.sum((resid_pred) ** 2)
r2_pred = r2_score(y_test, y_pred)
mae_pred = mean_absolute_error(y_test, y_pred)
mse_pred = mean_squared_error(y_test, y_pred)
rmse_pred = np.sqrt(mse_pred)
print(rss_pred)
print(r2_pred, mae_pred, mse_pred, rmse_pred)

91771.77211136799
0.8741432514378051 2.1564923554689073 13.476031147043757 3.670971417355869

resid_pred

array([ 2.68934101, -2.6447818 ,  2.55001958, ...,  4.4728535 ,
        1.61532064,  1.47541722])

PACE: Execute

Consider the questions in your PACE Strategy Document to reflect on the Execute stage.

Task 9a. Results

Use the code cell below to get actual,predicted, and residual for the testing set, and store them as columns in a results dataframe.

# Create a `results` dataframe
### YOUR CODE HERE ###
results = pd.DataFrame()
results['actual'] = y_test
results['predicted'] = y_pred
results['residuals'] = resid_pred

results.head()

	actual	predicted	residuals
0	12.5	9.810659	2.689341
1	6.0	8.644782	-2.644782
2	12.0	9.449980	2.550020
3	20.5	19.721934	0.778066
4	14.0	14.440292	-0.440292

Task 9b. Visualize model results

Create a scatterplot to visualize actual vs. predicted.

# Create a scatterplot to visualize `predicted` over `actual`
### YOUR CODE HERE ###
sns.scatterplot(data=results, x='actual', y='predicted')
plt.show()

Visualize the distribution of the residuals using a histogram.

# Visualize the distribution of the `residuals`
### YOUR CODE HERE ###

sns.histplot(results['residuals'])
plt.show()

# Calculate residual mean
### YOUR CODE HERE ###
results['residuals'].mean()

0.022208185869656265

Create a scatterplot of residuals over predicted.

# Create a scatterplot of `residuals` over `predicted`
### YOUR CODE HERE ###
sns.scatterplot(data=results, x='residuals', y='predicted')
plt.show()

Task 9c. Coefficients

Use the coef_ attribute to get the model's coefficients. The coefficients are output in the order of the features that were used to train the model. Which feature had the greatest effect on trip fare?

# Output the model's coefficients
coefficients = model.params

coefficients

Intercept                              12.674925
C(rush_hour)[T.1.281920660157489]       0.183481
C(RatecodeID)[T.1.1524551660464013]     7.062086
C(RatecodeID)[T.2.3610460269342153]    15.154610
C(RatecodeID)[T.3.569636887822029]      9.648232
C(RatecodeID)[T.4.778227748709843]     23.124788
C(RatecodeID)[T.118.38576867216435]    48.870687
C(VendorID)[T.0.8957071444206769]      -0.073360
mean_distance                           5.918854
mean_duration                           3.379850
dtype: float64

What do these coefficients mean? How should they be interpreted?

The greater the coef; the greater the impact the variable have in the predicted outcome. A larger number either positive or negative has a more significant impact on the prediction.

Task 9d. Conclusion

1. What are the key takeaways from this notebook?

2. What results can be presented from this notebook?

As a take away, even when concidering variables with a high correlation into the model it is possible to produce strong prediction through MLR without much overfitting. MLR is very usfull not only to more accuretly predict a dependent varible, but to understand the impact of different variables on the dependent varible. The error metric can be somewhat high, but that gives as our confidence band for our results.

From this notebook we can present our model with a R2 of 0.87 which explains 87% of the value of why given our selected independent variables. We can present the mean distance and duration value from trips from specific pick up and drop off location and how much those affect the fare amount. We can present a predicted fare amount based on the information available within a narrow margin of error (mae of 2.16).

Congratulations! You've completed this lab. However, you may not notice a green check mark next to this item on Coursera's platform. Please continue your progress regardless of the check mark. Just click on the "save" icon at the top of this notebook to ensure your work has been logged.