Selecting the best model with Hyperparameter tuning.
The first three chapters focused on model validation techniques. In chapter 4 we apply these techniques, specifically cross-validation, while learning about hyperparameter tuning. After all, model validation makes tuning possible and helps us select the overall best model. This is the Summary of lecture "Model Validation in Python", via datacamp.
import pandas as pd
import numpy as np
Introduction to hyperparameter tuning
- Model Parameters
- Learned or estimated from the data
- The result of fitting a model
- Used when making future predictions
- Not manually set
- Model Hyperparameters
- Manually set before the training occurs
- Specify how the training is supposed to happen
- Hyperparameter tuning
- Select hyperparameters
- Run a single model type at different value sets
- Create ranges of possible values to select from
- Specify a single accuracy metric
Creating Hyperparameters
For a school assignment, your professor has asked your class to create a random forest model to predict the average test score for the final exam.
After developing an initial random forest model, you are unsatisfied with the overall accuracy. You realize that there are too many hyperparameters to choose from, and each one has a lot of possible values. You have decided to make a list of possible ranges for the hyperparameters you might use in your next model.
Your professor has provided de-identified data for the last ten quizzes to act as the training data. There are 30 students in your class.
from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor(n_estimators='warn', max_features='auto', random_state=1111)
rfr.get_params()
max_depth = [4, 8, 12]
# Minimum samples for a split
min_samples_split = [2, 5, 10]
# Max features
max_features = [4, 6, 8, 10]
Hyperparameter tuning requires selecting parameters to tune, as well the possible values these parameters can be set to.
import random
# Fill in rfr using your variables
rfr = RandomForestRegressor(
n_estimators=100,
max_depth=random.sample(max_depth, 1)[0],
min_samples_split=random.sample(min_samples_split, 1)[0],
max_features = random.sample(max_features, 1)[0]
)
# Print out the parameters
print(rfr.get_params())
Preparing for RandomizedSearch
Last semester your professor challenged your class to build a predictive model to predict final exam test scores. You tried running a few different models by randomly selecting hyperparameters. However, running each model required you to code it individually.
After learning about RandomizedSearchCV(), you're revisiting your professors challenge to build the best model. In this exercise, you will prepare the three necessary inputs for completing a random search.
from sklearn.metrics import make_scorer, mean_squared_error
# Finish the dictionary by adding the max_depth parameter
param_dist = {
"max_depth": [2, 4, 6, 8],
"max_features": [2, 4, 6, 8, 10],
"min_samples_split": [2, 4, 8, 16]
}
# Create a random forest regression model
rfr = RandomForestRegressor(n_estimators=10, random_state=1111)
# Create a scorer to use (use the mean squared error)
scorer = make_scorer(mean_squared_error)
To use RandomizedSearchCV(), you need a distribution dictionary, an estimator, and a scorer—once you've got these, you can run a random search to find the best parameters for your model.
Implementing RandomizedSearchCV
You are hoping that using a random search algorithm will help you improve predictions for a class assignment. You professor has challenged your class to predict the overall final exam average score.
In preparation for completing a random search, you have created:
-
param_dist: the hyperparameter distributions -
rfr: a random forest regression model -
scorer: a scoring method to use
from sklearn.model_selection import RandomizedSearchCV
# Build a random search using param_dist, rfr, and scorer
random_search = RandomizedSearchCV(estimator=rfr,
param_distributions=param_dist,
n_iter=10,
cv=5,
scoring=scorer
)
Although it takes a lot of steps, hyperparameter tuning with random search is well worth it and can improve the accuracy of your models. Plus, you are already using cross-validation to validate your best model.
tic_tac_toe = pd.read_csv('./dataset/tic-tac-toe.csv')
# Create dummy variables using pandas
X = pd.get_dummies(tic_tac_toe.iloc[:, 0:9])
y = tic_tac_toe.iloc[:, 9]
y = tic_tac_toe['Class'].apply(lambda x: 1 if x == 'positive' else 0)
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(max_features='auto')
param_dist = {
'max_depth': [2, 4, 8, 12],
'min_samples_split': [2, 4, 6, 8],
'n_estimators':[10, 20, 30]
}
rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist, n_iter=10,
cv=5, scoring=None, random_state=1111)
rs.fit(X, y)
rs.best_params_
From the result of random search, we can find that the optimal parameter is:
{'n_estimators': 20, 'min_samples_split': 4, 'max_depth': 12}
Selecting the best precision model
Your boss has offered to pay for you to see three sports games this year. Of the 41 home games your favorite team plays, you want to ensure you go to three home games that they will definitely win. You build a model to decide which games your team will win.
To do this, you will build a random search algorithm and focus on model precision (to ensure your team wins). You also want to keep track of your best model and best parameters, so that you can use them again next year (if the model does well, of course).
sports = pd.read_csv('./dataset/sports.csv')
X = sports.drop('win', axis=1)
y = sports['win']
rfc = RandomForestClassifier()
param_dist = {
'max_depth': range(2, 12, 2),
'min_samples_split': range(2, 12, 2),
'n_estimators': [10, 25, 50]
}
from sklearn.metrics import precision_score
# Create a precision scorer
precision = make_scorer(precision_score)
# Finalize the random search
rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist,
scoring=precision, cv=5, n_iter=10,
random_state=1111)
rs.fit(X, y)
# Print the mean test scores:
print('The accuracy for each run was: {}.'.format(rs.cv_results_['mean_test_score']))
# Print the best model scores:
print('The best accuracy for a single model was: {}'.format(rs.best_score_))
Your model's precision was 91%! The best model accurately predicts a winning game 91% of the time. If you look at the mean test scores, you can tell some of the other parameter sets did really poorly. Also, since you used cross-validation, you can be confident in your predictions. Well done!