import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = (8, 8)


## Two-output models

### Simple two-output model

In this exercise, you will use the tournament data to build one model that makes two predictions: the scores of both teams in a given game. Your inputs will be the seed difference of the two teams, as well as the predicted score difference from the model you built in chapter 3.

The output from your model will be the predicted score for team 1 as well as team 2. This is called "multiple target regression": one model making more than one prediction.

from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model

# Define the input
input_tensor = Input(shape=(2, ))

# Define the output
output_tensor = Dense(2)(input_tensor)

# Create a model
model = Model(input_tensor, output_tensor)

# Compile the model


### Fit a model with two outputs

Now that you've defined your 2-output model, fit it to the tournament data. I've split the data into games_tourney_train and games_tourney_test, so use the training set to fit for now.

This model will use the pre-tournament seeds, as well as your pre-tournament predictions from the regular season model you built previously in this course.

As a reminder, this model will predict the scores of both teams.

games_tourney = pd.read_csv('./dataset/games_tourney.csv')

season team_1 team_2 home seed_diff score_diff score_1 score_2 won
0 1985 288 73 0 -3 -9 41 50 0
1 1985 5929 73 0 4 6 61 55 1
2 1985 9884 73 0 5 -4 59 63 0
3 1985 73 288 0 3 9 50 41 1
4 1985 3920 410 0 1 -9 54 63 0
games_season = pd.read_csv('./dataset/games_season.csv')

season team_1 team_2 home score_diff score_1 score_2 won
0 1985 3745 6664 0 17 81 64 1
1 1985 126 7493 1 7 77 70 1
2 1985 288 3593 1 7 63 56 1
3 1985 1846 9881 1 16 70 54 1
4 1985 2675 10298 1 12 86 74 1
from tensorflow.keras.layers import Embedding, Input, Flatten, Concatenate, Dense
from tensorflow.keras.models import Model

# Count the unique number of teams
n_teams = np.unique(games_season['team_1']).shape[0]

# Create an embedding layer
team_lookup = Embedding(input_dim=n_teams,
output_dim=1,
input_length=1,
name='Team-Strength')

# Create an input layer for the team ID
teamid_in = Input(shape=(1, ))

# Lookup the input in the team strength embedding layer
strength_lookup = team_lookup(teamid_in)

# Flatten the output
strength_lookup_flat = Flatten()(strength_lookup)

# Combine the operations into a single, re-usable model
team_strength_model = Model(teamid_in, strength_lookup_flat, name='Team-Strength-Model')

# Create an Input for each team
team_in_1 = Input(shape=(1, ), name='Team-1-In')
team_in_2 = Input(shape=(1, ), name='Team-2-In')

# Create an input for home vs away
home_in = Input(shape=(1, ), name='Home-In')

# Lookup the team inputs in the team strength model
team_1_strength = team_strength_model(team_in_1)
team_2_strength = team_strength_model(team_in_2)

# Combine the team strengths with the home input using a Concatenate layer,
# then add a Dense layer

out = Concatenate()([team_1_strength, team_2_strength, home_in])
out = Dense(1)(out)

# Make a model
p_model = Model([team_in_1, team_in_2, home_in], out)

# Compile the model

# Fit the model to the games_season dataset
p_model.fit([games_season['team_1'], games_season['team_2'], games_season['home']],
games_season['score_diff'],
epochs=1, verbose=True, validation_split=0.1, batch_size=2048)

games_tourney['pred'] = p_model.predict([games_tourney['team_1'],
games_tourney['team_2'],
games_tourney['home']])

138/138 [==============================] - 0s 2ms/step - loss: 11.9445 - val_loss: 12.4844

games_tourney_train = games_tourney[games_tourney['season'] <= 2010]
games_tourney_test = games_tourney[games_tourney['season'] > 2010]

model.fit(games_tourney_train[['seed_diff', 'pred']],
games_tourney_train[['score_1', 'score_2']],
verbose=False,
epochs=10000,
batch_size=256);


### Inspect the model (I)

Now that you've fit your model, let's take a look at it. You can use the .get_weights() method to inspect your model's weights.

The input layer will have 4 weights: 2 for each input times 2 for each output.

The output layer will have 2 weights, one for each output.

model.get_weights()

[array([[ 0.54203993, -0.6890212 ],
[30.134958  , 26.926764  ]], dtype=float32),
array([66.90998, 67.09042], dtype=float32)]
games_tourney_train.mean()

season        1997.548544
team_1        5560.890777
team_2        5560.890777
home             0.000000
seed_diff        0.000000
score_diff       0.000000
score_1         71.786711
score_2         71.786711
won              0.500000
pred             0.139160
dtype: float64

### Evaluate the model

Now that you've fit your model and inspected it's weights to make sure it makes sense, evaluate it on the tournament test set to see how well it performs on new data.

print(model.evaluate(games_tourney_test[['seed_diff', 'pred']],
games_tourney_test[['score_1', 'score_2']],
verbose=False))

8.851910591125488


## Single model for classification and regression

### Classification and regression in one model

Now you will create a different kind of 2-output model. This time, you will predict the score difference, instead of both team's scores and then you will predict the probability that team 1 won the game. This is a pretty cool model: it is going to do both classification and regression!

In this model, turn off the bias, or intercept for each layer. Your inputs (seed difference and predicted score difference) have a mean of very close to zero, and your outputs both have means that are close to zero, so your model shouldn't need the bias term to fit the data well.

input_tensor = Input(shape=(2, ))

# Create the first output
output_tensor_1 = Dense(1, activation='linear', use_bias=False)(input_tensor)

# Create the second output(use the first output as input here)
output_tensor_2 = Dense(1, activation='sigmoid', use_bias=False)(output_tensor_1)

# Create a model with 2 outputs
model = Model(input_tensor, [output_tensor_1, output_tensor_2])

from tensorflow.keras.utils import plot_model

plot_model(model, to_file='../images/multi_output_model.png')

plt.imshow(data);


### Compile and fit the model

Now that you have a model with 2 outputs, compile it with 2 loss functions: mean absolute error (MAE) for 'score_diff' and binary cross-entropy (also known as logloss) for 'won'. Then fit the model with 'seed_diff' and 'pred' as inputs. For outputs, predict 'score_diff' and 'won'.

This model can use the scores of the games to make sure that close games (small score diff) have lower win probabilities than blowouts (large score diff).

The regression problem is easier than the classification problem because MAE punishes the model less for a loss due to random chance. For example, if score_diff is -1 and won is 0, that means team_1 had some bad luck and lost by a single free throw. The data for the easy problem helps the model find a solution to the hard problem.

from tensorflow.keras.optimizers import Adam

# Compile the model with 2 losses and the Adam optimizer with a higher learning rate

# Fit the model to the tournament training data, with 2 inputs and 2 outputs
model.fit(games_tourney_train[['seed_diff', 'pred']],
[games_tourney_train[['score_diff']], games_tourney_train[['won']]],
epochs=20,
verbose=True,
batch_size=16384);

Epoch 1/20
1/1 [==============================] - 0s 688us/step - loss: 11.6251 - dense_2_loss: 9.9799 - dense_3_loss: 1.6452
Epoch 2/20
1/1 [==============================] - 0s 640us/step - loss: 11.5974 - dense_2_loss: 9.9550 - dense_3_loss: 1.6424
Epoch 3/20
1/1 [==============================] - 0s 647us/step - loss: 11.5692 - dense_2_loss: 9.9305 - dense_3_loss: 1.6387
Epoch 4/20
1/1 [==============================] - 0s 529us/step - loss: 11.5404 - dense_2_loss: 9.9063 - dense_3_loss: 1.6342
Epoch 5/20
1/1 [==============================] - 0s 606us/step - loss: 11.5109 - dense_2_loss: 9.8821 - dense_3_loss: 1.6288
Epoch 6/20
1/1 [==============================] - 0s 571us/step - loss: 11.4808 - dense_2_loss: 9.8582 - dense_3_loss: 1.6226
Epoch 7/20
1/1 [==============================] - 0s 628us/step - loss: 11.4507 - dense_2_loss: 9.8352 - dense_3_loss: 1.6155
Epoch 8/20
1/1 [==============================] - 0s 545us/step - loss: 11.4200 - dense_2_loss: 9.8126 - dense_3_loss: 1.6074
Epoch 9/20
1/1 [==============================] - 0s 506us/step - loss: 11.3889 - dense_2_loss: 9.7905 - dense_3_loss: 1.5984
Epoch 10/20
1/1 [==============================] - 0s 714us/step - loss: 11.3576 - dense_2_loss: 9.7692 - dense_3_loss: 1.5884
Epoch 11/20
1/1 [==============================] - 0s 723us/step - loss: 11.3259 - dense_2_loss: 9.7485 - dense_3_loss: 1.5774
Epoch 12/20
1/1 [==============================] - 0s 639us/step - loss: 11.2936 - dense_2_loss: 9.7281 - dense_3_loss: 1.5655
Epoch 13/20
1/1 [==============================] - 0s 830us/step - loss: 11.2606 - dense_2_loss: 9.7079 - dense_3_loss: 1.5527
Epoch 14/20
1/1 [==============================] - 0s 1ms/step - loss: 11.2268 - dense_2_loss: 9.6878 - dense_3_loss: 1.5391
Epoch 15/20
1/1 [==============================] - 0s 596us/step - loss: 11.1925 - dense_2_loss: 9.6679 - dense_3_loss: 1.5246
Epoch 16/20
1/1 [==============================] - 0s 530us/step - loss: 11.1575 - dense_2_loss: 9.6481 - dense_3_loss: 1.5094
Epoch 17/20
1/1 [==============================] - 0s 660us/step - loss: 11.1216 - dense_2_loss: 9.6283 - dense_3_loss: 1.4934
Epoch 18/20
1/1 [==============================] - 0s 909us/step - loss: 11.0850 - dense_2_loss: 9.6084 - dense_3_loss: 1.4766
Epoch 19/20
1/1 [==============================] - 0s 622us/step - loss: 11.0478 - dense_2_loss: 9.5888 - dense_3_loss: 1.4590
Epoch 20/20
1/1 [==============================] - 0s 510us/step - loss: 11.0100 - dense_2_loss: 9.5693 - dense_3_loss: 1.4407


### Inspect the model (II)

Now you should take a look at the weights for this model. In particular, note the last weight of the model. This weight converts the predicted score difference to a predicted win probability. If you multiply the predicted score difference by the last weight of the model and then apply the sigmoid function, you get the win probability of the game.

model.get_weights()

[array([[ 0.7577214],
[-1.0725204]], dtype=float32),
array([[-0.2949189]], dtype=float32)]
games_tourney_train.mean()

season        1997.548544
team_1        5560.890777
team_2        5560.890777
home             0.000000
seed_diff        0.000000
score_diff       0.000000
score_1         71.786711
score_2         71.786711
won              0.500000
pred             0.139160
dtype: float64
from scipy.special import expit as sigmoid

# Weight from the model
weight=0.14

# Print the approximate win probability predicted close game
print(sigmoid(1 * weight))

# Print the approximate win probability predicted blowout game
print(sigmoid(10 * weight))

0.5349429451582145
0.8021838885585818

/home/chanseok/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)


### Evaluate on new data with two metrics

Now that you've fit your model and inspected its weights to make sure they make sense, evaluate your model on the tournament test set to see how well it does on new data.

Note that in this case, Keras will return 3 numbers: the first number will be the sum of both the loss functions, and then the next 2 numbers will be the loss functions you used when defining the model.

print(model.evaluate(games_tourney_test[['seed_diff', 'pred']],
[games_tourney_test[['score_diff']], games_tourney_test[['won']]],
verbose=False))

[10.702475547790527, 9.355356216430664, 1.347119688987732]