The same model in scikit-learn

Classroom resources

Load these into Waypoints Prepare: choose Load URL and paste the copied address, or download the file and import it. In Prepare, the worksheet's Teacher edition shows the answers; the student copy hides them.

• Slides Download Copy address for Prepare Present from the browser, or export to PowerPoint or PDF.
• Worksheet Download Copy address for Prepare Print the student copy; the Teacher edition has the answers.

Students will be able to

• I can train and test a model with scikit-learn.
• I can explain what fit, score and predict do, from building them by hand.
• I can compare models and measurements fairly on the same test set.

Timing

Predict and Run 10 min · Investigate 10 min · Modify 15 min · Make 10 min

Before class

Read the unit primer first. scikit-learn loads the first time a program imports it (several seconds; about 19 MB, cached by the browser afterwards). X is a pandas table of measurements and y the species; train_test_split(..., random_state=1) shuffles and splits; fit stores the training examples (for nearest neighbour) or learns rules (for a tree); score is accuracy on the given data (the bug task scores on training data: 100%); predict needs a table with the same columns. Results: 94% test accuracy with 1 neighbour, 93% with 5, 93% for a tree of depth 3. In the Make, adding body mass drops nearest neighbour to about 70% (grams swamp the distance) while the tree reaches 97%: a real lesson about scale, which the info question explains.

Running the lesson

1. Predict: ask what score on the training data means, from last lesson.
2. Investigate: do the match as a class, pointing at last lesson's code.
3. Make: collect the class's best combinations on the board.

The hard part (on purpose)

Mapping library lines to their hand-built versions. Ask students to annotate each line with the lesson it came from before running.

When students get stuck

Answers to the checks

1. Predict: What does it print?
   1.0. score(X_train, y_train) tests the model on the penguins it learned from, and with n_neighbors=1 each one finds itself, exactly as in the last lesson.
2. By hand and by library: Match each scikit-learn line with what you wrote by hand.
   train_test_split(X, y, test_size=0.25, random_state=1) → B. shuffle with a seed, then penguins[:250] and penguins[250:]; KNeighborsClassifier(n_neighbors=1) → A. nearest(): copy the closest penguin's species; model.fit(X_train, y_train) → D. keeping the training penguins as the examples; model.score(X_test, y_test) → C. accuracy(test, train): the share guessed right
3. The same columns: Why is the mystery penguin made with columns=X.columns?
   The model needs the same measurements, in the same order and with the same names, as it was trained on
   The model learned from bill length then flipper length. A new example must be given the same way, or its numbers mean something else.
4. Spot the bug: This says the model is 100% accurate. Click the line that is wrong.
   line 12
   It scores the model on X_train, the penguins it learned from. The test set, X_test and y_test, shows how good it really is.
5. Where this leads: Adding body mass (in grams, around 4000) made the nearest-neighbour model worse, but not the tree. Why?
   Grams are big numbers, so body mass swamps the distance; a tree looks at one measurement at a time, so it does not mind
   Distance-based models need measurements on similar scales; scikit-learn's StandardScaler rescales them. Choosing and preparing the data is half of machine learning.

Solutions

One possible solution for each task. Other correct programs are fine.

Modify › Five neighbours

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

penguins = pd.read_csv("penguins.csv")
X = penguins[["bill_length_mm", "flipper_length_mm"]]
y = penguins["species"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)
print("Test accuracy: " + str(round(model.score(X_test, y_test) * 100)) + "%")

mystery = pd.DataFrame([[47.0, 215.0]], columns=X.columns)
print("Mystery penguin: " + model.predict(mystery)[0])

Modify › A decision tree

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

penguins = pd.read_csv("penguins.csv")
X = penguins[["bill_length_mm", "flipper_length_mm"]]
y = penguins["species"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)

model = KNeighborsClassifier(n_neighbors=1)
model.fit(X_train, y_train)
print("Test accuracy: " + str(round(model.score(X_test, y_test) * 100)) + "%")

mystery = pd.DataFrame([[47.0, 215.0]], columns=X.columns)
print("Mystery penguin: " + model.predict(mystery)[0])

tree = DecisionTreeClassifier(max_depth=3, random_state=1)
tree.fit(X_train, y_train)
print("Decision tree: " + str(round(tree.score(X_test, y_test) * 100)) + "%")

the best measurements

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

penguins = pd.read_csv("penguins.csv")

y = penguins["species"]
choices = [
    ["bill_length_mm", "flipper_length_mm"],
    ["bill_length_mm", "bill_depth_mm"],
    ["bill_length_mm", "bill_depth_mm", "flipper_length_mm"],
    ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"],
]
for columns in choices:
    X = penguins[columns]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
    knn = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)
    tree = DecisionTreeClassifier(max_depth=3, random_state=1).fit(X_train, y_train)
    print(", ".join(columns) + ": neighbours " + str(round(knn.score(X_test, y_test) * 100)) + "%, tree " + str(round(tree.score(X_test, y_test) * 100)) + "%")

Stretch and support

Stretch: Add StandardScaler (from sklearn.preprocessing) before the neighbours model with make_pipeline, and see the body-mass result recover.

Support: Give students the DecisionTreeClassifier lines, and ask them only to score it on the test set.

Exit ticket

What does model.fit(X_train, y_train) do?

It gives the model the training examples to learn from (for nearest neighbour, it keeps them to compare with later).

What each step is for

• Investigate: scikit-learn's split, fit, score and predict are the steps you built by hand; knowing them makes the library's answers something you can check, not magic.
• Modify: Models have settings, and different kinds of model learn differently; a fair test compares them.
• the best measurements: Goals only: try different measurements, test each fairly, and explain what you find.

Where it leads

Next lesson puts the model behind a Gradio app. The decision tree's rules can be printed with sklearn.tree.export_text.