Find where the penguin model fails: a species it has never seen, impossible measurements and harder questions, and learn to say what a model cannot do.
Models in the news make mistakes that matter: missing people, wrong answers said confidently. Knowing how they fail is part of building them.
Optional. A model can only learn from the data it was given. This lesson tries the penguin model on things it was not built for, and finishes with a model card: a short, honest description of what a model does, how well, and what it cannot do.
Make your prediction first. Help opens once you've locked it in.
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier
penguins = pd.read_csv("penguins.csv")
columns = ["bill_length_mm", "flipper_length_mm"]
model = KNeighborsClassifier(n_neighbors=5)
model.fit(penguins[columns], penguins["species"])
emperor = pd.DataFrame([[80.0, 350.0]], columns=columns)
print(model.predict(emperor)[0])
The model was trained on Adelie, Chinstrap and Gentoo penguins only. What can it possibly answer?
Run it. Two of the mystery penguins are ordinary; one is far too big and one far too small to be any of these species. Look at how sure the model is about each.
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier
penguins = pd.read_csv("penguins.csv")
columns = ["bill_length_mm", "flipper_length_mm"]
model = KNeighborsClassifier(n_neighbors=5)
model.fit(penguins[columns], penguins["species"])
mysteries = pd.DataFrame([[39.0, 190.0], [48.0, 195.0], [80.0, 350.0], [20.0, 120.0]], columns=columns)
guesses = model.predict(mysteries)
chances = model.predict_proba(mysteries)
for i in range(len(guesses)):
print(str(list(mysteries.iloc[i])) + ": " + guesses[i] + ", " + str(round(max(chances[i]) * 100)) + "% sure")
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier
penguins = pd.read_csv("penguins.csv")
columns = ["bill_length_mm", "flipper_length_mm"]
model = KNeighborsClassifier(n_neighbors=5)
model.fit(penguins[columns], penguins["species"])
mysteries = pd.DataFrame([[39.0, 190.0], [48.0, 195.0], [80.0, 350.0], [20.0, 120.0]], columns=columns)
guesses = model.predict(mysteries)
chances = model.predict_proba(mysteries)
for i in range(len(guesses)):
print(str(list(mysteries.iloc[i])) + ": " + guesses[i] + ", " + str(round(max(chances[i]) * 100)) + "% sure")
What the 100% means.
What a model knows about the world.
Use kneighbors to find how far each mystery is from its nearest penguin, and say "not like any penguin in the data" instead of a guess when it is too far.
distances, places = model.kneighbors(mysteries) gives each mystery's distances to its nearest penguins.
If distances[i][0] (the nearest) is more than 10, print that it is not like any penguin in the data.
Test fairly how well bill and flipper length predict species, sex and island. Which is hardest, and why might that be?
pd.read_csv(...).dropna(), then loop over ["species", "sex", "island"] as y, with a fair test for each.
Say "not like any penguin in the data" when a mystery is far from every penguin.
Run your program. This ticks when your code does all of the above.
Test fairly how well bill and flipper length predict species, sex and island.
Run your program. This ticks when your code does all of the above.
Write a model card for your app from the last lesson (in a comment at the top of the program, or on paper): what it predicts and from which measurements, where the data came from, its accuracy on a fair test, and three situations where it should not be trusted.
You have built models by hand, used a real library and put a model in an app. The same ideas (examples, fair tests, honest limits) are behind every machine-learning system, from Teachable Machine to large language models.
An optional lesson. Students find where the penguin model fails (an unseen species, impossible measurements, confident wrong answers), add a "not like any penguin" check with kneighbors, test harder questions (sex, island), and write a model card. About 45 minutes.
Open the teacher guide: answers, solutions, slides and a worksheet.
Optional: not needed for completing the course. A classifier can only choose between the classes in its training data, so an emperor penguin (not in the data) is called Gentoo; impossible measurements still get a 100%-sure answer, because the vote share compares neighbours with each other and ignores how far away they are. model.kneighbors(...) returns the distances, which the first Modify uses to refuse far-away inputs. A fair test shows species is easiest (88% here with this split), sex harder (73%) and island hardest (64%) from bill and flipper length alone. The data comes from three islands near Palmer Station, 2007 to 2009; a model knows only that world. The info question moves to a human context (a school using past records): be ready for discussion about fairness and human review. A model card is a short honest description used in industry.
Separating confidence from correctness. Ask students to argue for and against trusting a 100% answer before they code the check.
Stretch: Find the island model's mistakes and explain them (some species live on more than one island).
Support: Give students the kneighbors line, and ask them to write the if.
The micro:bit and the computer course trains Teachable Machine models; the same limits apply. Discuss large language models with the same three questions: what data, how tested, what it cannot do.