Hold back some penguins to test on: a model tested on the data it learned from looks perfect, and only new data shows how good it really is.
If you saw the exam questions and answers beforehand, a perfect score would prove nothing. Models are the same.
How good is the nearest-neighbour model? To find out, it must be tested on penguins it has not seen. So the penguins are split: a training set the model learns from (its examples), and a test set kept back to check it. penguins[:250] is the first 250; penguins[250:] is the rest. The data file is in order of species, so it is shuffled first (random.seed(1) makes the shuffle the same every time).
Make your prediction first. Help opens once you've locked it in.
penguins = ["Adelie", "Adelie", "Chinstrap", "Gentoo", "Gentoo"]
train = penguins[:3]
test = penguins[3:]
print(test)
[:3] is the first three items; [3:] is from place 3 to the end. What is in the test set, and is it a fair test?
Run it (it takes a moment: every test penguin is compared with all 250 training penguins). Compare the two scores. Then change random.seed(1) to another number and run again.
import csv
import random
penguins = []
with open("penguins.csv") as f:
for row in csv.DictReader(f):
penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
random.seed(1)
random.shuffle(penguins)
train = penguins[:250]
test = penguins[250:]
def distance(a, b):
return ((a["bill"] - b["bill"]) ** 2 + (a["flipper"] - b["flipper"]) ** 2) ** 0.5
def nearest(new, known):
best = known[0]
for p in known:
if distance(p, new) < distance(best, new):
best = p
return best
def accuracy(group, known):
correct = 0
for p in group:
if nearest(p, known)["species"] == p["species"]:
correct = correct + 1
return round(100 * correct / len(group))
print("Tested on the penguins it learned from: " + str(accuracy(train, train)) + "%")
print("Tested on new penguins: " + str(accuracy(test, train)) + "%")
import csv
import random
penguins = []
with open("penguins.csv") as f:
for row in csv.DictReader(f):
penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
random.seed(1)
random.shuffle(penguins)
train = penguins[:250]
test = penguins[250:]
def distance(a, b):
return ((a["bill"] - b["bill"]) ** 2 + (a["flipper"] - b["flipper"]) ** 2) ** 0.5
def nearest(new, known):
best = known[0]
for p in known:
if distance(p, new) < distance(best, new):
best = p
return best
def accuracy(group, known):
correct = 0
for p in group:
if nearest(p, known)["species"] == p["species"]:
correct = correct + 1
return round(100 * correct / len(group))
print("Tested on the penguins it learned from: " + str(accuracy(train, train)) + "%")
print("Tested on new penguins: " + str(accuracy(test, train)) + "%")
Why it means nothing.
A fair mix.
A test the model has already seen.
With this split the model scores 100% on the test set. Click the line that is wrong.
Train on only 30 penguins. What happens to the test accuracy? Why?
train = penguins[:30], and keep test = penguins[250:] so the test is the same.
Show how many of each species in the test set were guessed right. Which species is hardest, and why might that be?
Two dictionaries, right and total, keyed by species. For each test penguin, add 1 to total; if the guess is right, add 1 to right too.
Start a species at 0 the first time it turns up, as in Data detective's counting.
Train on only 30 penguins and compare the test accuracy.
Run your program. This ticks when your code does all of the above.
Show the test results for each species separately.
Run your program. This ticks when your code does all of the above.
Bring back the rules from Lesson 1 as guess(bill, flipper) and test them on the same test penguins as the nearest-neighbour model. Which wins? Is the difference big enough to matter?
Test Lesson 1's rules and the nearest-neighbour model on the same test penguins, and print both scores.
Run your program. This ticks when your code does all of the above.
Next: the same model, the same split and the same test in a few lines of scikit-learn, the library real data scientists use.
Students split the penguins into training and test sets after shuffling, see nearest neighbour score 100% on its own training data and about 95% on new penguins, then train on less data, break results down by species and compare the Lesson 1 rules fairly. About 45 minutes.
Open the teacher guide: answers, solutions, slides and a worksheet.
Read the unit primer first. The data file is in species order, so random.shuffle (with random.seed(1) for repeatable results) comes before the split; the Predict shows the problem with an unshuffled list. Slices: penguins[:250] is the first 250, penguins[250:] the other 92. Tested on its training penguins, nearest neighbour scores 100% (each penguin is its own nearest match, at distance 0); on the test penguins it scores 95%. The bug task tests on part of the training set. Lesson 1's rules score 93% on the same test set, nearly as good: simple models can be strong, and a fair test is what shows it. The per-species task shows Chinstrap is hardest (fewer examples, and similar to Adelie).
Accepting that 100% is a bad sign. Ask why each training penguin finds itself, and let them explain it to a partner.
Stretch: Run the test with five different seeds and average the scores (a simple form of cross-validation).
Support: Give students the right and total dictionaries started, and ask them to add 1 in the right places.
Next lesson: scikit-learn's train_test_split, fit and score do all of this in a few lines.