Students split the penguins into training and test sets after shuffling, see nearest neighbour score 100% on its own training data and about 95% on new penguins, then train on less data, break results down by species and compare the Lesson 1 rules fairly.
Load these into Waypoints Prepare: choose Load URL and paste the copied address, or download the file and import it. In Prepare, the worksheet's Teacher edition shows the answers; the student copy hides them.
Predict and Run 10 min · Investigate 10 min · Modify 15 min · Make 10 min
Read the unit primer first. The data file is in species order, so random.shuffle (with random.seed(1) for repeatable results) comes before the split; the Predict shows the problem with an unshuffled list. Slices: penguins[:250] is the first 250, penguins[250:] the other 92. Tested on its training penguins, nearest neighbour scores 100% (each penguin is its own nearest match, at distance 0); on the test penguins it scores 95%. The bug task tests on part of the training set. Lesson 1's rules score 93% on the same test set, nearly as good: simple models can be strong, and a fair test is what shows it. The per-species task shows Chinstrap is hardest (fewer examples, and similar to Adelie).
Accepting that 100% is a bad sign. Ask why each training penguin finds itself, and let them explain it to a partner.
| What you will see | A question to ask | The fix |
|---|---|---|
| The test accuracy is 100%. | Is any test penguin also in the training set? | test = penguins[250:], with no overlap. |
| KeyError: 'Chinstrap' in the per-species task. | Was the species started at 0 before adding 1? | if s not in total: total[s] = 0 and right[s] = 0. |
| The program takes a few seconds. | How many distances does it work out? | Nothing is wrong; 92 test penguins times 250 training penguins is 23,000 distances. |
One possible solution for each task. Other correct programs are fine.
import csv
import random
penguins = []
with open("penguins.csv") as f:
for row in csv.DictReader(f):
penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
random.seed(1)
random.shuffle(penguins)
train = penguins[:30]
test = penguins[250:]
def distance(a, b):
return ((a["bill"] - b["bill"]) ** 2 + (a["flipper"] - b["flipper"]) ** 2) ** 0.5
def nearest(new, known):
best = known[0]
for p in known:
if distance(p, new) < distance(best, new):
best = p
return best
def accuracy(group, known):
correct = 0
for p in group:
if nearest(p, known)["species"] == p["species"]:
correct = correct + 1
return round(100 * correct / len(group))
print("Tested on the penguins it learned from: " + str(accuracy(train, train)) + "%")
print("Tested on new penguins: " + str(accuracy(test, train)) + "%")
import csv
import random
penguins = []
with open("penguins.csv") as f:
for row in csv.DictReader(f):
penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
random.seed(1)
random.shuffle(penguins)
train = penguins[:250]
test = penguins[250:]
def distance(a, b):
return ((a["bill"] - b["bill"]) ** 2 + (a["flipper"] - b["flipper"]) ** 2) ** 0.5
def nearest(new, known):
best = known[0]
for p in known:
if distance(p, new) < distance(best, new):
best = p
return best
def accuracy(group, known):
correct = 0
for p in group:
if nearest(p, known)["species"] == p["species"]:
correct = correct + 1
return round(100 * correct / len(group))
print("Tested on the penguins it learned from: " + str(accuracy(train, train)) + "%")
print("Tested on new penguins: " + str(accuracy(test, train)) + "%")
right = {}
total = {}
for p in test:
s = p["species"]
if s not in total:
total[s] = 0
right[s] = 0
total[s] = total[s] + 1
if nearest(p, train)["species"] == s:
right[s] = right[s] + 1
for s in total:
print(s + ": " + str(right[s]) + " of " + str(total[s]))
import csv
import random
penguins = []
with open("penguins.csv") as f:
for row in csv.DictReader(f):
penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
random.seed(1)
random.shuffle(penguins)
train = penguins[:250]
test = penguins[250:]
def distance(a, b):
return ((a["bill"] - b["bill"]) ** 2 + (a["flipper"] - b["flipper"]) ** 2) ** 0.5
def nearest(new, known):
best = known[0]
for p in known:
if distance(p, new) < distance(best, new):
best = p
return best
def accuracy(group, known):
correct = 0
for p in group:
if nearest(p, known)["species"] == p["species"]:
correct = correct + 1
return round(100 * correct / len(group))
def guess(bill, flipper):
if flipper > 206:
return "Gentoo"
if bill > 44:
return "Chinstrap"
return "Adelie"
rule_right = 0
for p in test:
if guess(p["bill"], p["flipper"]) == p["species"]:
rule_right = rule_right + 1
print("Rules: " + str(round(100 * rule_right / len(test))) + "%")
print("Nearest neighbour: " + str(accuracy(test, train)) + "%")
Stretch: Run the test with five different seeds and average the scores (a simple form of cross-validation).
Support: Give students the right and total dictionaries started, and ask them to add 1 in the right places.
Why does a model need a test set it has never seen?
On data it learned from it can just remember the answers; only new data shows how well it will do on penguins it meets later.
Next lesson: scikit-learn's train_test_split, fit and score do all of this in a few lines.