---
marp: true
theme: default
html: true
---
# A fair test
Machine learning with Gradio · Models by hand
---
## Today
**We are learning to** hold back some penguins to test on: a model tested on the data it learned from looks perfect, and only new data shows how good it really is.
- I can explain why a model must be tested on data it has not seen.
- I can split data into training and test sets, shuffling first.
- I can compare two models fairly on the same test set.
---
## Why it matters
> If you saw the exam questions and answers beforehand, a perfect score would prove nothing. Models are the same.
---
💬 Discussion
## Predict
```python
penguins = ["Adelie", "Adelie", "Chinstrap", "Gentoo", "Gentoo"]
train = penguins[:3]
test = penguins[3:]
print(test)
```
**What does it print?**
- ['Adelie', 'Adelie', 'Chinstrap']
- ['Chinstrap', 'Gentoo', 'Gentoo']
- An error
- ['Gentoo', 'Gentoo']
---
✅ Check
## What happened?
* ['Gentoo', 'Gentoo']. [3:] is from place 3 to the end. The test set is all Gentoo, because the list was in order of species: a test with no Adelies tests nothing about them.
---
✍️ Activity
## Run it
Open the lesson at this step and try it together.
[studio.learningwaypoints.com/tutorials/code/ml/ml-testing/?full=step-2#step-2](https://studio.learningwaypoints.com/tutorials/code/ml/ml-testing/?full=step-2#step-2)
---
💬 Discussion
## Investigate
_The program is 37 lines long: open it in the tutorial and read it together._
[Open this step](https://studio.learningwaypoints.com/tutorials/code/ml/ml-testing/?full=step-3#step-3)
---
✅ Check
## The perfect score
**Why is the score 100% on the training penguins?**
- Each penguin's nearest match is itself, at distance 0, so it always copies its own species
- The model was trained twice
- The training penguins are easier
- It is a lucky shuffle
* Answer: Each penguin's nearest match is itself, at distance 0, so it always copies its own species
---
✅ Check
## Why shuffle
**Why shuffle before splitting?**
- To hide the answers
- Shuffling makes the model learn faster
- The file is in order of species, so without a shuffle the test set would be nearly all one species
- Python needs it
* Answer: The file is in order of species, so without a shuffle the test set would be nearly all one species
---
✅ Check
## Spot the bug
**With this split the model scores 100% on the test set. Click the line that is wrong.**
```
1 import csv
2
3 penguins = []
4 with open("penguins.csv") as f:
5 for row in csv.DictReader(f):
6 penguins.append({"species": row["species"], "bill": float(row["bill_length_mm"]), "flipper": float(row["flipper_length_mm"])})
7
8 train = penguins[:250]
9 test = penguins[:92]
10 print(len(train), len(test))
```
* Answer: line 9
---
✍️ Activity
## Modify
- **Less to learn from:** Train on only **30** penguins. What happens to the test accuracy? Why?
- **Each species:** Show how many of each species in the test set were guessed right. Which species is hardest, and why might that be?
[Open this step](https://studio.learningwaypoints.com/tutorials/code/ml/ml-testing/?full=step-4#step-4)
---
✍️ Activity
## Make: rules or neighbours?
Bring back the rules from Lesson 1 as `guess(bill, flipper)` and test them on the **same** test penguins as the nearest-neighbour model. Which wins? Is the difference big enough to matter?
[Open this step](https://studio.learningwaypoints.com/tutorials/code/ml/ml-testing/?full=step-5#step-5)
---
## Where this leads
Next: the same model, the same split and the same test in a few lines of scikit-learn, the library real data scientists use.
---
✅ Check
## Exit ticket
**Why does a model need a test set it has never seen?**
* On data it learned from it can just remember the answers; only new data shows how well it will do on penguins it meets later.