Train a recycling sorter

Classroom resources

Load these into Waypoints Prepare: choose Load URL and paste the copied address, or download the file and import it. In Prepare, the worksheet's Teacher edition shows the answers; the student copy hides them.

• Slides Download Copy address for Prepare Present from the browser, or export to PowerPoint or PDF.
• Worksheet Download Copy address for Prepare Print the student copy; the Teacher edition has the answers.

Students will be able to

• I can train an image model and test it on photos it has never seen.
• I can work out accuracy, overall and for each group of test photos.
• I can find what a model's mistakes share and fix it by changing its training data.

Timing

Predict 3 min · Train and test 12 min · Read the results 10 min · Fix the data 15 min · Diagnose 4 min · Make and Evaluate 6 min

Before class

The idea: a model learns only from the examples it is given. It works on new inputs that look like its training photos and fails on ones that don't. The fix is always in the data, not the settings.

How it works: students train in Teachable Machine (Google's free site, no account needed to train), export with Upload my model, and paste the link into the board. The board tests the model on 40 fixed photos it has never seen, in five groups: plain background, messy bench, low light, crumpled or at odd angles, and things that are neither paper nor plastic. Checks tick on the board's scores, not on the training.

Before class: on a student laptop on the school network, open Teachable Machine, train a tiny model, export it with Upload, and paste the link into the board. Bring a box of clean paper and plastic (and a few cans, jars or fruit) so every desk has variety.

Privacy: training happens in the browser and photos are not uploaded. Upload my model puts the trained model (not the photos) online, and anyone with the link can use it. Train on objects, not faces.

No webcam? The lesson links two folders of photos to upload instead: plain (30 per class on a white board: trains the biased model) and varied (messy benches, low light, odd angles and a Neither class: trains a good one). The photos come from TrashNet (MIT licence); the messy benches and low light are made from them.

The test photos are similar in style to the downloadable ones, so a model trained only on the varied folder can do well without the webcam. That is fine: the point is what changes the scores.

Running the lesson

1. Predict: take a show of hands. Most expect 'about the same'.
2. Train and test: give students 8 minutes to train. Plain is usually high and messy low. Ask: what is different about the messy photos?
3. Read the results: model the table live with one student's results, one line at a time. Then students do the low-light version themselves.
4. Fix the data: one change at a time. Students who add more photos of the same desk see no change: that is Sam's mistake, so point back to it.
5. Neither: ask what label a can gets, and how confident the model is.
6. Evaluate: collect a few answers about what a recycling centre should test.

The hard part (on purpose)

Accepting that a model trained on one desk fails on a messy bench. Ask what the test photos have that the training photos don't, before adding more data.

When students get stuck

Answers to the checks

1. Predict: Your model gets your own items right almost every time on the webcam. The board tests it on other paper and plastic photographed on a messy bench. Its score on the messy bench will be…
   Much lower. The model learned from photos on one plain background, so the background is part of what it learned. On a messy bench it is often wrong.
2. Train and test: Train two classes (Paper and Plastic) on one plain background, paste your link and test it.
   
3. Accuracy: A model gets 23 of 32 test photos right. What is its accuracy, as a percentage (to the nearest whole number)?
   72
   Accuracy is correct ÷ total: 23 ÷ 32 = 0.72, or 72%.
4. What the failures share: This model scores: plain 8/8, messy 4/8, low light 5/8, not paper or plastic 0/8. Its training photos were all on one plain desk. Why does it fail on the messy bench?
   It cannot tell the item from the background it was trained on
   Every training photo had the same plain desk, so the desk is part of what the model learned. On a cluttered bench it is guessing.
5. Your turn: Another model scores well everywhere except low light. Which fix is most likely to work?
   Add training photos taken in low light
   Look for what the failures share (low light), then add training photos that have it. Change the data, not the settings.
6. Sam's model: Sam's model scores 50% on the messy bench. Sam adds 100 more photos of the same items on the same desk. The messy score doesn't change. What went wrong?
   The new photos were like the old ones, so they added no variety
   More of the same data teaches the model nothing new. Sam needs photos on different backgrounds.
7. Backgrounds: Retrain with photos on at least three different backgrounds. Get the messy bench to 75% or more.
   
8. Unbalanced classes: You train Paper with 60 photos and Plastic with 10. Which way does the model lean when it is unsure?
   Towards Paper
   A class with many more examples wins most close calls. Give each class about the same number of photos.
9. Neither: The model labels a can or a bottle of glass as paper or plastic. Add a Neither class and get the 'not paper or plastic' group to 75% or more.
   
10. Midday photos: A model was trained on photos taken at midday by a window. It scores 100% plain, 88% messy, 25% low light. What is the problem, and the fix?
    No dim photos in training: add photos taken in low light
11. A confident can: A model scores 90% on paper and plastic but labels every can as Plastic with 95% confidence. What is the problem, and the fix?
    It has no class for other things: add a Neither class
12. Lopsided classes: Paper has 80 photos and Plastic has 15. The model calls most plastic items Paper. What is the problem, and the fix?
    The classes are unbalanced: give each class about the same number of photos
13. Make a sorter you would trust: Build a sorter that gets 80% or more overall and 75% or more in low light.
    

Solutions

One possible solution for each task. Other correct programs are fine.

Fix the data › Backgrounds

Open task: any program that does what the step asks.

Fix the data › Unbalanced classes

Open task: any program that does what the step asks.

Fix the data › Neither

Open task: any program that does what the step asks.

Make a sorter you would trust

Open task: any program that does what the step asks.

Stretch and support

Stretch: Add a fourth class (Cardboard or Glass). Which pairs of classes does the model mix up most, and why?

Support: Use the downloaded photos instead of the webcam: plain first (the biased model), then varied.

Exit ticket

A model scores 95% on its own desk and 50% in the canteen. What do you do?

Find what the canteen photos have that the training photos don't (background, light), and add training photos like that.

What each step is for

• Predict: Students expect a model that works on their own desk to have learned what paper is. Committing to that first makes the messy-bench result mean something.
• Train and test: Seeing the plain and messy scores side by side settles the prediction: the model works on photos like its training photos.
• Read the results: Accuracy by group shows where a model fails, not just how much; what the failures share points to what was missing from the training data.
• Fix the data: Each change to the training data, one at a time, shows its own effect on the scores.
• Diagnose: Naming the problem and the data fix without training again checks that the idea transfers.
• Make a sorter you would trust: Meeting a target on every group, not just overall, is what testing before trusting means.

Where it leads

Next: send the model's labels to a micro:bit that sorts with a servo flap.

Real world: automated recycling sorters, and any camera system tested on people or places it wasn't trained on (who might it fail for?).