Real data is messy. Write a function that cleans one value, check it with test cases, and skip the rows it cannot fix.
Data people type is never tidy: blanks, typos, numbers in words. Cleaning it is most of a data scientist's job, and test cases prove the cleaning works.
This lesson uses survey.csv, a made-up class survey: each student's favourite genre and hours of music a week. Open it in the Files tab. Some answers are blank, some are in words, some are impossible. Your job: clean what you can, skip what you cannot, and prove the cleaning works.
Make your prediction first. Help opens once you've locked it in.
answers = ["5", "", "three", "4.5"]
total = 0
for a in answers:
try:
total = total + float(a)
except ValueError:
print("Skipped: '" + a + "'")
print(total)
float() turns text into a decimal number, and causes a ValueError when it cannot. try: runs a line that might fail; if it causes a ValueError, Python runs the except ValueError: lines instead of stopping. Which of the four answers can float() turn into a number?
Run it. First the test cases run (assert lines), then the survey is cleaned. clean_hours gives back a number, or None (Python's word for "no value") when the answer cannot be used. hours is None asks whether it is None.
Then break a test on purpose: change 168 to 16 and run it again. Put it back.
import csv
def clean_hours(text):
try:
hours = float(text)
except ValueError:
return None
if hours < 0 or hours > 168:
return None
return hours
assert clean_hours("5") == 5
assert clean_hours("4.5") == 4.5
assert clean_hours("three") is None
assert clean_hours("") is None
assert clean_hours("-1") is None
print("All tests pass")
with open("survey.csv") as f:
rows = list(csv.DictReader(f))
total = 0
count = 0
for row in rows:
hours = clean_hours(row["hours"])
if hours is None:
print("Skipped " + row["student"] + ": '" + row["hours"] + "'")
else:
total = total + hours
count = count + 1
print("Average: " + str(round(total / count, 1)) + " hours of music a week, from " + str(count) + " answers")
import csv
def clean_hours(text):
try:
hours = float(text)
except ValueError:
return None
if hours < 0 or hours > 168:
return None
return hours
assert clean_hours("5") == 5
assert clean_hours("4.5") == 4.5
assert clean_hours("three") is None
assert clean_hours("") is None
assert clean_hours("-1") is None
print("All tests pass")
with open("survey.csv") as f:
rows = list(csv.DictReader(f))
total = 0
count = 0
for row in rows:
hours = clean_hours(row["hours"])
if hours is None:
print("Skipped " + row["student"] + ": '" + row["hours"] + "'")
else:
total = total + hours
count = count + 1
print("Average: " + str(round(total / count, 1)) + " hours of music a week, from " + str(count) + " answers")
Not a number, or not possible.
A test that is silent when it passes.
A test catches a mistake.
The tests stop with AssertionError on the line that checks "400". Click the line that is wrong.
" hip hop ", "POP" and "pop " should all count as the same genre. Write clean_genre(text) with two test cases, and print each student's cleaned genre.
def clean_genre(text): return text.strip().lower()
Test it: assert clean_genre(" Hip Hop ") == "hip hop". Then print each student's cleaned genre.
One student's answer is in the file twice. Skip an answer from a student who has already been counted.
Before the loop, seen = []. Skip a row if row["student"] in seen.
When a row is used, seen.append(row["student"]).
Write clean_genre(text), which removes spaces at the ends and makes the genre lower case, with two test cases.
Run your program. This ticks when your code does all of the above.
Skip a student's answer if it has already been counted.
Run your program. This ticks when your code does all of the above.
Draw a bar chart of the class's favourite genres, as in Lesson 3. Use clean_genre so the same genre is counted once, and skip blank answers. Then add a test case of your own.
Chart the class's favourite genres, cleaned, with blank answers skipped.
Run your program. This ticks when your code does all of the above.
Test cases are how programmers trust code: every app you use has thousands of them. Next: put the music data behind an app.
Students clean a messy survey with a function that returns a number or None, check it with assert test cases, then tidy genres, skip repeated rows and chart the cleaned genres. About 50 minutes.
Open the teacher guide: answers, solutions, slides and a worksheet.
Read the unit primer first. survey.csv is made up (no real students) with deliberate mess: capitals and spaces in genres, a blank genre, a blank hours answer, "three", -1, 400 (a week has 168 hours) and a repeated row. clean_hours returns a number, or None ("no value") when the answer cannot be used; hours is None checks for it (is is the usual way to compare with None). try / except ValueError is introduced here in one sentence (the Predict): try a line, and if it causes that error, run the except lines instead of stopping. Deeper Python's errors lesson goes further. Test cases are assert lines: silent when true, and the program stops with AssertionError when false. The bug task joins the two range checks with and, which can never be true; the tests catch it. Talk about honesty: a report says how many answers were used.
Writing a test case before the fix. When a student is stuck, ask what the function should give for the failing input, and have them write that assert first.
Stretch: Write clean_year(text) for the year_level column (only 7 to 12 allowed) with test cases, and print the average hours for each year level.
Support: Give students clean_genre written out, and ask them to write the two test cases for it.
Next lesson puts the data behind a Gradio app with pandas, which has its own cleaning tools. Test cases come back in every later course.