Is this car any good?
You have just been handed a car that steers itself. It rolls along a two-lane road at 8 m/s, about 29 km/h, and nobody is touching the wheel. Before you change a single line of its code, you need an answer to a plain question: is it any good?
"Good" needs a number. By the end of this lesson you will take a whole drive, nearly two thousand moments long, and boil it down to three numbers, and one count, that say how well the car kept its lane. You will also know which of the three to trust for which question, and which tempting number to never trust at all.
Here is where you are heading. This is the report card for the car you were handed, after one lap of a test road:

Every number on that card comes from a recording of the drive, and in the lab you compute each one yourself. The rest of this lesson rebuilds the card, one piece at a time.
Optional: how we got 29 km/h
A speed in metres per second says how many metres you cover in one second. To say it in kilometres per hour, count kilometres instead of metres (1 km is 1000 m, so divide by 1000) and count in hours instead of seconds (1 h is 3600 s, so multiply by 3600):
That is about 29 km/h.
The car you were handed
Twenty times a second, once every 0.05 seconds, the car runs the same loop:
- The camera takes a picture of the road ahead.
- The perception box looks at the picture and reports where the lane is.
- The controller box turns that report into a steering angle.
- The car moves a little, and the camera takes the next picture.

In this lesson you open neither box. You instrument the car instead: you record what it did and judge the result, the way a driving examiner judges a driver without reading their mind.
Approximation. The car lives in a flat-world teaching simulator. The road is paint on a perfectly flat plane, and the car moves like a bicycle (a simple model that treats each pair of wheels as one wheel) with a 2.7 m wheelbase, the distance from its front axle to its rear axle. Real cars roll, slip and bounce. Here none of that happens, on purpose, so that every wobble you see has one cause.
Placeholder. Both boxes inside the provided car are stand-ins. You build your own perception box in Your first perception box and your own controller in Closing the loop.
Here is what the camera sees, and the path the car drove:

The car is tested on three roads. straight is 200 m of straight road. gentle is a stadium-shaped loop: two 100 m straights joined by two wide bends of radius 80 m. curvy is a 30 m straight followed by six tight quarter-circle bends of radius 20 m, alternating left and right.
Guess first: which road do you expect the provided car to find hardest, and why?

Answer: curvy. Its bends are a quarter of the radius of gentle's, so the car must turn four times as sharply, and each bend turns the other way from the last. The provided car does not finish it. You will see exactly where it goes wrong near the end of this lesson.
Optional: how we got 702.7 m and 218.5 m
Each length is the distance along the road, measured along the yellow line, from the start of the route to its end, added up piece by piece. A circle of radius has circumference , where is the fixed ratio of any circle's circumference to its diameter. So a half circle is long and a quarter circle is .
gentle is two straights of 100 m and two half-circle bends of radius m:
curvy is a 30 m straight and six quarter-circle bends of radius 20 m:
The paint of an open road runs 40 m past the end of the route, so the camera still has lines to look at as the car finishes. Those 40 m are not part of the route, which is why the figure marks the route's end with a square.
Lateral error
Watch the car from above. The lane is 3.6 m wide, with a dashed yellow line on its left and a solid white line on its right. At some moments the car sits dead centre; at others it has drifted left or right.
At each moment you want to know one thing: how far sideways is the car from the middle of its lane? Measure it at the front axle, the line between the two front wheels, because that is where the simulator mounts the camera.
Try a number. If the front axle sits half a metre to the left of the lane centre, the answer is "0.5 m, to the left". Saying "to the left" every time gets tedious, so give left a plus sign and right a minus sign: +0.5 m. A car 0.3 m to the right of centre scores −0.3 m. A car exactly on the lane centre scores 0.
Call this number , read simply "e", for error. Its name is the lateral error; lateral means sideways.
The lane is 3.6 m wide, so each painted line is 1.8 m from the middle: the yellow line at m and the white line at m. When the front axle reaches a line, the simulator ends the run.

Left is positive everywhere in this course: for offsets, for headings (which way the car points), for steering angles. Almost every sign bug you will write comes from forgetting it.
Sometimes you care how far off the car is, not which side. Then you drop the sign and write , read "the size of e". A car at −0.3 m and a car at +0.3 m both have m, and the run ends when reaches 1.8 m.
Optional: who else measures this
You are not the first to pick this number. Stanford's robot car Stanley, which won the 2005 DARPA Grand Challenge desert race, steered by exactly this measurement:
"The key error metric is the cross-track error, x(t), […] which measures the lateral distance of the center of the vehicle's front wheels from the nearest point on the trajectory." (Thrun et al., 2006, p. 684)
They write it x(t), read "x of t". Cross-track error is Stanley's name for what this course calls .
A drive is a list of numbers
The simulator measures at every step, 20 times a second. One lap of gentle takes the provided car 88.85 s, so the whole lap is a list of 1,778 numbers, written , where is how many there are and the small number under each says which step it came from.
A list of measurements taken one after another in time is called a time series. In the lab the recording arrives as an object called tel, short for telemetry, the record of everything the car measured on its drive. Its lateral errors are tel.lat, one number per step.
Guess first: gentle turns left twice. While the car is going round a left-hand bend, do you expect to be positive (car left of centre) or negative (car right of centre)?

Answer: positive. In each bend the car drifts toward the inside of the turn, and the inside of a left-hand bend is on the left. On the straights stays within a few millimetres of 0: its lowest value on the whole lap is one millimetre right of centre, too small to see in the figure. Through both bends the car is left of centre the whole way.
Optional: how we got N = 1778
The simulator records one value at the start of every step, beginning with step at time . Step is recorded at time seconds, and the lap ends at the step where the car has driven the whole route, at s. That last step is
Counting the steps gives values. The first value, , is the one recorded at step . There is one more value than there are 0.05 s gaps between them, just as a fence with 1777 gaps has 1778 posts.
You may notice that m is more than the 702.7 m of the road. That is right: the car drives in the right-hand lane, which is the outside lane of gentle's left-hand bends, so its own path is longer than the road. Measured on the recording, the path is 8.1 m longer: about 11 m extra from driving on the outside, less about 3 m given back when the car drifts toward the inside of each bend.
The tempting shortcut
You have 1,778 numbers and you want one. The obvious move is to average them. A small average sounds like a car that stays near the middle.
Try it on a short drive first. Here are five lateral errors, in metres:
Guess first: this car was off centre at every one of its five steps. What do you think its average is?

Answer: zero. Add them up and divide by how many there are:
A perfect score, for a car that was never once on the lane centre. The errors to the left cancelled the errors to the right.
This happens on real drives too, not only on made-up ones. Take two cars that differ from the provided one in a single way: when they are off centre, they push back toward the lane centre harder. The 2× car pushes back twice as hard as the provided car, and the 4× car four times as hard. (How a steering setting like that works is the subject of Closing the loop; here they are simply two more cars.) Both drive the whole of curvy without leaving the lane. The provided car cannot join this comparison: it runs out of lane on curvy, as you will see below.
Their plain averages are +0.023 m for the 2× car and +0.024 m for the 4× car. Rounded to two decimal places, both are +0.02 m.
Guess first: the plain average cannot tell these two cars apart. Did they drive about equally well?

Answer: no. The 2× car swings from 0.72 m left of the lane centre to 0.82 m right of it, while the 4× car never goes more than 0.35 m from it. The 2× car's swings to the left and to the right cancel almost exactly, so its average lands right next to the 4× car's. On a fairer measure, which you meet next, the 2× car is 0.50 m from the lane centre on average and the 4× car 0.09 m. One car is nearly six times further off than the other, and the plain average shows no difference at all.
The plain average does answer a question, just a different one: on average, which side does the car favour? That is called the car's bias. It is worth knowing, but it is not a score, and it must never be used as one.
Three honest numbers
The fix is to stop errors cancelling. Count every error as a cost, whichever side of the line it fell on. There are three good ways to do that, and each answers its own question.
Mean absolute error: how far off, typically?
Drop the signs first, then average. On the five errors:
They add up to , and . So this car sat 0.28 m from centre, typically.

Written out in full for these five steps, the calculation is
For a lap of 1,778 steps you would never write out every term, so mathematicians use a shorthand:
Read it aloud as "one over N, times the sum, for k from 1 to N, of the size of e-k". The tall symbol is the Greek capital letter sigma, and it means "add up what follows", here for every step from to .
Confusion is normal here. Sigma is only shorthand for a long sum. If it slows you down, read it as the written-out sum above; it means exactly that.
This number is the mean absolute error, written mean. Absolute is the mathematician's word for "with the sign dropped".
Maximum absolute error: how bad was the worst moment?
Sometimes the typical moment does not matter. What matters is the single worst one, because that is when a wheel might touch the line. Take the largest of the sizes: for the five errors, m. The shorthand is
read "the largest, over all steps k, of the size of e-k". Its name is the maximum absolute error.
Root mean square: how far off, counting big errors extra?
The third number squares each error before averaging, which turns every error into a positive cost and makes the big ones count for more. On the five errors:
- Square each one: .
- Take the mean of the squares: they add to , and .
- Take the square root: m.
There is a picture behind those three steps. Squaring a length gives the area of a square with that side.

Squares of metres are square metres, and the square root brings you back to metres. That is why this number can sit beside mean on the same scale.
The shorthand reads from the inside out, the same order you just worked in: square, take the mean, take the root.
Its name is the steps read backwards: root mean square, or RMS.
Now look back at the report card at the top of this lesson. For the provided car's lap of gentle, mean is 0.38 m, RMS is 0.44 m and max is 0.53 m, and you can say what each of them means.
Why RMS punishes big errors
Here are two five-step drives:
- Drive A: . Always a little off.
- Drive B: . Perfect, then one big swerve.
Both have the same mean: m.
Guess first: do they have the same RMS? If not, which is bigger, and by roughly how much?

Answer: Drive B's RMS is more than twice Drive A's.
- A: every square is , their mean is , and m.
- B: the squares are , their mean is , and m.
The swerve of 1.0 m is 5 times the steady 0.2 m, but its square is 25 times as big. Squaring grows big numbers much faster than small ones, so one large error dominates the average of the squares.
That matches driving. Being 0.2 m off centre all the time never brings a wheel near a line. Being 1.0 m off for a moment might. When the worst moments matter most, look at RMS and max, not only at mean.
One more fact you can check on every example here: RMS is never smaller than mean (0.303 ≥ 0.28, and 0.447 ≥ 0.2). The two are equal only when every is the same, as in Drive A. The more uneven the errors, the wider the gap.
Counting steps off the lane
Some questions have a yes-or-no answer at each step. "Was the car more than 0.9 m from centre just now?" The course uses 0.9 m, half the 1.8 m from the middle to a line, as its off-lane limit. Why 0.9 m and not 1.8 m? is measured at the front axle, a single point, and a typical real car is about 1.8 m wide (an approximation), so with its axle 0.9 m out a wheel is already near the paint. The simulator's car is a point with no width, but the limit is set as if it had one. A step is off the lane when
The comparison is strict: a step at exactly 0.9 m is on the limit, not over it.

To count off-lane steps, ask the question at every step and count the yeses. With the five errors and a stricter limit of 0.35 m, the sizes give no, no, no, yes, yes: 2 steps. To turn a count of steps into time, divide by 20 steps per second.
Guess first: the provided car leaves curvy 48 m into the 218.5 m route. Before it went off, how long had it spent more than 0.9 m from centre: a blink, or whole seconds?

Answer: whole seconds. The car spends 40 steps more than 0.9 m from centre, which is seconds, before it reaches the line at 48 m. All 40 are in the first bend, over the last 15 m before the car leaves, from 33 m to 48 m.
What does doing nothing score?
Is 0.38 m good? A number means little until you have something to set it against. So score the laziest car imaginable: one that holds the wheel dead straight and never steers at all. Call it the do-nothing car.
Guess first: on which of the three roads, if any, does the do-nothing car get a perfect score?

Answer: straight. A car that starts in the middle of a straight lane and never turns stays in the middle, so all three of its numbers are 0.00 m, the same as a perfect driver's. That is why one road is never enough: on straight, a car with no steering at all is indistinguishable from a flawless one. On gentle the do-nothing car is gone 14.7 s into the lap, 117 m along the road, at the first bend.
Its mean on gentle comes out at only 0.10 m, lower than the provided car's 0.38 m. That is not a better drive. It is an average over the 117 m the car managed before its run ended, and most of that was straight road. A number from an unfinished run cannot be set against a full lap. Compare two cars' numbers only when both drove the same road to the end.
Now 0.38 m has its place. A perfect car scores 0; a car that never steers crosses the painted line at the first bend. The provided car finishes the lap, but it uses up to 0.53 m of its 1.8 m of room to do it. That is the bar your own controller has to beat in Closing the loop.
Arrays, not loops
In the lab, tel.lat is a NumPy array (NumPy is Python's standard library for arrays, imported as np): a list of numbers that you can do arithmetic on all at once. You could compute mean with a Python loop:
total = 0.0
for x in e:
total += abs(x)
mean_abs = total / len(e)
NumPy lets you write the formula instead:
mean_abs = np.abs(e).mean()

np.abs(e) makes a new array holding the size of every element, and .mean() adds them up and divides by how many there are. In the same way, e ** 2 squares every element, np.sqrt takes a square root, .max() finds the largest element, and a comparison such as np.abs(e) > 0.9 asks its question of every element and returns an array of True and False. Python counts True as 1 and False as 0, so .sum() of that array is the number of off-lane steps. Applying one operation to a whole array at once is called vectorisation.
Vectorised code is shorter, it reads like the maths, and it runs much faster, because the loop happens inside fast code written in C instead of in Python, one element at a time. You will time the difference in the lab. The grader checks your answers against the plain-loop calculation, so the two styles must agree.
Recap
- A drive is a time series of lateral errors , one every 0.05 s, with left of centre positive.
- The plain average of lets left and right cancel. It measures the car's bias, never its quality.
- mean says how far off the car typically was, RMS counts big errors extra, max is the worst moment, and counting steps with m says how long the car was off the lane.
Next question: twenty times a second, the camera hands the perception box a picture. What is a picture to a computer, and how could you find a yellow line in one? That is the next lesson, Images are arrays.
