A friend of mine has recently been struggling to pass her driving tests. That’s right, tests: she’s currently on attempt number 6. According to her instructor and practice partners, she’s a good driver and each time she has failed, it has been for a different reason. Sometimes it just seems to be bad luck. In one of her attempts she had to back around a corner and there in the road was a horse and (but sadly not on) a mobility scooter. Sometimes the reason has been ambiguity about how much ‘hesitation’ or ‘assertiveness’ is too much or too little. But after repeated failures, my friend can’t help but wonder whether the issue is genuinely a problem with her driving, or some bias in the test centre or against female drivers generally.
Fortunately the government provides statistics that allow us to test these latter hypotheses and I’ve downloaded the data for 2013-14 (the last year with complete records) to try and understand my friend’s situation.
The simplest model for this sort of situation is a binomial distribution so that whether or not you fail is simply a random event with a fixed probability. In a fair system, the same random process would apply nationwide and so you could model the situation with a funnel plot, as shown in the plot below.
The data also allow us to break down the results by gender and in the plot below you can see that the average pass rate for men is higher than that for women. It’s worth noting that of the 340 test centres in the database, only 12 have pass rates that are higher for women than for men. For interest, I’ve also highlighted the test center where my friend has taken her tests.
By inspection it’s pretty clear that a basic binomial model is not a good representation of what’s going on here. Clearly the more tests a centre conducts, the lower its pass rate. And while it’s hard to determine causality, the most likely explanation is that in more difficult test centres, the pass rate is lower so people have to retake their test increasing the number of tests conducted.
We can improve on the simple binomial model then by adding a predictor for the number of tests conducted and, since there is clearly a difference by gender, I’ve used a multi-level model formulation that allows for the intercept to vary by gender. The results show that when considering an average test center:
- the pass rate changes by approximately -1.6% for every additional thousand tests conducted, and
- the probability of passing is 46% for females and 53% for males.
So for a female taking her test at the center in question, you would expect a 43% chance of passing purely at random. This means that there is a 6% chance of having no passes in 5 attempts. It’s a small consolation but this isn’t much worse than the national average. Females do have lower pass rates than males though that can’t be attributed to discrimination from this data alone.
My only advice is (a) try a smaller test centre (there are several options in her city) and (b) keep trying. After 10 attempts, she can be 99.6% sure of passing.