The statistics of driving tests

A friend of mine has recently been struggling to pass her driving tests. That’s right, tests: she’s currently on attempt number 6. According to her instructor and practice partners, she’s a good driver and each time she has failed, it has been for a different reason. Sometimes it just seems to be bad luck. In one of her attempts she had to back around a corner and there in the road was a horse and (but sadly not on) a mobility scooter. Sometimes the reason has been ambiguity about how much ‘hesitation’ or ‘assertiveness’ is too much or too little. But after repeated failures, my friend can’t help but wonder whether the issue is genuinely a problem with her driving, or some bias in the test centre or against female drivers generally.

Fortunately the government provides statistics that allow us to test these latter hypotheses and I’ve downloaded the data for 2013-14 (the last year with complete records) to try and understand my friend’s situation.

The simplest model for this sort of situation is a binomial distribution so that whether or not you fail is simply a random event with a fixed probability. In a fair system, the same random process would apply nationwide and so you could model the situation with a funnel plot, as shown in the plot below.

Pass rates for driving tests in the UK, 2013-14

Pass rates for driving tests in the UK, 2013-14

The data also allow us to break down the results by gender and in the plot below you can see that the average pass rate for men is higher than that for women. It’s worth noting that of the 340 test centres in the database, only 12 have pass rates that are higher for women than for men. For interest, I’ve also highlighted the test center where my friend has taken her tests.

Pass rates for driving tests in the UK by gender, 2013-14.  My friend's test centre is highlighted in red.

Pass rates for driving tests in the UK by gender, 2013-14. My friend’s test centre is highlighted in red.

By inspection it’s pretty clear that a basic binomial model is not a good representation of what’s going on here. Clearly the more tests a centre conducts, the lower its pass rate. And while it’s hard to determine causality, the most likely explanation is that in more difficult test centres, the pass rate is lower so people have to retake their test increasing the number of tests conducted.

We can improve on the simple binomial model then by adding a predictor for the number of tests conducted and, since there is clearly a difference by gender, I’ve used a multi-level model formulation that allows for the intercept to vary by gender. The results show that when considering an average test center:

  • the pass rate changes by approximately -1.6% for every additional thousand tests conducted, and
  • the probability of passing is 46% for females and 53% for males.

So for a female taking her test at the center in question, you would expect a 43% chance of passing purely at random. This means that there is a 6% chance of having no passes in 5 attempts. It’s a small consolation but this isn’t much worse than the national average. Females do have lower pass rates than males though that can’t be attributed to discrimination from this data alone.

My only advice is (a) try a smaller test centre (there are several options in her city) and (b) keep trying. After 10 attempts, she can be 99.6% sure of passing.

Generating an academic CV with R and YAML

CV screenshotFor the past couple years, I’ve been using Kieran Healy’s lovely template for my academic CV. Kieran’s code is a customised *.tex file which, of course, has the virtue of simplicity. All a person needs to do is update it with glorious achievements from time to time and re-compile; this is exactly what I’ve been doing since adopting the template in 2011.

But since that time, I’ve had a little niggle in the back of my mind, a concern that despite the elegance of the typography, the underlying software design left something to be desired. If you are manually updating a TeX file with your vita information, then how do you deal with these use cases?

  • Generating a one or two-page short CV for a research proposal.
  • Generating a stand-alone publications list with a complete different format from what’s on your CV.
  • Synchronising your publications database between your reference manager and CV.
  • Recording data about some CV items in greater detail than you would want to display on your CV. For example, you might give a talk and want to remember who invited you, but that’s not really relevant to your vita.

All of this suggests that there should be a separation of content and style, just like a webpage. A full PDF is only one way to display CV data and I wanted something that would be easy to maintain, yet would allow me to generate new output formats quickly if necessary.
Continue reading

Renovation trains

Speaking of retrofitting Holland, there’s a fascinating paper in the new BRI, New energy retrofit concept: ‘renovation trains’ for mass housing, that looks at mass-scale retrofitting in the Netherlands (as opposed to Holland, Michigan). Here’s the relevant blurb from the abstract (emphasis added):

A pilot project exploring this strategy has been completed in Kerkrade, the Netherlands. Several innovations were introduced, tested and refined for the renovation of 150 houses: a serial retrofit process, a renovation time of ten days per house with the houses inhabited during the process, and responsibility for energy performance assigned to the construction team. The construction process is a ‘renovation train’ moving from one house to the next. The experiences are now being used and developed for a follow-up project: a new concept can be used to scale up the process from 100 to 10 000 renovations a year.

The paper is open access (at least for the moment) so go check it out. Alternatively, here’s a video showing how it works:

Not only is the speed of the renovations impressive, but I think the aesthetics are pretty good too (see Figure 3). There are certainly a lot of houses in the UK that look like the ‘before’ houses. My only question is whether the lovely new white render will last. I was in Birmingham last weekend and there is a block of flats near the MAC with the same white facade. They looked great when new but now, maybe five years later, they are discoloured with green algae.

‘You must think me naive ever to have thought this way’

Some further thoughts on professional over-confidence, via Zia Haider Rahman’s excellent ‘In the Light of What We Know‘, pp. 128–129:

The irony is that scientists are much less certain about what they say than politicians, policy-makers, and pundits. The certainty of the kind you see in the face of a politician declaiming on tax increases or hear in the voice of a commentator condemning or endorsing a foreign-policy decision, or the certainty you detect in the words of an op-ed writer pontificating on one thing or another – I used to think that they arrived at their certainty after considering an issue in great depth and finding that the evidence fell overwhelmingly in favour of a specific position. You must think me naive ever to have thought this way. But I did. I used to think that a good argument was the midwife to certainty. If, as I now believe, it is the wish that fathers the thought, then certainty is the lingering imprint of a wish on thoughts and arguments, like DNA retained in progeny, acting invisibly but with visible effects.

See also policy-based evidence.

Retrofitting Holland

From Holland, Michigan (pop: 33,000):

A report detailing costs and projected energy savings for 25 houses in the pilot showed home improvement expenses averaging $14,087. Annual energy savings from those retrofits averaged $865, for a 32 percent energy savings.

Those savings don’t reach the potential 53 percent savings cited in the visionary Garforth report that was the basis of Holland’s Community Energy Plan. But that report also estimated home improvement costs averaging $28,000 for “deep energy retrofits.”

“So actually it’s a pretty good value,” City Manager Ryan Cotton told the council Wednesday night.

While this is a small pilot, there are a few lessons for the UK and elsewhere: Continue reading

DECC, please label your axes

I’ve been exploring DECC’s Energy Consumption in the UK data on non-domestic buildings as part of our new-ish EPSRC project, Future FM. The goal of the project is to develop improved data analysis techniques for energy management in non-domestic buildings in order to support predictive maintenance, highly customised operational schedules, lower carbon emissions, and other benefits.

The first work package is a horizon-scanning exercise. Although we briefly reviewed the literature for the proposal, we now want to examine in more detail the current state-of-practice and the most promising new technologies and techniques from the academic literature; hence the review of energy consumption trends in the UK.
Continue reading