In my last post, I looked at the success rates for EPSRC Fellowship applications using funnel plots. As luck would have it, Alex Hulkes and Derek Gillespie from EPSRC then got it touch to say that they had done a similar internal analysis and would I be interested in the data? Yes please!

The new data set considers EPSRC research grants as a whole, and gives the number of applications and success rates for 137 UK universities for four years, 2009-10 to 2012-13. I wanted to add a bit of colour to this data, literally and figuratively, and so I added a column to indicate whether the university was member of the research-intensive Russell Group or the (now defunct) 1994 Group. The basic funnel plot is shown below; note that I’ve normalised the results for each year to allow direct comparison.

Unfortunately with so many institutions, it’s a bit tricky to make out individual universities. Here I’ve highlighted an example university from each of the three groups. It would be fun to make an interactive version of this using D3 but since it took me ages to make these maps, I think I might pass on this for the moment.

Another question is how the performance of each university has changed over time; I’ve focused on the Russell Group universities to keep things legible. The first question is how to measure “performance”. I’ve done this by calculating, for each university, what is the probability of observing the actual number of successful applications given their total number of submitted proposals and the overall average success rate for that year. In R, this corresponds to the pbinom function and the result gives a score between 0 and 1. I’ve scaled this between 0 (bad) to 100 (good).

To make the plots, I’m using a new and improved version of some code that I wrote earlier for generating Tufte-style slopegraphs. The revised code allows the user to choose different layout algorithms. So in the first plot, the order of the universities is determined only by their rank in 2009-2010; each group line is then constrained not to overlap. This makes it easy to see how a particular university has changed over time, but not their overall rank in subsequent years.

In the second version, the position is based on rank in each year.

We might ask which university is “best”. I’ve calculated this as the average corrected probability of success in each year (see above), weighted by the number of submitted proposals. The table below gives the top 10 universities within each group.

Group | University | Adjusted success rate | Applications submitted |
---|---|---|---|

Russell Group | University of Cambridge | 0.934 | 442 |

University of Bristol | 0.924 | 310 | |

University of Oxford | 0.907 | 437 | |

Durham University | 0.906 | 159 | |

University College London | 0.797 | 523 | |

Newcastle University | 0.786 | 196 | |

University of Sheffield | 0.775 | 352 | |

University of Leeds | 0.759 | 324 | |

Cardiff University | 0.742 | 182 | |

Imperial College London | 0.735 | 664 | |

1994 Group | Institute of Education, University of London | 0.882 | 3 |

University of Lancaster | 0.770 | 127 | |

Goldsmiths, University of London | 0.655 | 10 | |

Royal Holloway, University of London | 0.576 | 52 | |

University of Leicester | 0.574 | 82 | |

Birkbeck, University of London | 0.551 | 14 | |

University of East Anglia | 0.515 | 70 | |

Loughborough University | 0.453 | 251 | |

University of Sussex | 0.433 | 76 | |

University of Essex | 0.317 | 42 | |

Other | Institute of Development Studies | 1.000 | 1 |

National Institute of Agricultural Botany | 1.000 | 3 | |

Queen Margaret University Edinburgh | 1.000 | 1 | |

Rothamsted Research | 1.000 | 5 | |

Scottish Agricultural College | 1.000 | 1 | |

University of the Highlands and Islands | 1.000 | 1 | |

NERC British Geological Survey | 0.938 | 6 | |

EURATOM/CCFE | 0.908 | 2 | |

Transport Research Laboratory Ltd | 0.862 | 3 | |

John Innes Centre | 0.848 | 2 |

I also calculated the performance of the groups as a whole and the Russell Group comes out on top.

Group | Adjusted success rate | Average applications |
---|---|---|

submitted per university | ||

Russell Group | 0.676 | 289.7 |

1994 Group | 0.533 | 72.7 |

Other | 0.429 | 30.2 |

This last table made me wonder if there was any connection between the number of submitted applications and the adjusted success rate (i.e. taking into account the binomial model). In other words, by submitting more applications, are you changing the probability of success? This could be explored with a fancy multi-level model, but I’ve just gone for the basic scatter plot below. The short answer appears to be ‘not really’. The more formal answer is that a linear model has an insignificant fit with an adjusted r^{2} value of 0.011.

There are many more questions that a person could explore with this data, so kudos to EPSRC for making it available.

- Download the data

Ashley FordThanks,

interesting,

there is a spurious in the url for the data

James KeirsteadPost authorThanks Ashley. I edited something yesterday and it turns out the WordPress iOS app introduces lots of spurious line returns. Very frustrating, but should be fixed now.

Mark Claydon-SmithDear James

Thanks for this – very interesting.

One of the problems with basic success rates is that they can be distorted by scale effects – e.g. how they reflect large vs small research grants. One analysis I have used in the past is the ratio of (“value of grants awarded”/”number of proposals submitted”). This is a slightly different slant on the data, but it is a better indicator for /, and can provide a slightly different perspective on institutional performance/strategies. That said, it has been a several years since I last looked at this data in detail.

Regards

Mark Claydon-Smith

EPSRC

James KeirsteadPost authorHi Mark,

Although this data set doesn’t include any information on the value of awards, that would be very interesting to consider in an expanded analysis. A person could also investigate the effect of: number of collaborating institutions, diversity of collaborating institutions, level of external partner support, managed versus responsive mode, etc etc.

James

Tony HirstLove the slopegraphs and the funnel plots:-) Any chance you could post code fragments or a gist showing how you generated them?

James KeirsteadPost authorThanks Tony! For the funnel plots, this Gist (from the earlier fellowshipa analysis) should get you started. The only difference with the more recent ones is that I divided the confidence intervals by the average success rate for each given year (to allow direct comparison of all years) and added some transparency effects for highlighting single data points (using ggplot’s

`alpha`

parameter). For the slopegraphs, there’s a full demo on the project’s Github page. The rest of the code is just re-arranging the data. Oh, and I used the XML to load in the group affiliations of each university.Tony Hirst@James Thanks.. so is the second slopegraph using the “ranked” setting?

James KeirsteadPost authorYes that’s right.