These are some of the questions that Chris Kennedy’s new book, The Evolution of Great World Cities, seeks to answer. Kennedy is a professor of civil engineering at the University of Toronto with a soft spot for cities and economics. Launching the book in London last week, he noted that he originally wanted to investigate the wealth of cities in much the same way that Adam Smith had done for the wealth of nations. But along the way, the book changed into something subtly different and the result is a fascinating mix of macroeconomics, infrastructure engineering, history, and ecology.

Philadelphia's City Hall

The second theme is the connection between economic growth and the physical structure of cities. Take the automobile as an example. Its introduction in the early twentieth century led to a new mode of urban development, suburban sprawl, which although it has many disadvantages certainly leads to increased consumption. Cars need to manufactured, sold, and serviced; larger suburban homes need to be constructed with more materials and filled with consumer goods. The key issue here is not the infrastructure itself, but the modes of consumption that it necessitates. For example, one might expect that the 1990s IT revolution would have led to a demand side crisis. Just like Smith’s pin-makers, those displaced by the productivity gains of IT – bank tellers, backroom operations, and so on – would be unemployed and unable to consume. However IT also created new opportunities in PC manufacturing, software design, and innovative business models like EBay and Amazon. This new online way of life drove a new cycle of demand, rejuvenating the economy.

To me, this is the book’s most important contribution. This idea of the autonomous consumption of infrastructure, that is the societally mandatory level of minimum consumption created by these systems, is hugely important for understanding not just the economic growth that Kennedy is concerned about, but more broadly the sustainability of cities. North American urban sprawl is a perfect example. While it engenders high levels of consumption, maintaining that lifestyle depends on a throughput of resources, most importantly abundant and affordable transport fuel. Should these fuels become significantly more expensive, these cities would grind to a halt. The same level of infrastructure-mandated autonomous consumption would still be there, but it would now consume a much larger portion of a household’s income, leading to reduced savings, reduced investment in new opportunities, and eventual stagnation. In the language of sustainable development, the autonomous consumption of infrastructure represents a liability that must be serviced on an on-going basis.

The third theme is an analysis of urban economic processes as ecological systems. The relevant chapter offers several noteworthy ideas but it felt incomplete compared to the rest of the tightly-argued book. Nevertheless, an excellent description of Detroit’s decline is provided and the statistics cannot fail to astound: a population decline of 50% between 1930 and today, over 60 square miles of vacant abandoned land (44% of the city’s area), local tree species poking through factory floors that once produced millions of automobiles. Kennedy argues that, in much the same way that an ecosystem needs diversity to survive a changing environment, Detroit was too focused on cars in order to survive, both in terms of the limited diversity of its economy and the inflexibility of its infrastructure. One of the great what-ifs posed by the book is what Detroit might look like now had a 1923 proposal for a subway and integrated transit system been implemented.

Illustrated with case studies on cities as diverse as Toronto and Montreal, Philadelphia and New York, Seville, Paris, Dubai, and London, The Evolution of Great World Cities is a unique work of economic geography. Engineers often complain that economic models are too abstract to offer a meaningful understanding of the real world. Kennedy has therefore done both professions a great service by presenting a strong argument that it is the links between our built environment and economies that matter most.

Chris Kennedy has also written a blog post about the book over at the World Bank’s Sustainable Cities site.

The troublemaker is the SAT problem in section 5.5. The authors give values for two variables and for schools, rounded to integer values. Using the formulas given in the text, I then calculated estimates for the complete pooling statistics and came up with a posterior treatment effect, , of 7.7 and a variance of 16.6. However the text says these values should be 7.9 and 17.4 respectively.

I puzzled over this for quite a while, thinking that maybe I’d missed a prior/posterior distinction somewhere and my estimates were supposed to be subtlely shifted. But no, when I went and checked the original data source, I found that the values were reported to 4 sig figs. Repeating the calculation with the new values gives the expected results. Grr…

Here’s the data and R-code if anyone’s interested.

### Load SAT data from the Gelman et al book and the original Rubin paper
df <- data.frame(school=LETTERS[1:8],
                   book.y=c(28,8,-3,7,-1,1,18,12),
                   rubin.y=c(28.39,7.94,-2.75,6.82,-0.64,0.63,18.01,12.16),
                   book.sigma=c(15,10,16,11,9,11,10,18),
                   rubin.sigma=c(14.9,10.2,16.3,11,9.4,11.4,10.4,17.6))
 
### Rearrange data into a handy form
df <- melt(df,id="school")
vals <- colsplit(df$variable,"\\.",c("source","metric"))
df <- cbind(df,vals)
df <- df[,-2]
df <- cast(df,school+source ~ metric)
 
### Calculate the summary statistics
ddply(df,.(source),summarize,y=sum(y/sigma^2)/sum(1/sigma^2),sig=1/sum(1/sigma^2))

Created by Pretty R at inside-R.org

To get around this problem, one can use quasi-random low-discrepancy sequences which are designed to fill a parameter space efficiently. The R package, randtoolbox, provides implementations of common sequences, like the Halton or Sobol’, but the process involves a couple of steps that beg to be automated. The general process is:

1. Generate a n x p matrix of uniformly distributed quasi-random values, where n is the number of simulations you wish to run and p is the number of parameters.
2. For each column of the matrix, convert the quasi-random value to the parameter’s actual distribution by inverting the cdf curve. So if you have a uniformly distributed value of 0.5, and you want to convert it to a normal distribution with mean you would get a parameter value of for use in your simulation.
3. Run your simulation with these parameter values, and analyse the results

I’ve written a little R function to make this process easier. You simply pass it the number of simulations you want to run, and a list describing each parameter, and it will return the Monte Carlo sample as a data frame. At the moment, it’s pretty rudimentry, and each parameter is described by a name, a distribution name (matching the R abbreviations, e.g. “unif” for the uniform distribution, “norm” for the normal), and two parameters to describe the distribution.

# Generate a Monte Carlo sample
generateMCSample <- function(n, vals) {
  # Packages to generate quasi-random sequences
  # and rearrange the data
  require(randtoolbox)
  require(plyr)
 
  # Generate a Sobol' sequence
  sob <- sobol(n, length(vals))
 
  # Fill a matrix with the values
  # inverted from uniform values to
  # distributions of choice
  samp <- matrix(rep(0,n*(length(vals)+1)), nrow=n)
  samp[,1] <- 1:n
  for (i in 1:length(vals)) {
    l <- vals[[i]]
    dist <- l$dist
    params <- l$params
    samp[,i+1] <- eval(call(paste("q",dist,sep=""),sob[,i],params[1],params[2]))
  }
 
  # Convert matrix to data frame and label
  samp <- as.data.frame(samp)
  names(samp) <- c("n",laply(vals, function(l) l$var))
  return(samp)
}

Created by Pretty R at inside-R.org

Here’s a simple example to show how it can be used.

n <- 1000  # number of simulations to run
 
# List described the distribution of each variable
vals <- list(list(var="Uniform",
               dist="unif",
               params=c(0,1)),
          list(var="Normal",
               dist="norm",
               params=c(0,1)),
          list(var="Weibull",
               dist="weibull",
               params=c(2,1)))
 
# Generate the sample
samp <- generateMCSample(n,vals)
 
# Plot with ggplot2
library(ggplot2)
samp.mt <- melt(samp,id="n")
gg <- ggplot(samp.mt,aes(x=value)) +
  geom_histogram(binwidth=0.1) +
  theme_bw() +
  facet_wrap(~variable, ncol=3,scale="free")

Created by Pretty R at inside-R.org

And here’s the resulting picture:

Histogram of three parameters in a Monte Carlo sample

Any suggestions on how to improve this function so that it has a more generic description of a distribution would be appreciated (e.g. for distributions with n!=2 parameters).

However I recently switched to GitHub for some of my projects. This was partly out of curiosity, as a number of prominent projects and coders seem to swear by it (Twitter, Hadley Wickham, Kieran Healy). But more practically, I didn’t want to be tied to a single server any more and the idea of a Git repository as a self-standing archive was very appealing.

I’ve now been using Git for a couple weeks and, again, I’ve had to figure out a new mystical language of push, pull, fetch, merge, etc. If you’re familiar with SVN or CVS, the trickiest thing is trying to figure out the Git analogues of all your usual tricks. Git from the bottom up is a fantastic resource in this regard and provides a clear explanation of Git’s internal structure and how to work with it. For more mundane, how-do-I-do-x type things, Github’s Help is very, umm, helpful.

Anyway, all that’s not really here nor there. I mainly wanted to announce my first real Git project: lectures! It’s a collection of LaTeX style files and Python glue to make the creation of both lecture notes and slides much easier. If you’ve used beamer, you’ll know that you can build these two sets of documents from a single source file – provided you’ve got two separate header files. With lectures, all you have to do is write a single source file and run a simple command:

> build-lecture source style


Out pops two PDF files, one for presentation and one with your full notes. There is also support for some basic style configuration. I’ve tried to keep this to a minimum, such as changing fonts (with XeLaTeX for OpenType support) and colours.

Have a go, try the code, and let me know what you think!

PS: I’ve just noticed that a similar project exists called BeamerLecture. Looks like the main difference is that it produces four outputs, hiding answers in pre-class notes, which is a nice feature. The formatting is closer to the beamer defaults though and it doesn’t use XeLaTeX for easy font changes.

For those who are interested, the technical detail is explained at length in this paper and this blog (with examples). But what I want to do here is quickly illustrate how you can implement the common get/set method structure in R for a slightly different purpose.

In “traditional” OOP, you might have an object like a Shape with some attribute, say its area. In Java, this attribute would be accessed and altered using get and set methods, as in:

Shape s = new Shape();
s.setArea(10); // set the attribute value
int area = s.getArea(); // get the attribute value


Using this pattern offers advantages like protecting access to the value of the area field and ensuring that only valid values are set. While there is some disagreement about whether or not mutator methods represent good practice, every OOP coder must have had at least one occasion where a getter/setter was needed.

Which brings us back to R. I recently had a problem where I wanted to use something like a getter/setter in order to access a global variable consistently, from both inside and outside functions. Since variable scoping in R isn’t very intuitive, I wasn’t sure how to do this at first and so I thought the get/set paradim might be helpful, even though the value I was getting and setting wasn’t really associated with an object.

Here’s what I ended up using. The trick is to create an explicit environment that stores the variable’s value behind the scenes, so that the user doesn’t have to worry about scope.

# Declare an explicit environment to hold the variable
e1 <- new.env()
 
# Sets the value of the variable
setArea <- function(value) {
  assign("area", value, env=e1)
}
 
# Gets the value of the variable
getArea <- function() {
  return(get("area", e1))
}

Created by Pretty R at inside-R.org

Again, this isn’t really object-oriented programming as I’m not manipulating an object (other than the environment e1) but the get/set pattern has an OOP pedigree. So if you’re coming from a Java or C++ background and are having trouble figuring out variable scoping, the above code might be useful for you. However if you want to do proper object-oriented getting and setting, I highly recommend John Myles White’s example of R object-oriented polymorphism in get/set methods.

> head(data)
  zone  pop     elec       gas  A    B   I    J  L M     O   R    S     W
1    0 7412 46221768 124613714  0    0   0    0  0 0     0   0    0     0
2    4 7428 37345875 100944002 60 3060 120 1020  0 0  4900 390  510 28635
3    7 7464 20914281  64109628  0 1155 510  255  0 0 11000 225 2475 29580
4   10 7412 46221768 124613714  0  680   0  390  0 0  5145   0    0  9300
5   14 7128 69233086  36611811  0 1335 105  210 60 0  5970   0 2520 14910
6   17 7608 40783190  59343776  0  150   0  150  0 0  1500   0    0   555

Created by Pretty R at inside-R.org

I then performed a basic regression using lm, removing the intercept (the “- 1″) as I want the population coefficient to serve a similar purpose for this analysis:

lm.elec <- lm(elec/365 ~ pop + A + B + I + J + L + M + O + R + S + W - 1, data)

Created by Pretty R at inside-R.org

Using Andrew Gelman’s helpful arm package, I can get a quick overview of the result as shown below. It doesn’t look too bad at first, but then I noticed all sorts of negative coefficients for the activity levels. This makes no sense: when individuals perform activities such as going to work, going to school, or shopping, we expect that their demand for electricity should go up, not down.

> display(lm.elec)
lm(formula = elec/365 ~ pop + A + B + I + J + L + M + O + R +
    S + W - 1, data = data)
    coef.est coef.se
pop    13.99     1.35
A     177.07    83.87
B     -63.90    27.80
I      -0.80    21.05
J      91.27    34.19
L   -1075.76   391.53
M      -5.57    73.50
O      55.47     9.90
R    -336.14   178.98
S      28.12     3.84
W      -8.19     3.16
---
n = 391, k = 11
residual sd = 159994.17, R-Squared = 0.65

Created by Pretty R at inside-R.org

Since a linear regression is essentially an optimization problem, my immediate thought was: can I just constrain the coefficient values so that they are all positive? This would mean that some activities might have no significant effect on consumption, but at least they couldn’t have a negative impact. And it turns out, yes, you can do this using the nnls package and function.

The nnls function is not quite as user-friendly as lm so the first thing you have to do is manually define your input variable matrix and output vector. For example:

A <- as.matrix(data[,c("pop","A","B","I","J","L","M","O","R","S","W")])
b.elec <- data$elec/365

The analysis can then be run as:

nnls.elec <- nnls(A,b.elec)

Similarly, we can’t use predict to generate our results and have to manually perform the matrix multiplication. This can be done as shown below.

coef.elec <- coef(nnls.elec)
pred.elec.nnls <- as.vector(coef.elec%*%t(A))

This method also doesn’t give you an r² value per se, but you can estimate it with the following dummy regression:

> lm.elec.dummy <- lm(b.elec~pred.elec.nnls - 1)
> display(lm.elec.dummy)
lm(formula = b.elec ~ pred.elec.nnls - 1)
               coef.est coef.se
pred.elec.nnls 1.00     0.04
---
n = 391, k = 1
residual sd = 164861.13, R-Squared = 0.62

As you can see, the r² value is slightly lower in this case than the standard lm model but in terms of interpreting the coefficients the result makes much more sense. This can be clearly seen in the following graph, where demands for electricity and gas rise during the day as expected, rather than sinking in the basic case.

Simulated profiles for electricity and gas, comparing lm and nnls regressions

So there you go. If you ever need to run a regression and ensure that all the coefficients are greater than or equal to zero, nnls is your friend.

But whatever you want to call, there’s no denying that these plots are an elegant and concise way to display trends in grouped data. In this post, I’ll show you an attempt I’ve made at creating a slopegraph in R using ggplot. Here’s the original figure from Tufte’s book, Beautiful Evidence, to give you an idea of what we’re aiming for.

Edward Tufte's slopegraph of cancer survival rates

First, we need to get some data. The basic structure of a data set for this kind of plot is a long table with columns for each group, measurement year, and measurement value. In Tufte’s plot, the groups are various types of cancer, and the measure is the estimated survival rate (in %) taken at 5 year intervals. You can find the raw data here but I’ve already arranged it in the necessary format for you in the Github project.

A slopegraph is intended to “compare changes over time for a list of nouns located on an ordinal or interval scale”. It does this by plotting the relative rank of each group at a given point in time, the measured value, and a line indicating the change in value between two measurement points. At first glance, this should be a fairly easy plotting task especially with ggplot2 and the naive code might look something like this:

ggplot(data, aes(x=year,y=value,group=group)) + geom_line()

Created by Pretty R at inside-R.org

However this produces all sorts of nasty collisons, as there may be many labels plotted at similar values. Tufte solves these sorts of problems by creating his figures largely by hand (well, with Illustrator). This gives full control over the typography, spacing and so on, but it’s not very reproduceable.

With R, we can automate the plotting process although our aesthetic tweaks can only be approximate. What I’ve done is to add an offset between each group at a given year to ensure that a certain amount of minimum spacing is maintained. The algorithm isn’t perfect, as a value of x in a given year may be positioned higher than x + 1 in an adjacent year depending on the characteristics of each year’s data. (For a particularly ugly example, look at the liver and intraheptic bile duct group in the plot below.) Tufte’s plot also has this characteristic, but he arranges the lines so that they never cross: clearer to read but potentially confusing when making horizontal comparisons.

Once the data set is prepared, we need to plot it. The first thing to do here is to create a new ggplot theme which strips away most of the plot decoration. I did this by typing theme_bw at the command line, copying the resulting output, and modifying it to my needs. We can then make the plot using standard ggplot commands. The plot consists of (in plotting order):

1. Lines linking each group
2. White points to provide a background for each number
3. Text labels for each numeric value
4. Text labels for the groups
5. A scale adjustment to ensure that the labels appear properly

The final result is below. It’s kind of a big plot, so CFB.

R version of slopegraph for cancer survival rates

I’ve put the code and data up on Github so feel free to fork it and tweak things to your liking.

Cities account for approximately two-thirds of global primary energy consumption and have large heat and power demands. CHP (combined heat and power) systems offer significant primary energy-efficiency gains and emissions reductions, but they can have high upfront investment costs and create nuisance pollution within the urban environment. Urban planners therefore need to understand the tradeoffs between limitations on CHP plant size and the performance of the overall energy system. This paper uses a mixed-integer linear programming model to evaluate urban energy system designs for a range of city sizes and technology scenarios. The results suggest that the most cost-effective and energy-efficient scenarios require a mix of technology scales including CHP plants of appropriate size for the total urban demand. For the cities studied here (less than 200,000 people), planning restrictions that prevent the use of CHP technologies could lead to total system cost penalties of 2% (but with significantly different cost structures) and energy-efficiency penalties of up to 24% when measured against a boiler-only business-as-usual case.

The performance of this particular model could be better, as several of the scenario results had large solution gaps which makes it difficult to see the underlying trends. However I think the method we used, applying an integer programming model to design multiple energy systems for generic cities, is innovative and will be of interest to others.

Keirstead, J., Samsatli, N., Shah, N., & Weber, C. (2011). The impact of CHP (combined heat and power) planning restrictions on the efficiency of urban energy systems Energy DOI: 10.1016/j.energy.2011.06.011

I recently analysed these emissions as part of a research project for EDF Energy. The idea was to assess the carbon footprint of rush hour, looking at regional variations in distance travelled, transport mode, and total emissions. If you’re interested in the official results, the study was covered in the Evening Standard, Independent, Business Green and a few blogs, and here’s the official press release. But I wanted to add a few supplementary results here.^*

To calculate the carbon emissions of rush hour, I used data from the 2008 UK National Travel Survey and tallied up all the trips between 700–1000 and 1600–1900. Each trip can be characterized by its distance, purpose, the transport mode used, the location of the trip, and many other attributes. This was the first time I’d used this data set and I was surprised to see just how much information is collected.

For me, the most interesting result was to see how distinct Greater London is from the other British regions. Look at the figures below: loads of public transport, lots of commuting travel, and surprisingly high ownership of gas-guzzlers (band J and above).

Modal share of rush-hour trips. Private transport in red, public transport in blue.

Purpose of rush-hour trips

Vehicle ownership by area type

Another useful plot is the distribution of activities by time of day. You can see here that the evening rush “hour” is more dispersed than the morning rush, with the afternoon school-run starting around 3 pm and heavy travel continuing until about 7 o’clock.

Distribution of daily trips, shaded by purpose

It’s a rich data source and we’ve only scratched the surface. Hopefully we can do some follow-up research to look in detail at variety of trips and their suitability to different low carbon transport options.

^* Of course, these additional comments are my own opinions and have nothing to do with EDF.

“When you are studying any matter, or considering any philosophy, ask yourself only what are the facts and what is the truth that the facts bear out. Never let yourself be diverted either by what you wish to believe or by what you think could have beneficient social effects if it were believed. But look only and solely at what are the facts.”

At first glance Mark Lynas’s new book, The God Species: How the Planet Can Survive the Age of Humans, has nothing to do with Professor Russell. It is ostensibly about planetary boundaries, the nine physical limits which humanity must respect if we are to maintain our current quality of life on Earth. These boundaries – in biodiversity, climate, nitrogen, land use, freshwater, toxics, aerosols, ocean acidification, and the ozone layer – are well-described with clear explanations of the relevant science, the implications of violating the boundary, and the technologies and policies that could help us avoid crossing these limits. Lynas also draws out the links between the boundaries, for example between climate change and ocean acidification, to illustrate both the win-win benefits of certain mitigation options and conversely the negative feedbacks that might occur as boundaries are crossed. There are a few minor presentational niggles – a comparative sentence that uses two units for area, a throwaway reference to declining wood consumption without accounting for fuel substitution effects, the subtitle (do we care about the planet itself or our life on it?) – and like many other popular science books, editors seem to be adverse to including graphs even when something like the Mauna Lua CO2 timeseries with the 350 ppm threshold marked would be a concise illustration of the climate boundary. But overall it’s an excellent introduction to an emerging and important way of viewing global environmental issues.

But in many ways, the real story of The God Species is Lynas’s re-discovery Russell’s maxim (knowingly or not). Lynas has been a devoted environmental campaigner for years and throughout the text, he is at pains to point out that the planetary boundaries view of environmental problems means adopting solutions that mainstream environmental groups may find hard to swallow. For example, discussions on the potential of nuclear power, genetic engineering, or geo-engineering include lines such as “I know many environmentalists will read all this with a sense of growing horror. A technofix! How outrageous! What a cop-out!”. While Lynas’s pragmatic view is to be commended, I found the “you’re not going to like what I have to tell you…” tone somewhat distracting. It felt as though the reader had missed out on a long and tormented backstory, like catching a bitter passing aside between two ex-lovers at a party.

The God Species, then, has the feeling of two strong themes shoe-horned into one volume. The first subject, the science of planetary boundaries, is fascinating and the author brings these issues to the attention of a wider audience with a deft touch. But a longer treatment would permit a more detailed discussion of the science and the suggested solutions. For example, the discussion on the sustainability of economic growth would benefit from reference to Bill Nordhaus, William Ayres and others in the resource economics literature. Similarly, when assessing the role of privatisation in the delivery of freshwater, the choice is presented as a fairly strict dichotomy between “efficient” private and “inefficient” public provision without really getting into the role of institutions and regulations to ensure that either system works effectively. More detail on the science of how planetary boundaries are identified and monitored could also be valuable.

The second theme, on the environmental movement’s selective use of scientific data, also deserves more attention. Lynas hints at this in the last chapter, providing a fascinating eyewitness account of the Copenhagen climate negotiations (see also this Guardian piece), but a more detailed analysis of how environmental groups have painted themselves into corners on climate change and other issues by arguing against politically feasible outcomes for the sake of environmental purism would be literary and political dynamite.

But perhaps that’s something for Lynas’s next project. On its own terms, The God Species is an excellent introduction to a new pragmatic way of looking at the environment, one that as Russell noted is supported only by the facts and “the truth that the facts bear out”.

‘The God Species’ by Mark Lynas is published by Fourth Estate on July 7th and is available for pre-order from Amazon.co.uk