The Value of Experience

The great thing about the web is that if you can imagine a question and if you spend a little time looking you can find often someone that already answered the question. This weekend I wondered: there's all this talk about who's experienced enough to be president - what does the data actually say about how experience correlates with performance? Turns out that they asked the same question at electoral-vote.com and got some interesting answers.

The first scatter plot (or maybe it's the second?) above plots total years of experience against rank for all presidents up to W. Bush, where rank=1 for Abraham Lincoln, and rank=42 for Warren Harding (who?). This is a consensus ranking by historians, so it's somewhat subjective, but there's no apparent correlation between x and y here at all. The second scatter plot (or was it the first?) compares experience to a random permutation of rank. Can you see any difference? I can't. If you're curious, and you compute r^2 for this dataset, then you get a value of 0.008 - essentially nil (an r^2 around 0.5 would be a strong correlation for data of this sort). I poked around in Excel for a while, and the data looks just as uncorrelated if you look at gubernatorial experience, for instance, and a couple of other variants I tried.

So what does that mean? obviously, experience matters for any job, right? Well, one point is that there is a sample bias here - the data is entirely on people that were at least arguably well-qualified for the job, not the result of controlled scientific experiment in which scientists took [wo]men off the street at random and dropped them into the White House. (Insert Sarah Palin joke here if you like). It's wrong to say experience doesn't matter: more accurately the data says that among people qualified enough to be elected, more experience does not correlate with better performance.

An second point: from a machine learning point of view, certainly not all experiences are created equal. Two important properties are the recency and diversity of experiences - for instance, 1,000 samples of poll data collected last week are better than 10,000 samples of polling data collected 50 years ago, and 1,000 samples collected from 50 states is better than 10,000 samples collected from just Utah and Nevada. This data says little about the quality of the experience. Will John McCain's experience as a child and young man in the 1930's and 40's help him be a better president? How much does he actually know now, midway through his third decade in congress, that he didn't know midway through his first?

Other explanations are also possible. Maybe being president is just too unlike other jobs for experience to matter much. Or maybe the Lincolns and FDRs of the world are so remarkable that they rise quickly through the ranks and are still great, while the Buchanans, Garfields, and Van Burens will never be more than mediocre no longer how long they slog along.

Or maybe there's some other explanation.

So what does this all mean? What am I, a pundit? In terms of this week's talking points, neither party scores much - Team Red seems to have all but abandoned the experience-vs-celebrity story and are busy trying to co-opt Team Blue's message of change/new faces. But there's still a lot of noise about how much experience Palin has, and how it compares to Obama's, and I think it's worth noting that a major fraction of the talking-head time this cycle has been (and certainly will be) spent on a point that is a matter of faith, not a matter of objective fact. Once you look at the data, it's clear that simple miles-on-the-odometer experience does not make tend make one a better president - or at least, if a trend exists, it is extremely small.