FiveThirtyEight.com: Politics Done Right: The Two Progressivisms

I became a big fan of Nate Silver during the election, then stopped reading him for a while after it. My loss actually, since he's just as insightful when he's not crunching poll numbers, eg with his latest post on The Two Progressivisms.

World's least informative superbowl play-by-play

Yet, strangely compelling....

Lessons to learn from the financial crisis

I just finished reading a great NYT article on the financial crisis and what they call "the strange story of Harry Markopolos... who, for nine years, tried to explain to the Securities and Exchange Commission that ... Madoff’s investment performance...was mathematically impossible." Markopolos wrote a 17-page report in 2005(!) crunching the numbers and arguing the most likely case was, quote, that "Madoff Securities is the world’s largest Ponzi Scheme.” Which of course it was.

In April 2008 Markopolos tried again, with an e-mail message to Jonathan S. Sokobin the new head of the commission’s office of risk assessment. “Attached is a submission I’ve made to the S.E.C. three times in Boston,” he wrote. “Each time Boston sent this to New York. Meagan Cheung, branch chief, in New York actually investigated this but with no result that I am aware of. In my conversations with her, I did not believe that she had the derivatives or mathematical background to understand the violations.”

The NYT authors are all over this:

How does this happen? How can the person in charge of assessing Wall Street firms not have the tools to understand them? Is the S.E.C. that inept? Perhaps, but the problem inside the commission is far worse — because inept people can be replaced. The problem is systemic. The new director of risk assessment was no more likely to grasp the risk of Bernard Madoff than the old director of risk assessment because the new guy’s thoughts and beliefs were guided by the same incentives: the need to curry favor with the politically influential and the desire to keep sweet the Wall Street elite.

Maybe their analysis is right - but personally, I'm not sure that wishful thinking and incompetence are the whole story. So much of life has become so complex that it's poorly understood, and we seem to mostly accept this. Case in point: I have a PhD in computer science and about 25 years experience, and I've done everything from soldiering together microprocessors to writing compilers to proving impossibility theorems. Yesterday I spent 45 minutes fixing the wireless on my laptop - and I don't understand why what I did worked. I could in principle, I'm sure - but that would take even more time, which I'd rather spend doing other stuff.

So I can readily imagine the director of risk assessment saying: gosh, this is complicated. I don't understand it very well. Some experts say it's a problem, some say it's not...hmm, what should I do? What should he do? if you're in this situation every week, or every month, how do you decide what to dig into, and what to ignore? How do you tell the difference between what you think is true, and what you hope is true? How many "experts" are in this situation every week, or every day?

Today's thought is: how should you handle complexity of this sort? because there is a right way and a wrong way. If I was to boil the right way down to a few words, they would be:

• Study the data. Always start with what you know - the data. Plot it, graph it, run your favorite models on it, look at the outliers. After you've hacked at it for a while, you'll start to get a feel for what it's like. Madoff was an outlier in several respects - rate of return for the last few years, but also organizationally. If you're interested in modeling the market, you'd want to discount him as representative - if you were interested in detecting fraud, outliers are the things you want to watch most closely.
  
• Do the math. Complexity is your enemy, and good math is your best friend and most powerful weapon against it. (There is of course evil math, which makes the simple complex instead of making the complex simple, but hopefully you're smart enough to recognize the difference.)
  
• Ask questions. When you don't understand something, ask questions until you do. If you do understand something, think it through a little more, until you find a question...you'll usually find out that either you understand it a little better after you do...or sometimes, that nobody really understands it.

The great thing about these lessons is that you can learn exactly the same things from so many recent experiences...anyone for discussing Iraq or global warming?

Today's deep thought is: how much of modern computer science is about making the complexities in our world simpler and more understandable - and how much is making the world more complex and more opaque? And how complicated can a system be and still be controlled, maintained, and regulated by human beings?

The Last Veridian Note

On sabbatical this year at Google, I keep noticing how little of my old office space I've taken with me, and how little I miss it.  As Bruce Sterling points out:  "objects can no longer protect you from want, from humiliation – in fact they are causes of humiliation, as anyone with a McMansion crammed with Chinese-made goods and an unsellable SUV has now learned at great cost."  Almost makes me feel like cleaning up at home...

More data on tax policies

Above is yet another graph, using data from the same source - a simple one just showing the average tax rate in each income bracket: the bottom 20% of the population, the next 20%, and so on, up to the top 0.1%. The famous "Joe the plumber" is just at the around the 95% percentile - at the top end of that long, long range (from about the 40 percentile mark to the 90 percentile mark, $40k/yr to about $170k/yr) where the tax policies of the two candidates are pretty much the same. Or if you prefer, at the bottom of that long range (from $250k/yr to $2,800k/yr and up), where the policies are different.

I find it exceedingly strange that there's so much rhetoric about very small differences (eg for Joe the plumber's tax rate) and so little discussion about the fairly large differences in tax rates for the top end of the range. The idea that tax credits are socialist is just wacky - these have been pushed by conservatives for decades now as an alternative to heavier-weight social programs, and in fact McCain's health care plan is based on tax credits. And both plans are clearly progressive in the sense that higher incomes pay higher tax rates - the difference is only in degree, not kind.

Even if you accept the claim that the very top of the scale for tax rates is important, the difference in top tax rate is also not especially large in a global sense - in particular moving from 29-38% doesn't turn the US from a free-market bastion to a socialistic Sweden wanna-be. In act, it doesn't dramatically change the ranking of the US for top personal income tax rate. Look at this chart (based on 2005 data) from Wikipedia, and imagine moving the top US bracket from 28% or so to 38% or so:

Or if you prefer here are some selected values from the data tables on the WP page that I sliced out and graphed myself against the Obama and McCain proposals. It's clearly a difference...but it's just as clearly not a switch from capitalism to Marxism.

More visual comparisons

From Mark Thoma at The Economists View, here's a pie chart showing earmarks as a percentage of the total Federal budget.



..and here is offshore drilling production, by year, as a fraction of total oil consumption.

What you miss when you go to bed early

Another tragic victim of socialized medicine....

My brilliant idea of the week...

...how about announcing a concert that hasn't happened yet, instead of one that's over? Our next show is a

Musical Open House for Obama featuring folk / blues / ragtime / bluegrass band Smokestack Lightining & friends.

Sunday Oct 26, 7:30 - 9:30pm
at the House of Nick Coles and Jen Matesa
331 S. Fairmount Street, Pittsburgh 15232

No cover - just donate what you can if you appreciate the music!

RSVP at the snazzy Obama-provided event page or listen to some samples on myspace if you're not one of those "low information" listeners.

Visualizing the Long Tail for Tax Policy: Part 2

Above is a corresponding visualization of the Obama plan - but it's a little more complicated. Again, you can click to zoom in.

Obama's plan includes both tax cuts and tax increases. All of Obama's tax cuts go to the bottom 900 people (the poorest 90%). The total cuts for the bottom 90% is much larger than for McCain's plan, but it's also distributed quite differently, as the inset shows.

For the next 90 people (percentiles 90-99%) it's basically a wash - the bottom half of these will get a tax cut, but a very slight one, and the top half of those 90 will see an increase: in terms of field position, they'll take a penalty, and have to move back to the left. How far, you ask? On average across these 90 people, the penalty is about 6 inches each - not enough to see on the gridiron I drew.

For the next 9 people, in the 99-99.9% bracket, the average tax increase is big enough to be visible: about a 4 yards penalty. An offsides penalty, maybe. I drew the approximate shift of each of these 9 people on the graph, with dotted blue arrows pointing from the old to the new position.

For the last guy, the penalty is nearly 22 yards, moving him back to the 70 or 75. Maybe a holding call.

I didn't bother to draw the appropriate-size stack of bills for the tax increases- you'll just have to image it. I didn't draw field-position arrows for the McCain tax plan because nobody moves by more than a foot or so - except for the guy down by the 94-yard line, who just about makes it to the goal.

I should also probably point out (in case it's not obvious) that neither of these plans is likely to be implemented exactly in their proposed form - even if the economy and stock market hadn't implored the last couple of weeks, which has obviously changed things, there's a long process of digestion before any major legislation is produced by congress, and like digestion, what comes out is often quite dissimilar from what goes in. It's probably best to think of this as a statement of what values the candidates stand support, rather than a prediction of how your taxes will change.

So to summarize:

• The Obama plan makes the spread between the highest-paid and lowest-paid Americans noticibly smaller. The McCain plan makes it slightly larger.
• The bottom 50-60% of the country by income do much better under the Obama plan).
• The tax plans are not very different for the 60-95% bracket, between about $70k and $170k per year.
• The 95-99% bracket ($170-$240k) clearly does better under McCain - by a substantial amount in dollars each, but a small percentage (a little more than 2%, actually) in income.
• The top 1%, and especially the top 0.1%, do much much better under McCain.

Go Steelers!

Visualizing the Long Tail for Tax Policy

They say software is hard to understand because it's "invisible" - but aren't so many things? For fun, I downloaded the raw data on Obama and McCain's proposed tax policies from http://www.taxpolicycenter.org - they have it all in Excel form, which makes crunching it convenient and easy - and tried to construct what I considered a reasonable visualization. Their data is broken out to show the effective tax increase or decrease for the first quintile, the second quintile, etc, with the top quintile broken down to partitions of the top 10%, 5%, 1% and 0.1%. I wanted to show the range of salaries in each partition, the number of people in each partition, and the change in tax. I decided that it just can't be done - the ranges are just two broad to see on, without using hard-to-explain constructs like a loglog plot.

Here's a mental image that might help. Think of a football field, 1000 typical Americans, and some big stacks of money in small bills. Specifically, think of $1 and $5 shuffled together and piled up so that each pile contains about equal numbers of $1's and $5's. If I did the math right (they say a US bill is 0.043" thick) then a money stack one foot tall is worth about $10k - so the football field is $3M long and $1.6M wide, I'm about $57k tall, and my waist size used to be about $30k (but with my diet, I lost about $1600 around the middle).

Now to visualize income distribution, let's take those 1000 typical Americans, and let them walk down the gridiron like this: start at one goal line, and for every $10k of yearly income, walk one foot toward the other end zone...so if you make $90k/year, e.g., you'll end up on the three-yard line, and if you make $300k/year, you'll end up on the 10. Subject to that rule, space everyone out as much as you can.

The first 900 of those 1000 people will end up somewhere before the 4 yard line, because about 90% of Americans make about $120k or less.

Another 90 people will end up between the 4 yard line and the 8 yard line - that is, they make between $120k and $240k per year.

Another 9 people will end up between the 8 yard line at the 20 - they make between $240k and $600k per year.

That accounts for 999 out of 1000 people, but this is America, land of Bill Gates and Warren Buffett - so we're not done yet. The last guy will stand, on average, at the 94 yard line - with an average income of $2.8M per year. Remember this is income, not life savings.

Now, for the tax cuts. Let's start with McCain's. EIn his plan, everyone gets a tax cut, and we'll hand it out as stacks of money. McCain has a stack about 9 feet high to be shared by the first 900 people; a stack about 5 feet high to be shared by the next 90 people; another 9.5 foot stack to be shared by the next 9 people; and a final stack about 6.5 feet high - about the height of Ben Rothlisberger - that goes to that one lonely old man (surely, he's old?) down on the 94 yard line.

Above is a picture I drew of this...you can click to zoom in.


This is approximate, and loses some detail - which maybe I'll get to in another post - but it illustrates the main properties of McCain's plan: there is an extremely long tail in income distribution in the US, and as a consequence, an extremely long tail of tax-cut returns. If you imagine the money stacks are all the same (which is order-of-magnitude correct), then

• For the bottom 90%, the per-person tax cut is about 1/10 that of people in the 90-99% bracket.
• For the 90-99% partition, the per-person tax cut is about 1/10 that of people in the 99-99.9% bracket.
• For the 99-99.9% partition, the per-person tax cut is about 1/10 that of people in the top one-tenth of one percent.

"I shot a moose once"

And now, an informational message, for those who may not fully recognize the intricacies and dangers of moose hunting.

The Sarah Palin Turing test

I was wondering .... was it just me, or did anyone else have the reaction "Gosh, her answer sounds like it was generated from a statistical language model trained on a corpus of random Republican talking points?"

Snazzy Presidential Debate Search Engine

Available at the NYT.

Summary of the morning's news...

Sometime last night, the bailout became no bailout. Obama, via the NYT:

“What I’ve found, and I think it was confirmed today, is that when you inject presidential politics into delicate negotiations, it’s not necessarily as helpful as it needs to be,” Mr. Obama told reporters Thursday evening. “Just because there is a lot of glare of the spotlight, there’s the potential for posturing or suspicions. When you’re not worrying about who’s getting credit, or who’s getting blamed, then things tend to move forward a little more constructively.”

The Republican side of the story, as told by Kevin Smith, aid to House Republican Leader Boehner:

But a top aide to Mr. Boehner said it was Democrats who had done the political posturing. The aide, Kevin Smith, said Republicans revolted, in part, because they were chafing at what they saw as an attempt by Democrats to jam through an agreement on the bailout early Thursday and deny Mr. McCain an opportunity to participate in the agreement.

The Value of Experience

The great thing about the web is that if you can imagine a question and if you spend a little time looking you can find often someone that already answered the question. This weekend I wondered: there's all this talk about who's experienced enough to be president - what does the data actually say about how experience correlates with performance? Turns out that they asked the same question at electoral-vote.com and got some interesting answers.

The first scatter plot (or maybe it's the second?) above plots total years of experience against rank for all presidents up to W. Bush, where rank=1 for Abraham Lincoln, and rank=42 for Warren Harding (who?). This is a consensus ranking by historians, so it's somewhat subjective, but there's no apparent correlation between x and y here at all. The second scatter plot (or was it the first?) compares experience to a random permutation of rank. Can you see any difference? I can't. If you're curious, and you compute r^2 for this dataset, then you get a value of 0.008 - essentially nil (an r^2 around 0.5 would be a strong correlation for data of this sort). I poked around in Excel for a while, and the data looks just as uncorrelated if you look at gubernatorial experience, for instance, and a couple of other variants I tried.

So what does that mean? obviously, experience matters for any job, right? Well, one point is that there is a sample bias here - the data is entirely on people that were at least arguably well-qualified for the job, not the result of controlled scientific experiment in which scientists took [wo]men off the street at random and dropped them into the White House. (Insert Sarah Palin joke here if you like). It's wrong to say experience doesn't matter: more accurately the data says that among people qualified enough to be elected, more experience does not correlate with better performance.

An second point: from a machine learning point of view, certainly not all experiences are created equal. Two important properties are the recency and diversity of experiences - for instance, 1,000 samples of poll data collected last week are better than 10,000 samples of polling data collected 50 years ago, and 1,000 samples collected from 50 states is better than 10,000 samples collected from just Utah and Nevada. This data says little about the quality of the experience. Will John McCain's experience as a child and young man in the 1930's and 40's help him be a better president? How much does he actually know now, midway through his third decade in congress, that he didn't know midway through his first?

Other explanations are also possible. Maybe being president is just too unlike other jobs for experience to matter much. Or maybe the Lincolns and FDRs of the world are so remarkable that they rise quickly through the ranks and are still great, while the Buchanans, Garfields, and Van Burens will never be more than mediocre no longer how long they slog along.

Or maybe there's some other explanation.

So what does this all mean? What am I, a pundit? In terms of this week's talking points, neither party scores much - Team Red seems to have all but abandoned the experience-vs-celebrity story and are busy trying to co-opt Team Blue's message of change/new faces. But there's still a lot of noise about how much experience Palin has, and how it compares to Obama's, and I think it's worth noting that a major fraction of the talking-head time this cycle has been (and certainly will be) spent on a point that is a matter of faith, not a matter of objective fact. Once you look at the data, it's clear that simple miles-on-the-odometer experience does not make tend make one a better president - or at least, if a trend exists, it is extremely small.

politics shmolitics, as long as it swings

A clever YouTube video produced by my good friend Judy Minot.

Another performance by Smokestack Lightinin'

This one was at the bizarrely inspired 1st Annual Alexander Berkman Festival, and was captured in full audio glory by Er1k Riebling (thanks Er1k!) The whole set (20min) consisted of:

1. Pay Me My Money Down
2. Big Boss Man
3. Fiddle Tunes
4. Larimer St.
5. Homestead Strike Song
   

Upcoming gig

I'll be playing (mostly mandolin) Tues night in Braddock, with the band Smokestack Lightenin' for a fundraiser for Unseam’d Shakespeare Company’s Out of this Furnace: project. The headliner is Anne Feeney.

Where: Elks Lodge #883- 424 Library Street, Braddock, Pennsylvania
When: Tuesday May 20, 2008 7:00 PM-11:00PM
Tickets: $25 (or pay what you can), available at the door (cash/check only)
Contact: Tim Dawson, 412-621-0244 for directions and information

We have violated the prime directive

Noah Smith and I are co-supervising Tae Yano on a project involving analysis of political blogs, and Tae left a pile of results and code on her CMU web site as a way of communicating with us...world-readable. Surprisingly someone at one of the blogs she spidered, Little Green Footballs, actually noticed, leading to a lot of investigative work in this fascinating thread:

Anyone know what this page at Carnegie Mellon means? It’s some kind of experiment that involves comments posted at LGF, and I have a feeling it’s not friendly.
...
comment #5: Maybe it's post-modern poetry, academia-style? Using LGF comments as gibberish to transcend interpretation?
...
comment #18: Obviously it's a KGB plot.
...
#12: there's a Kos directory too. Probably just too what they consider to be partisan sites to test some kind of language algorithms or something
...
#20: I think they are doing a text analysis looking for Charles' sock puppets -- like the way some trolls accuse Robert Spencer of posting as Hugh
...
#23: Looks like they are doing another election-year study looking at patterns of argument or other aspects of communication.
...
#47: Who ever thought that garbage labeled as "research" could be so sophisticated and at the same time be so worthless. ... Polishing farts to get Piled higher and Deeper has become, for the most part, an ephemeral exercise in self deception. [ouch! sounds like my last grant review].
...
#48: This isn't that advanced of stuff - we are working on a simillar concept for internal corporate communications. This has been around a while.
....
#60: EXPERIMENTATION WITHOUT REPRESENTATION! I demand compensation!
...
#61: I wonder if linking to their experiment will introduce fatal feedback loops, leading to a disturbance in the Krell mind-field and possible space-time discontinuity?
...
#79: It looks to my un-Pythonesque eye like a programmer is using the cosine function to create some kind of mapping on how comments refer to one another based on comment location in the thread; and looking at different types of postings from Charles (open thread, breaking news, news outlet criticisms) to see if there are any trends in the discussions that vary according to the posting type. That has to be one of the worst run-on sentences I've ever written.
...
#92: [#64: "What is port sniffing?"] after you pour some in the glass, but before you take a sip, swirl it around a bit and inhale whist the dark chocolate is melting on your tongue......
...
#162: Ahh, so basically it's a merger between post-Marxist deconstructionism and THX-1138-esque Orwellian personality dehumanization.
...
#172: I liked the method of withdrawing all posts by 204 commenters, then asking the system to predict what those posts were (or *if* the post existed, I think) based on all the rest.
...
#187: ok here's how I see it. There is a whole group at Carnegie Mellon CS department doing research in Artificial Intelligence by running statistics on blogs. A grad student Tae Yano stashed her research files on a server and left them wide open to WWW. The files pertain to running basic stats on DK and LGF, up to cosine similarity. The AI goal would be to have a blog-commenting software indistinguishable from a blog-commenting human. For all I know, this, or any other comment, is already written by a python script.
...
#277: [Quoted from a grant proposal Noah & I wrote that got unearthed, along with various other papers, pictures of Noah's cats, Tae's cv, her picture, her programming project on knitting, wedding announcement, etc:] "Political text is often indirect, sarcastic, repetitive, hyperbolic, emotional, biased, manipulative, and riddled with unstated assumptions." - Byte me.
...
#314: So some students are doing research on blogs. No big deal. It was interesting to see what it was about, but that's about it. I don't think they feel the need to protect their files. Why should they? Who would want to mess with that? It's just a school project, I don't see point of "exposing" the students and publicizing their information. At least that's my take.
...
#316: I noticed that the research group to which she belongs is partially funded by a DARPA grant.
...
#361: And the same techniques/analyses will probably work on Arabic-language websites, too.
...
#428: Are terrorists ‘phone banking’ for Barack?
...
#474: magine a natural language program that could respond to comments with charm and style, sort of a robo-blogger. Now imagine an army of them, all set to monitor a different political blog, run by a campaign manager for a politician. Add to its writing ability an encyclopedic memory, with instant access to famous quotes, historical facts, trivia, statistics, and every word ever uttered by the opposition. You now have an army of ultimate bloggers, all completely under the control of one campaign manager... no more "going off message" by some underpaid/volunteer lackey, just high quality counter-opinion, ready to be inserted into the blogs of anyone who disagrees with your candidate. This research will eventually lead to robo-blogging to kill emerging scandals and alternative opinions on issues... no more Rathergates as they will be smothered in the cradle by the most charming bloggers around -- the poli-bots.

...
#482: She's trying for a data-mining tool tailored for blogs that separates "useful, thoughtful" information from all the mindless dreck in the blogosphere. Lotsa luck with that! As an aside, I've been on the Carnegie Mellon campus, toured the Computer Science department, and met with CS faculty. It's a gorgeous campus. The school clearly has big bucks. CMU holds numerous contracts with various government agencies related to the information technology aspects of defense, computer security, homeland security, and similar "black ops" topics. At least some people on that campus have intimate access to NSA, DOD, and CIA. It's a spooky place.
...
#489: We should, for amusements sake, keep an eye open for a KOS diary about this. There may be some entertaining histrionics and conniption fits over their being the unwitting subject of DARPA funded research.
...
#492: But but but... Markos is a CIA agent.... dKos is a DARPA funded research project....
...
#496: Don't spill the beans. The Koslings haven't figured any of that out yet. Agent Markos will have a rough time of it when he is exposed as being a double secret agent of the Zionist conspiracy. Don't blow his cover.
...
#497: From this, we conclude that LGF not only has MORE numbers than DKOS, but BIGGER numbers as well. If YOU TOO want bigger numbers, choose LGF brand blogs.
...
#514: Was just thinking that all you people have too much time on your hands...but then it occurred to me I'm sitting here reading all this.
...
#518: Looks like the whole thing has gone 404. My guess is that she just wanted a corpus of data for some programming project and, now that the object under study is aware of her, it's no longer useful. I don't see any dark purpose here. How evil can someone be who writes knitting software?
...
#519: I suspect that if the mice KNOW they are in an experiment, they will not produce the same results,as they would otherwise, thus invalidating the experiment.
...
#520: [re: #21 zombie: "Talk about pointless. People get PhDs for this crap."] Not really pointless. The "value" of this may be questionable though, at this point in time. As the web and bogosphere has grown exponentially in influence, there has been great interest in determining if real life outcomes can be affected by influential opinions posted on blogs -- and then re-created in numerous other places to mimick majority viewpoints. There are numerous companies invloved in this research-like activity which can be tailored for marketing, business intelligence financial trading, political campaign managment uses (for example, Umbria of Denver, CO was just sold to JD Power). They use primarily data visualization tools (like those used now by lawyers engaged in electronic data discovery) -- These are similar to the software tools used by intelligence services to monitor, track, analyze, for example, wireless (web/phone) transmissions and discussions originating in the US, and destined for overseas delivery in places like, say, Iran.
...
#531: Yea- shut down. I didn't get to it in time to even see what it was all about. So, now I'm depressed. And stuff.

Since this morning, Tae chmod'ed all her code to hide it, but I suggested to Tae that she keep it visible, since folks were having so much fun with it. (Of course now she's embarassed and wants to clean up her code first...) Noah and I also wrote an open letter to LGF explaining what was happening.

I'm mostly amused (as you can tell), but also impressed at (1) how much was uncovered about Tae, Noah and me and (2) how much of this obscure statistical NLP code the LGF community was able to figure out, communally. The peril and power of the internet.

Warning: software is packaged by weight not volume, contents may settle on shipping

From Farber's IP list: EULAs have come to malware. So much for honor among thieves.