Cranial Darwinism

Just give up and fight, already

2011-09-04T08:22:00.000-07:00

Some sort of local tipping point as people read about the ozone decision - now even my friends are fed up with Obama.

The first year or so of his presidency wasn't surprising if you'd read his books. If you look at the policies and discount the pretty rhetoric, he's not especially liberal - about where the Clintons are. And sure, he's avoiding the angry black man thing. And, sure he's prioritizing in a bad economy, and so on. And most importantly, he really believes strongly in the importance of re-establishing a center in American politics. And I agree, that's a great goal.

But that ain't gonna happen, and he should have figured this out by now. Instead, he keeps moving rightward, leaving the left stretched thinner and thinner behind him ... and every time he does, instead of moving in to meet him part way, Team Red just scuttles further away to the right.

NELL gets political

2011-02-04T06:39:00.000-08:00

Tom Mitchell's "Never Ending Learning System" is set up to Tweet a random selection of facts is extracts from the web. Recently it tweeted "I think "Barack H . Obama" is #American"... poor system doesn't know when it's getting into controversial issues.

(H/T Burr Settles)

Rightwing bloggers admit: no, we really don't understand statistics?

2010-02-14T13:30:00.000-08:00

I have to admit, I'm fascinated by the dynamics of how information is pushed around the Web. The academic work on this seems to mostly ignore that what people link to is dictated by what they want to believe.

Today's Memeorandum is dominated by one story, which started with a BBC interview where Phil Jones took a bunch of questions from climate skeptics. One was this one:

B - Do you agree that from 1995 to the present there has been no statistically-significant global warming?

Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level. Achieving statistical significance in scientific terms is much more likely for longer periods, and much less likely for shorter periods.

Rough translation: we're slightly less than 95% confident there is warming in that period. Alternatively, there's a little more than 1 chance in 20 that there is not warming 1995-now.

Then a UK paper, Mail Online, published a report on the interview, with the headline - apparently based solely on the paragraph above - Climategate U-turn as scientist at centre of row admits: There has been no global warming since 1995 - a remarkable interpretation. And this is the discussion on the blogs now, via Memeorandum Colors (click to expand):

One or two of these are pointing out that the Online Mail headline is misleading - but most seem to be just repeating it uncritically ....even though the BBC interview is online, just a google search away, and even though a quick read of even the text Online Mail story shows that the headline is misleading.

Free Markets and Healthcare

2009-09-14T16:51:00.000-07:00

After a bunch of Facebook comments back and forth an old friend of mine pointed me to this article...which presents a more market-oriented discussion of possible reforms.

It's interesting, and raises a number of good points. Goldhill's comments on the inefficiency of insurance are, of course, also the selling point for single-payer:

for every two doctors in the U.S., there is now one health-insurance employee—more than 470,000 in total. In 2006, it cost almost $500 per person just to administer health insurance. Much of this enormous cost would simply disappear if we paid routine and predictable health-care expenditures the way we pay for everything else—by ourselves.

Since these costs also go away if the insurance system is simplified. And many of the issues he discusses - IT inefficiencies, study of treatment effectiveness, etc - are in some of all of the current healthcare proposals.

But there some places where Goldhill seems off base - viewing things through a very middle-class eye. The essence of his proposal is savings accounts for health expenses, plus catastrophic insurance with a high cutoff - the current typical $2000 cutoff moved up to $50k or so. But those numbers probably soundsquite different for you and I than for most people - in the US the median household income in the US is about $50k. In fact there have been pre-tax savings accounts for a long time and I don't see they've been much of a help in limited costs.

More fundamentally, I have two misgivings about the approach. Goldhill starts off with philosophy:

based on my own work experience, I also believe that unless we fix the problems at the foundation of our health system—largely problems of incentives—our reforms won’t do much good, and may do harm. To achieve maximum coverage at acceptable cost with acceptable quality, health care will need to become subject to the same forces that have boosted efficiency and value throughout the economy. We will need to reduce, rather than expand, the role of insurance; focus the government’s role exclusively on things that only government can do ... and rely more on ourselves, the consumers, as the ultimate guarantors of good service, reasonable prices, and sensible trade-offs between health-care spending and spending on all the other good things money can buy

Well and good, I suppose, but it's pretty dangerous to start with philosophy rather than actual data about what works and what doesn't. There are indeed some things governments do well, and some things they don't. But a hundred years ago, private fire companies were popular - but the US has long since gone to a government-funded model for that. If you look at the rest of the first world, every single other country has more government involvement than we do--and our current model is hardly free-market, being based heavily on tax incentives with corresponding regulations for employers. And the vast majority spend less and get results that are better than the US on most measurable dimensions.

I believe the flaw in Goldhill's free market argument is illustrated well be his examples. For Lasik surgery, an elective surgery that is seldom covered by insurance, prices have come down over the past few years, in response to market pressures. For his poor Dad's fatal infection, the final bill has an astronomical $600k+ (at least on paper). But people are just not that likely too shop around for the best price in that kind of a medical emergency - due to some combination of pain, panic or unconciousness - so here the hospital is more like a fire company than a Lasik-surgery beautification clinic.

My guess is that expenditures for these sorts of major-illness emergencies swamp the expenses that would be more sensibly allocated with better incentives. (Data, anyone?)

Goldhill admits "Some of the ideas now on the table may well be sensible in the context of our current system" and "These ideas [i.e. in Goldhill's essay] stand well outside the emerging political consensus about reform"...in other words, he knows he's proposing a fairly radical solution - one that hasn't been tried elsewhere. Honestly, unless you're a philosopher, who would want to take a chance like that? A more rational way of introducing more marketplace into the system would be to try some incremental ways to get more competition, say in a single US state...maybe Massachusetts?...and see how they work before trying them out at scale.

In fact I suspect that a workable government-funded portable the health care plan would be a huge boon to the free market. When a former startup I worked for went bust, I had the delightful experience of needing to find insurance for my family in a hurry (nine days, I still remember) - with no COBRA or any other backup plan to lean on. That experience was been a significant factor in subsequent job decisions I've made. It's a huge distortion of the job market when smaller companies have this sort of handicap to deal with in competing for hires. Pushing for a "free market" approach where it's not appropriate in health - at the expense of biasing the job market against startups and small companies - does not seem like a sensible tack to take.

CMU ahead of the curve again

2009-06-07T08:37:00.000-07:00

I just listened to Obama's speech in Cairo from last week: with all the noise about it, I haven't heard anything posted about what I thought was the most interesting bit:

And this is important because no development strategy can be based only upon what comes out of the ground, nor can it be sustained while young people are out of work. Many Gulf states have enjoyed great wealth as a consequence of oil, and some are beginning to focus it on broader development. But all of us must recognize that education and innovation will be the currency of the 21st century -- (applause) -- and in too many Muslim communities, there remains underinvestment in these areas. I'm emphasizing such investment within my own country. And while America in the past has focused on oil and gas when it comes to this part of the world, we now seek a broader engagement.
On education, we will expand exchange programs, and increase scholarships, like the one that brought my father to America. (Applause.) At the same time, we will encourage more Americans to study in Muslim communities. And we will match promising Muslim students with internships in America; invest in online learning for teachers and children around the world; and create a new online network, so a young person in Kansas can communicate instantly with a young person in Cairo.

On economic development, we will create a new corps of business volunteers to partner with counterparts in Muslim-majority countries. And I will host a Summit on Entrepreneurship this year to identify how we can deepen ties between business leaders, foundations and social entrepreneurs in the United States and Muslim communities around the world.

On science and technology, we will launch a new fund to support technological development in Muslim-majority countries, and to help transfer ideas to the marketplace so they can create more jobs. We'll open centers of scientific excellence in Africa, the Middle East and Southeast Asia, and appoint new science envoys to collaborate on programs that develop new sources of energy, create green jobs, digitize records, clean water, grow new crops. Today I'm announcing a new global effort with the Organization of the Islamic Conference to eradicate polio. And we will also expand partnerships with Muslim communities to promote child and maternal health.

This was the closing portion of the speech, so maybe everyone else flamed out with indignation at something or other before it...but it was also by far the most concrete proposal. Sounds like CMU in Qatar is way ahead of the curve here.....

Google squared

2009-06-05T11:50:00.000-07:00

Matt Hurst published some results today of some of his own experiments with Google^2. It's kind of interesting to compare Google^2 to SEAL, which has more limited goals (it doesn't try and find attributes). I don't think either one is clearly superior, from my trials, but it seems like SEAL does quite well comparatively.

Matt's first query, "small boats" gives mediocre results for Google^2, but SEAL gives

#	Found Items
1	Kayaks
2	Dinghies
3	Kayak
4	Catamarans
5	Canoe
6	Sunfish
7	rowboats
8	Sailboats
9	Laser
10	Daysailers

which is quite decent (modulo kayak and kayaks being the same). "Sailing boats" gives spottier results in SEAL:

#	Found Items
1	Yachts
2	Catamarans
3	Catamaran
4	Dinghies
5	Sailing Yachts
6	Beneteau
7	Mega Yachts
8	Dinghy
9	Yacht
10	Optimist

SEAL's result for "netbooks" is great, except "linux" creeps in somewhere in the top 30 results...

#	Found Items
1	Samsung NC10
2	Acer Aspire One
3	Lenovo IdeaPad S10
4	Dell Inspiron Mini 9
5	HP Mini 1000
6	Asus Eee PC 901
7	asus-eee-pc-1000h
8	MSI Wind
9	MSI Wind U100
10	Dell Inspiron Mini 12
11	asus-eee-pc-900
12	Toshiba NB100
13	samsung-n120
14	HP Mini 2140
15	asus-eee-pc-s101
16	asus-eee-pc-1000he
17	msi-wind-u123
18	Samsung NC20
19	sony-vaio-p
20	archos-10
21	hp-mini-note-2133
22	Asus Eee PC 1000
23	Samsung N310
24	Linux
25	MSI Wind U120
26	Samsung N110
27	ASUS Eee PC 1002HA
28	Asus Eee PC T91
29	Asus Eee PC 900A
30	Dell Inspiron Mini 10

"Scottish regions" gives mostly cities in SEAL (like in Google^2), to the extent that I can recognize what it gives. Matt's next queries are:

Test 5: ‘british overseas territories’ – SEAL gives a very reasonable list, as does Google^2, according to Matt.

Test 6: ‘plants native to the pacific northwest’ – Matt rates Google^2 as "very poor {gardening, botany, Hardcover, Your account, Add your first tag, Share your own customer images, paperback)". SEAL starts with "Trees, Plant nursery, Washington" and then goes downhill....but it at least knows that it didn't find much and tells you that the list has low confidence.

Test 7: ‘novels of o’brian’ – excellent results from Google^2, and mediocre low-confidence results from SEAL.

Test 8: ‘jedi masters’ – Matt's comments for Google^2 is "oops {Obi-Wan kenobi, Star Wars: The Clone Wars, Mace Windu, Star Wars Episode VI:return of the jedi, Jedi Temple, Kit Fisto, Younglings}". SEAL gives

#	Found Items
1	Mace Windu
2	Obi-Wan Kenobi
3	Luke Skywalker
4	Yoda
5	Qui-Gon Jinn
6	Ki-Adi-Mundi
7	Shaak Ti
8	Jedi Council
9	Plo Koon
10	Saesee Tiin

Nice, except for #8, but why is Mace-Windu first? still, this is pretty good.

Test 9: ‘movies of brad pitt’ – Google^2 gets "not bad, a reasonable list of movies with Director and Language fields. Note that it doesn’t attempt to normalize ‘USA’ and ‘United States’ suggesting that the system uses very superficial representations internally." SEAL gets poor results, with a low-confidence warning.

Test 10: ‘progressive rock bands’ – Google^2 gets a "fail" from Matt. SEAL gets

#	Found Items
1	Yes
2	Pink Floyd
3	King Crimson
4	Genesis
5	Rush
6	Jethro Tull
7	Gentle Giant
8	Dream Theater
9	Jazz
10	Van Der Graaf Generator
11	Marillion
12	Camel
13	The Beatles
14	Porcupine Tree
15	Procol Harum
16	Kansas
17	Frank Zappa
18	Led Zeppelin
19	Magma
20	Art rock
21	The Moody Blues
22	Deep Purple
23	Tool
24	Tangerine Dream
25	The Mars Volta
26	Punk rock
27	Progressive metal
28	Classical music
29	The Wall
30	Symphonic rock

FiveThirtyEight.com: Politics Done Right: The Two Progressivisms

2009-02-16T07:09:00.000-08:00

I became a big fan of Nate Silver during the election, then stopped reading him for a while after it. My loss actually, since he's just as insightful when he's not crunching poll numbers, eg with his latest post on The Two Progressivisms.

World's least informative superbowl play-by-play

2009-02-05T05:57:00.000-08:00

Yet, strangely compelling....

Lessons to learn from the financial crisis

2009-01-04T09:02:00.000-08:00

I just finished reading a great NYT article on the financial crisis and what they call "the strange story of Harry Markopolos... who, for nine years, tried to explain to the Securities and Exchange Commission that ... Madoff’s investment performance...was mathematically impossible." Markopolos wrote a 17-page report in 2005(!) crunching the numbers and arguing the most likely case was, quote, that "Madoff Securities is the world’s largest Ponzi Scheme.” Which of course it was.

In April 2008 Markopolos tried again, with an e-mail message to Jonathan S. Sokobin the new head of the commission’s office of risk assessment. “Attached is a submission I’ve made to the S.E.C. three times in Boston,” he wrote. “Each time Boston sent this to New York. Meagan Cheung, branch chief, in New York actually investigated this but with no result that I am aware of. In my conversations with her, I did not believe that she had the derivatives or mathematical background to understand the violations.”

The NYT authors are all over this:

How does this happen? How can the person in charge of assessing Wall Street firms not have the tools to understand them? Is the S.E.C. that inept? Perhaps, but the problem inside the commission is far worse — because inept people can be replaced. The problem is systemic. The new director of risk assessment was no more likely to grasp the risk of Bernard Madoff than the old director of risk assessment because the new guy’s thoughts and beliefs were guided by the same incentives: the need to curry favor with the politically influential and the desire to keep sweet the Wall Street elite.

Maybe their analysis is right - but personally, I'm not sure that wishful thinking and incompetence are the whole story. So much of life has become so complex that it's poorly understood, and we seem to mostly accept this. Case in point: I have a PhD in computer science and about 25 years experience, and I've done everything from soldiering together microprocessors to writing compilers to proving impossibility theorems. Yesterday I spent 45 minutes fixing the wireless on my laptop - and I don't understand why what I did worked. I could in principle, I'm sure - but that would take even more time, which I'd rather spend doing other stuff.

So I can readily imagine the director of risk assessment saying: gosh, this is complicated. I don't understand it very well. Some experts say it's a problem, some say it's not...hmm, what should I do? What should he do? if you're in this situation every week, or every month, how do you decide what to dig into, and what to ignore? How do you tell the difference between what you think is true, and what you hope is true? How many "experts" are in this situation every week, or every day?

Today's thought is: how should you handle complexity of this sort? because there is a right way and a wrong way. If I was to boil the right way down to a few words, they would be:

Study the data. Always start with what you know - the data. Plot it, graph it, run your favorite models on it, look at the outliers. After you've hacked at it for a while, you'll start to get a feel for what it's like. Madoff was an outlier in several respects - rate of return for the last few years, but also organizationally. If you're interested in modeling the market, you'd want to discount him as representative - if you were interested in detecting fraud, outliers are the things you want to watch most closely.
Do the math. Complexity is your enemy, and good math is your best friend and most powerful weapon against it. (There is of course evil math, which makes the simple complex instead of making the complex simple, but hopefully you're smart enough to recognize the difference.)
Ask questions. When you don't understand something, ask questions until you do. If you do understand something, think it through a little more, until you find a question...you'll usually find out that either you understand it a little better after you do...or sometimes, that nobody really understands it.

The great thing about these lessons is that you can learn exactly the same things from so many recent experiences...anyone for discussing Iraq or global warming?

Today's deep thought is: how much of modern computer science is about making the complexities in our world simpler and more understandable - and how much is making the world more complex and more opaque? And how complicated can a system be and still be controlled, maintained, and regulated by human beings?

The Last Veridian Note

2008-11-20T16:59:00.000-08:00

On sabbatical this year at Google, I keep noticing how little of my old office space I've taken with me, and how little I miss it. As Bruce Sterling points out: "objects can no longer protect you from want, from humiliation – in fact they are causes of humiliation, as anyone with a McMansion crammed with Chinese-made goods and an unsellable SUV has now learned at great cost." Almost makes me feel like cleaning up at home...

More data on tax policies

2008-10-26T09:06:00.001-07:00

Above is yet another graph, using data from the same source - a simple one just showing the average tax rate in each income bracket: the bottom 20% of the population, the next 20%, and so on, up to the top 0.1%. The famous "Joe the plumber" is just at the around the 95% percentile - at the top end of that long, long range (from about the 40 percentile mark to the 90 percentile mark, $40k/yr to about $170k/yr) where the tax policies of the two candidates are pretty much the same. Or if you prefer, at the bottom of that long range (from $250k/yr to $2,800k/yr and up), where the policies are different.

I find it exceedingly strange that there's so much rhetoric about very small differences (eg for Joe the plumber's tax rate) and so little discussion about the fairly large differences in tax rates for the top end of the range. The idea that tax credits are socialist is just wacky - these have been pushed by conservatives for decades now as an alternative to heavier-weight social programs, and in fact McCain's health care plan is based on tax credits. And both plans are clearly progressive in the sense that higher incomes pay higher tax rates - the difference is only in degree, not kind.

Even if you accept the claim that the very top of the scale for tax rates is important, the difference in top tax rate is also not especially large in a global sense - in particular moving from 29-38% doesn't turn the US from a free-market bastion to a socialistic Sweden wanna-be. In act, it doesn't dramatically change the ranking of the US for top personal income tax rate. Look at this chart (based on 2005 data) from Wikipedia, and imagine moving the top US bracket from 28% or so to 38% or so:

Or if you prefer here are some selected values from the data tables on the WP page that I sliced out and graphed myself against the Obama and McCain proposals. It's clearly a difference...but it's just as clearly not a switch from capitalism to Marxism.

More visual comparisons

2008-10-20T17:11:00.000-07:00

From Mark Thoma at The Economists View, here's a pie chart showing earmarks as a percentage of the total Federal budget.

..and here is offshore drilling production, by year, as a fraction of total oil consumption.

What you miss when you go to bed early

2008-10-16T06:02:00.000-07:00

Another tragic victim of socialized medicine....

My brilliant idea of the week...

2008-10-14T17:11:00.000-07:00

...how about announcing a concert that hasn't happened yet, instead of one that's over? Our next show is a

Musical Open House for Obama featuring folk / blues / ragtime / bluegrass band Smokestack Lightining & friends.

Sunday Oct 26, 7:30 - 9:30pm
at the House of Nick Coles and Jen Matesa
331 S. Fairmount Street, Pittsburgh 15232

No cover - just donate what you can if you appreciate the music!

RSVP at the snazzy Obama-provided event page or listen to some samples on myspace if you're not one of those "low information" listeners.

Visualizing the Long Tail for Tax Policy: Part 2

2008-10-09T14:45:00.000-07:00

Above is a corresponding visualization of the Obama plan - but it's a little more complicated. Again, you can click to zoom in.

Obama's plan includes both tax cuts and tax increases. All of Obama's tax cuts go to the bottom 900 people (the poorest 90%). The total cuts for the bottom 90% is much larger than for McCain's plan, but it's also distributed quite differently, as the inset shows.

For the next 90 people (percentiles 90-99%) it's basically a wash - the bottom half of these will get a tax cut, but a very slight one, and the top half of those 90 will see an increase: in terms of field position, they'll take a penalty, and have to move back to the left. How far, you ask? On average across these 90 people, the penalty is about 6 inches each - not enough to see on the gridiron I drew.

For the next 9 people, in the 99-99.9% bracket, the average tax increase is big enough to be visible: about a 4 yards penalty. An offsides penalty, maybe. I drew the approximate shift of each of these 9 people on the graph, with dotted blue arrows pointing from the old to the new position.

For the last guy, the penalty is nearly 22 yards, moving him back to the 70 or 75. Maybe a holding call.

I didn't bother to draw the appropriate-size stack of bills for the tax increases- you'll just have to image it. I didn't draw field-position arrows for the McCain tax plan because nobody moves by more than a foot or so - except for the guy down by the 94-yard line, who just about makes it to the goal.

I should also probably point out (in case it's not obvious) that neither of these plans is likely to be implemented exactly in their proposed form - even if the economy and stock market hadn't implored the last couple of weeks, which has obviously changed things, there's a long process of digestion before any major legislation is produced by congress, and like digestion, what comes out is often quite dissimilar from what goes in. It's probably best to think of this as a statement of what values the candidates stand support, rather than a prediction of how your taxes will change.

So to summarize:

The Obama plan makes the spread between the highest-paid and lowest-paid Americans noticibly smaller. The McCain plan makes it slightly larger.
The bottom 50-60% of the country by income do much better under the Obama plan).
The tax plans are not very different for the 60-95% bracket, between about $70k and $170k per year.
The 95-99% bracket ($170-$240k) clearly does better under McCain - by a substantial amount in dollars each, but a small percentage (a little more than 2%, actually) in income.
The top 1%, and especially the top 0.1%, do much much better under McCain.

Go Steelers!

Visualizing the Long Tail for Tax Policy

2008-10-04T19:00:00.000-07:00

They say software is hard to understand because it's "invisible" - but aren't so many things? For fun, I downloaded the raw data on Obama and McCain's proposed tax policies from http://www.taxpolicycenter.org - they have it all in Excel form, which makes crunching it convenient and easy - and tried to construct what I considered a reasonable visualization. Their data is broken out to show the effective tax increase or decrease for the first quintile, the second quintile, etc, with the top quintile broken down to partitions of the top 10%, 5%, 1% and 0.1%. I wanted to show the range of salaries in each partition, the number of people in each partition, and the change in tax. I decided that it just can't be done - the ranges are just two broad to see on, without using hard-to-explain constructs like a loglog plot.

Here's a mental image that might help. Think of a football field, 1000 typical Americans, and some big stacks of money in small bills. Specifically, think of $1 and $5 shuffled together and piled up so that each pile contains about equal numbers of $1's and $5's. If I did the math right (they say a US bill is 0.043" thick) then a money stack one foot tall is worth about $10k - so the football field is $3M long and $1.6M wide, I'm about $57k tall, and my waist size used to be about $30k (but with my diet, I lost about $1600 around the middle).

Now to visualize income distribution, let's take those 1000 typical Americans, and let them walk down the gridiron like this: start at one goal line, and for every $10k of yearly income, walk one foot toward the other end zone...so if you make $90k/year, e.g., you'll end up on the three-yard line, and if you make $300k/year, you'll end up on the 10. Subject to that rule, space everyone out as much as you can.

The first 900 of those 1000 people will end up somewhere before the 4 yard line, because about 90% of Americans make about $120k or less.

Another 90 people will end up between the 4 yard line and the 8 yard line - that is, they make between $120k and $240k per year.

Another 9 people will end up between the 8 yard line at the 20 - they make between $240k and $600k per year.

That accounts for 999 out of 1000 people, but this is America, land of Bill Gates and Warren Buffett - so we're not done yet. The last guy will stand, on average, at the 94 yard line - with an average income of $2.8M per year. Remember this is income, not life savings.

Now, for the tax cuts. Let's start with McCain's. EIn his plan, everyone gets a tax cut, and we'll hand it out as stacks of money. McCain has a stack about 9 feet high to be shared by the first 900 people; a stack about 5 feet high to be shared by the next 90 people; another 9.5 foot stack to be shared by the next 9 people; and a final stack about 6.5 feet high - about the height of Ben Rothlisberger - that goes to that one lonely old man (surely, he's old?) down on the 94 yard line.

Above is a picture I drew of this...you can click to zoom in.

This is approximate, and loses some detail - which maybe I'll get to in another post - but it illustrates the main properties of McCain's plan: there is an extremely long tail in income distribution in the US, and as a consequence, an extremely long tail of tax-cut returns. If you imagine the money stacks are all the same (which is order-of-magnitude correct), then

For the bottom 90%, the per-person tax cut is about 1/10 that of people in the 90-99% bracket.
For the 90-99% partition, the per-person tax cut is about 1/10 that of people in the 99-99.9% bracket.
For the 99-99.9% partition, the per-person tax cut is about 1/10 that of people in the top one-tenth of one percent.

"I shot a moose once"

2008-10-04T08:53:00.000-07:00

And now, an informational message, for those who may not fully recognize the intricacies and dangers of moose hunting.

The Sarah Palin Turing test

2008-09-30T11:48:00.001-07:00

I was wondering .... was it just me, or did anyone else have the reaction "Gosh, her answer sounds like it was generated from a statistical language model trained on a corpus of random Republican talking points?"

Snazzy Presidential Debate Search Engine

2008-09-28T11:13:00.000-07:00

Available at the NYT.

Summary of the morning's news...

2008-09-26T05:47:00.000-07:00

Sometime last night, the bailout became no bailout. Obama, via the NYT:

“What I’ve found, and I think it was confirmed today, is that when you inject presidential politics into delicate negotiations, it’s not necessarily as helpful as it needs to be,” Mr. Obama told reporters Thursday evening. “Just because there is a lot of glare of the spotlight, there’s the potential for posturing or suspicions. When you’re not worrying about who’s getting credit, or who’s getting blamed, then things tend to move forward a little more constructively.”

The Republican side of the story, as told by Kevin Smith, aid to House Republican Leader Boehner:

But a top aide to Mr. Boehner said it was Democrats who had done the political posturing. The aide, Kevin Smith, said Republicans revolted, in part, because they were chafing at what they saw as an attempt by Democrats to jam through an agreement on the bailout early Thursday and deny Mr. McCain an opportunity to participate in the agreement.

The Value of Experience

2008-09-09T16:54:00.000-07:00

The great thing about the web is that if you can imagine a question and if you spend a little time looking you can find often someone that already answered the question. This weekend I wondered: there's all this talk about who's experienced enough to be president - what does the data actually say about how experience correlates with performance? Turns out that they asked the same question at electoral-vote.com and got some interesting answers.

The first scatter plot (or maybe it's the second?) above plots total years of experience against rank for all presidents up to W. Bush, where rank=1 for Abraham Lincoln, and rank=42 for Warren Harding (who?). This is a consensus ranking by historians, so it's somewhat subjective, but there's no apparent correlation between x and y here at all. The second scatter plot (or was it the first?) compares experience to a random permutation of rank. Can you see any difference? I can't. If you're curious, and you compute r^2 for this dataset, then you get a value of 0.008 - essentially nil (an r^2 around 0.5 would be a strong correlation for data of this sort). I poked around in Excel for a while, and the data looks just as uncorrelated if you look at gubernatorial experience, for instance, and a couple of other variants I tried.

So what does that mean? obviously, experience matters for any job, right? Well, one point is that there is a sample bias here - the data is entirely on people that were at least arguably well-qualified for the job, not the result of controlled scientific experiment in which scientists took [wo]men off the street at random and dropped them into the White House. (Insert Sarah Palin joke here if you like). It's wrong to say experience doesn't matter: more accurately the data says that among people qualified enough to be elected, more experience does not correlate with better performance.

An second point: from a machine learning point of view, certainly not all experiences are created equal. Two important properties are the recency and diversity of experiences - for instance, 1,000 samples of poll data collected last week are better than 10,000 samples of polling data collected 50 years ago, and 1,000 samples collected from 50 states is better than 10,000 samples collected from just Utah and Nevada. This data says little about the quality of the experience. Will John McCain's experience as a child and young man in the 1930's and 40's help him be a better president? How much does he actually know now, midway through his third decade in congress, that he didn't know midway through his first?

Other explanations are also possible. Maybe being president is just too unlike other jobs for experience to matter much. Or maybe the Lincolns and FDRs of the world are so remarkable that they rise quickly through the ranks and are still great, while the Buchanans, Garfields, and Van Burens will never be more than mediocre no longer how long they slog along.

Or maybe there's some other explanation.

So what does this all mean? What am I, a pundit? In terms of this week's talking points, neither party scores much - Team Red seems to have all but abandoned the experience-vs-celebrity story and are busy trying to co-opt Team Blue's message of change/new faces. But there's still a lot of noise about how much experience Palin has, and how it compares to Obama's, and I think it's worth noting that a major fraction of the talking-head time this cycle has been (and certainly will be) spent on a point that is a matter of faith, not a matter of objective fact. Once you look at the data, it's clear that simple miles-on-the-odometer experience does not make tend make one a better president - or at least, if a trend exists, it is extremely small.

politics shmolitics, as long as it swings

2008-08-25T08:12:00.000-07:00

A clever YouTube video produced by my good friend Judy Minot.

Another performance by Smokestack Lightinin'

2008-07-26T17:55:00.000-07:00

This one was at the bizarrely inspired 1st Annual Alexander Berkman Festiv al, and was captured in full audio glory by Er1k Riebling (thanks Er1k!) The whole set (20min) consisted of:

Upcoming gig

2008-05-18T07:34:00.000-07:00

I'll be playing (mostly mandolin) Tues night in Braddock, with the band Smokestack Lightenin' for a fundraiser for Unseam’d Shakespeare Company’s Out of this Furnace: project. The headliner is Anne Feeney.

Where: Elks Lodge #883- 424 Library Street, Braddock, Pennsylvania
When: Tuesday May 20, 2008 7:00 PM-11:00PM
Tickets: $25 (or pay what you can), available at the door (cash/check only)
Contact: Tim Dawson, 412-621-0244 for directions and information

We have violated the prime directive

2008-05-16T11:10:00.000-07:00

Noah Smith and I are co-supervising Tae Yano on a project involving analysis of political blogs, and Tae left a pile of results and code on her CMU web site as a way of communicating with us...world-readable. Surprisingly someone at one of the blogs she spidered, Little Green Footballs, actually noticed, leading to a lot of investigative work in this fascinating thread:

Anyone know what this page at Carnegie Mellon means? It’s some kind of experiment that involves comments posted at LGF, and I have a feeling it’s not friendly.
...
comment #5: Maybe it's post-modern poetry, academia-style? Using LGF comments as gibberish to transcend interpretation?
...
comment #18: Obviously it's a KGB plot.
...
#12: there's a Kos directory too. Probably just too what they consider to be partisan sites to test some kind of language algorithms or something
...
#20: I think they are doing a text analysis looking for Charles' sock puppets -- like the way some trolls accuse Robert Spencer of posting as Hugh
...
#23: Looks like they are doing another election-year study looking at patterns of argument or other aspects of communication.
...
#47: Who ever thought that garbage labeled as "research" could be so sophisticated and at the same time be so worthless. ... Polishing farts to get Piled higher and Deeper has become, for the most part, an ephemeral exercise in self deception. [ouch! sounds like my last grant review].
...
#48: This isn't that advanced of stuff - we are working on a simillar concept for internal corporate communications. This has been around a while.
....
#60: EXPERIMENTATION WITHOUT REPRESENTATION! I demand compensation!
...
#61: I wonder if linking to their experiment will introduce fatal feedback loops, leading to a disturbance in the Krell mind-field and possible space-time discontinuity?
...
#79: It looks to my un-Pythonesque eye like a programmer is using the cosine function to create some kind of mapping on how comments refer to one another based on comment location in the thread; and looking at different types of postings from Charles (open thread, breaking news, news outlet criticisms) to see if there are any trends in the discussions that vary according to the posting type. That has to be one of the worst run-on sentences I've ever written.
...
#92: [#64: "What is port sniffing?"] after you pour some in the glass, but before you take a sip, swirl it around a bit and inhale whist the dark chocolate is melting on your tongue......
...
#162: Ahh, so basically it's a merger between post-Marxist deconstructionism and THX-1138-esque Orwellian personality dehumanization.
...
#172: I liked the method of withdrawing all posts by 204 commenters, then asking the system to predict what those posts were (or *if* the post existed, I think) based on all the rest.
...
#187: ok here's how I see it. There is a whole group at Carnegie Mellon CS department doing research in Artificial Intelligence by running statistics on blogs. A grad student Tae Yano stashed her research files on a server and left them wide open to WWW. The files pertain to running basic stats on DK and LGF, up to cosine similarity. The AI goal would be to have a blog-commenting software indistinguishable from a blog-commenting human. For all I know, this, or any other comment, is already written by a python script.
...
#277: [Quoted from a grant proposal Noah & I wrote that got unearthed, along with various other papers, pictures of Noah's cats, Tae's cv, her picture, her programming project on knitting, wedding announcement, etc:] "Political text is often indirect, sarcastic, repetitive, hyperbolic, emotional, biased, manipulative, and riddled with unstated assumptions." - Byte me.
...
#314: So some students are doing research on blogs. No big deal. It was interesting to see what it was about, but that's about it. I don't think they feel the need to protect their files. Why should they? Who would want to mess with that? It's just a school project, I don't see point of "exposing" the students and publicizing their information. At least that's my take.
...
#316: I noticed that the research group to which she belongs is partially funded by a DARPA grant.
...
#361: And the same techniques/analyses will probably work on Arabic-language websites, too.
...
#428: Are terrorists ‘phone banking’ for Barack?
...
#474: magine a natural language program that could respond to comments with charm and style, sort of a robo-blogger. Now imagine an army of them, all set to monitor a different political blog, run by a campaign manager for a politician. Add to its writing ability an encyclopedic memory, with instant access to famous quotes, historical facts, trivia, statistics, and every word ever uttered by the opposition. You now have an army of ultimate bloggers, all completely under the control of one campaign manager... no more "going off message" by some underpaid/volunteer lackey, just high quality counter-opinion, ready to be inserted into the blogs of anyone who disagrees with your candidate. This research will eventually lead to robo-blogging to kill emerging scandals and alternative opinions on issues... no more Rathergates as they will be smothered in the cradle by the most charming bloggers around -- the poli-bots.
...
#482: She's trying for a data-mining tool tailored for blogs that separates "useful, thoughtful" information from all the mindless dreck in the blogosphere. Lotsa luck with that! As an aside, I've been on the Carnegie Mellon campus, toured the Computer Science department, and met with CS faculty. It's a gorgeous campus. The school clearly has big bucks. CMU holds numerous contracts with various government agencies related to the information technology aspects of defense, computer security, homeland security, and similar "black ops" topics. At least some people on that campus have intimate access to NSA, DOD, and CIA. It's a spooky place.
...
#489: We should, for amusements sake, keep an eye open for a KOS diary about this. There may be some entertaining histrionics and conniption fits over their being the unwitting subject of DARPA funded research.
...
#492: But but but... Markos is a CIA agent.... dKos is a DARPA funded research project....
...
#496: Don't spill the beans. The Koslings haven't figured any of that out yet. Agent Markos will have a rough time of it when he is exposed as being a double secret agent of the Zionist conspiracy. Don't blow his cover.
...
#497: From this, we conclude that LGF not only has MORE numbers than DKOS, but BIGGER numbers as well. If YOU TOO want bigger numbers, choose LGF brand blogs.
...
#514: Was just thinking that all you people have too much time on your hands...but then it occurred to me I'm sitting here reading all this.
...
#518: Looks like the whole thing has gone 404. My guess is that she just wanted a corpus of data for some programming project and, now that the object under study is aware of her, it's no longer useful. I don't see any dark purpose here. How evil can someone be who writes knitting software?
...
#519: I suspect that if the mice KNOW they are in an experiment, they will not produce the same results,as they would otherwise, thus invalidating the experiment.
...
#520: [re: #21 zombie: "Talk about pointless. People get PhDs for this crap."] Not really pointless. The "value" of this may be questionable though, at this point in time. As the web and bogosphere has grown exponentially in influence, there has been great interest in determining if real life outcomes can be affected by influential opinions posted on blogs -- and then re-created in numerous other places to mimick majority viewpoints. There are numerous companies invloved in this research-like activity which can be tailored for marketing, business intelligence financial trading, political campaign managment uses (for example, Umbria of Denver, CO was just sold to JD Power). They use primarily data visualization tools (like those used now by lawyers engaged in electronic data discovery) -- These are similar to the software tools used by intelligence services to monitor, track, analyze, for example, wireless (web/phone) transmissions and discussions originating in the US, and destined for overseas delivery in places like, say, Iran.
...
#531: Yea- shut down. I didn't get to it in time to even see what it was all about. So, now I'm depressed. And stuff.

Since this morning, Tae chmod'ed all her code to hide it, but I suggested to Tae that she keep it visible, since folks were having so much fun with it. (Of course now she's embarassed and wants to clean up her code first...) Noah and I also wrote an open letter to LGF explaining what was happening.

I'm mostly amused (as you can tell), but also impressed at (1) how much was uncovered about Tae, Noah and me and (2) how much of this obscure statistical NLP code the LGF community was able to figure out, communally. The peril and power of the internet.

Warning: software is packaged by weight not volume, contents may settle on shipping

2008-05-05T07:13:00.000-07:00

From Farber's IP list: EULAs have come to malware. So much for honor among thieves.

Farecast and Fonolo

2008-05-01T08:41:00.000-07:00

From Dave Farber's IP, news of a new company called Fonolo: a ‘Google’ for phone menus. The basic idea is simple: spider all those annoying phone menus and make every point inside one a URL, which you can reach by having a bot navigate to that point for you, and then call you when it gets there. From the video it sounds like the spidering is done manually, and the navigation is done using ASR to check progress (since the menus change frequently).

Like Farecast this is a nice example of using AI to solve consumer problems, by mining information that is, in principle public (in one case airfare prices, in the other phone menus), but in practice, inaccessible. And like Farecast, it's not at all obvious that the companies controlling that information will be happy about that mining - especially if people end up using it as a souped-up version of GetHuman.

This feels like the thin edge of a wedge to me. I predict that within five years there will be open AI warfare between consumer-oriented bots and corporate-controlled information...then again, maybe this is just wishful thinking. After all, I'm in the AI munitions business.

Classifiers in the news

2008-04-24T06:51:00.000-07:00

Here's a first for me - a blog post consisting of an actual decision tree. It classifies counties as going Clinton vs Obama. Wonder what the weights for a linear classifier would look like?

Somehow this seems more appropriate the day after

2008-04-23T06:54:00.000-07:00

I was hoping for Obama to have a knockout win in Pennsylvania, just to end the madness...but no. Apparently, as the Three-Toed Sloth says, the voting will continue until morale improves.

Oh well, at least the telerobots will stop calling my house for a while....

Valuable resource for sentiment analysis?

2008-04-17T05:32:00.000-07:00

I didn't make it through last night's debate but it's an ill wind...one result seems to be a list of 11000+ comments that appear to be almost entirely negative in tone.

Sunday Forum: Nation of bridges

2008-04-14T14:35:00.000-07:00

If you like Pittsburgh, baseball, Michael Chabon, and Barack Obama - and who doesn't? - there was a lovely essay in my Sunday paper: Nation of bridges

Sterling interview

2008-04-04T10:19:00.000-07:00

Via Dave Farber's "Interesting Persons" list, an interview with Bruce Sterling on Life, the Internet and Everything, in which he discusses the role of colleges

A college is supposed to civilize the young. If you remove those functions merely in order to pump ones and zeros at student eyeballs, you are going to lose a lot of valuable cultural capital. An educated citizen is not a friction-free technocratic philistine myrmidon with an ISO rating. Those guys exist, don't get me wrong, but they bear the relationship to education that the Ron Paul campaign had to political reality.

and himself

They say that those who worship the Muses end up running the Museums. Similarly, all futurists are doomed to become historians. The avant-garde becomes the gray eminences...

2008-04-01T16:17:00.000-07:00

After hearing the David Evan's excellent talk "What Elements of an Online Social Networking Profile Predict Target-Rater Agreement in Personality?" I now know there there are three things I could tell you about that would (roughly speaking) help you more reliably assess my true personality. Here they are....

Something I'm proud of.
Something I'm kind of embarrassed by.
A humorous video.

They Told You Not To Reply

2008-03-23T16:19:00.000-07:00

Via Dave Farber's IP list, how email leaks have helped raise $5000 for Seattle-area dog pounds:

They Told You Not To Reply

The professor sings the blues

2008-03-19T13:56:00.000-07:00

By request....(and thanks Richard for the video!)

The Keys to My Heart

2008-03-12T07:36:00.000-07:00

...should use at least 128 bits, I'd say. Via Dave Farber's IP list - and giving a new meaning to the phrase "security for embedded systems": A Heart Device Is Found Vulnerable to Hacker Attacks

Cut Once (Again)

2008-03-07T05:57:00.000-08:00

For those that want to try out the latest learning technology in Thunderbird - the Cut Once Thunderbird Extension now has a spiffy new home page. And before you ask, no, I have no idea why the parrot. You'll have to ask Vitor or Ramnath.

Joining the 21st century

2008-03-05T09:08:00.000-08:00

I certainly don't publish enough for it to be necessary, but in an idle moment I used Dapper to wrap my publication list an build an RSS feed for it. As someone that worked on this a few years ago, I'm impressed - wrapping the page took only about 5 minutes, and it was quite easy to use. I didn't even really pay attention to the "documentation" (which is a Flash video).

Greenhous gases - doing the math

2008-03-03T14:06:00.001-08:00

Three times this winter I've turned the crank on my 1994 Ford Escort and had it not start right up. Admittedly once was a dead battery and once was a defective battery but still, I'm considering the reality that one of these days I'll have to overcome my natural lethargy actually replace the thing....so, I've been reading up on new cars...and carbon footprints...and have been trying to work out to what extent getting a hybrid is an effective way to save the planet and to what extent it's just sort of cool.

Somewhat surprisingly, the nicest carbon-footprint calculator I found was at the EPA. According to them, replacing both my cars with something that had better than double the gas mileage would save about 2300 lbs of CO2/year - but I'd save more with a either newer furnace, or with new windows for my house. I'm probably atypical here, as together Susan and I drive about 100 miles a week, but still, I was pretty surprised.

Another data-point came up today in Low-tech Magazine: according to them, at least, the carbon cost of manufacturing the 8m^2 of solar panels needed to power a house is between 60,000 and 940,000 kgs of C02 - and according to the EPA this is the same carbon cost as between 20 and, um, 300+? years of electricity use for an average family of 4. Of course, this could be way off base, but it seems to me that we need to set up either some sort of marketplace mechanisms to allocate CO2 savings efficiently - just following the trends may not get us there.

really, you had to be there

2008-02-27T05:48:00.000-08:00

But, Ralf Brown's pictures of my SCS day performance came out great.

Is 88.80.13.160 the new 09-f9-... ?

2008-02-21T07:03:00.000-08:00

Remember the great Digg revolt of 2007, when takedown efforts for a key needed to decrypt HD-DVD led to numerous ways to publish or conceal said key? (I think my favorite was

09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-bd
09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-be
09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-bf
[redacted]
09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c1
09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c2
09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c3

but that's just me.) From Steven Bellovin via IP, the web site Wikileaks has been disabled due to a court order...by removing the site from the DNS registry. As Steven points out:

Not surprisingly, the NY Times article quoted the web site as noting the similarity of this case to the Pentagon Papers case. The Times also noted how ineffectual the censorship attempt actually was -- not only are there alternate names wikileaks.be, wikileaks.de, and wikileaks.cx -- but the site is still reachable via 88.80.13.160.

State Secret Abuses Come to a Boil

2008-02-04T17:30:00.000-08:00

State Secret Abuses Come to a Boil - an interesting sidelight to my post below on asymmetric information is this story on how secrecy has been abused in the distant past by our government. Of course any abuses in the recent past would still be secret.

Two more papers out of the review tunnel

2008-02-04T14:51:00.000-08:00

Ramesh Nallapati and William W. Cohen (2008): Link-PLSA-LDA: A New Unsupervised Model for Topics and Influence of Blogs in ICWSM-2008.

To appear in April, probably after some minor revisions.

Frank Lin and William W. Cohen (2008): The MultiRank Bootstrap Algorithm: SemiSupervised Political Blog Classification and Ranking Using SemiSupervised Link Classification in ICWSM-2008.

To appear in April, as a poster, in (I assume) abbreviated form.

Kevin Kelly -- The Technium

2008-02-04T14:37:00.000-08:00

Long ago, I remember reading a science fiction story where aliens nearly destroy the world by dumping devices that would copy anything - a book, a flashlight, a hamburger - given an original and a pile of sand (or any other raw matter). This doesn't actually wreck the world economy... in the story...but of course it's fiction. However, given how much of the expensive parts of the world is information, the premise is no longer that far-out.

Kevin Kelly gives an excellent analysis of why free information (or even free goods) doesn't mean the end of the world.

Measure twice, cut once

2008-01-30T06:54:00.000-08:00

My students Vitor and Ramnath have developed a Thunderbird plugin that implements recipient recommendation and leak detection for email. It modifies Thunderbird by adding an additional pane that pops up after you send a message, giving you one final chance to fix any errors in your recipient list.

The plugin is called Cut Once - the name comes from a phrase used repeatedly by those weird Rastafarian guys in Neuromancer. There's a brief writeup on how to use it, but it's pretty self-explanatory: just download it, open Thunderbird, and go to the tools->addon menu to install. After you've installed it, you train by opening your folder of "Sent" mail and pressing the "train" button. (This took about an hour for my 9000+ old messages, which is pretty good for something written in JavaScript.)

Any comments/feedback are appreciated. Or, if messing with your email client is too extreme for you, you could just read Vitor's ECIR 2008 paper, which is the latest one to come out of the review tunnel.

Personal control of personal information

2008-01-19T14:16:00.000-08:00

Recently I've run into a number of posts discussing privacy from a different prospective - noting that while you, as a consumer/citizen, have limited control over where and how information about you is kept, governments and businesses often do try to control how information about their activities.

For instance, Glenn Reynolds recently wrote in Popular Mechanics:

...government officials and big corporations often want to watch us, but they don't want to be watched in return. Shopping malls are full of security cameras, but many have signs at the entrance telling customers that no photography or video recording is allowed. Police cars have dashboard cameras... But try shooting photos or video of police or other public officials as they go about their business and you might find yourself in wrist restraints. ... Under the law, citizens have no right not to be photographed in public places. So why should people who make their living on the taxpayers' dime enjoy greater freedom from public scrutiny than the taxpayers themselves?

An article with a similar tone also appeared in Computer World, which gave a link to a compelling example of how a citizen's records of police actions were evidence of police wrongdoing (NYT headline: "Recorded on a Suspect’s Hidden MP3 Player, a Bronx Detective Faces 12 Perjury Charges"). The author's summary of the situation:

...surveillance in general ... upsets the balance of power. Whoever has the tape has the power to use, not use, selectively use or misuse the information or proof or evidence recorded.

This an interesting line of argument. There's something that seems fundamentally wrong about living in a society where surveillance is "endemic", but maybe the most jarring thing is not the loss of privacy, but the loss of power and control.

With that in mind, there's a very interesting move afoot to set up an open standard to describe user "attention" data - which I gather includes browsing history, mostly, but could also certainly include any other information about what a user is interested in...Netflix reviews, Amazon purchases, search queries, you name it. The hope is to break this data away from the many different sites (each if which controls a piece of it) and put it in the hands of users, who can then go and get purchase recommendations (or what have you) from whoever does the best job.

Not an entirely new idea, but a fascinating one none the less. Moore's law, along with progress in collaborative filtering/machine learning techniques, means that the barrier to being able to save this sort of data and do something interesting with it is just going to keep dropping, so one can certainly imagine a more horizontal market for recommendations opening up over the next few years. I don't see any particularly horrible technical roadblocks, but I do see a lot of interesting technical problems (e.g. reference resolution over this data!) and of course there might be pushback from the people that control the data now.

Update: There's some discussion of this from Fernando.

Wikis at school

2008-01-10T13:04:00.000-08:00

In Ars Technica, last October, I saw a story about a professor that assigned her students the task of writing a Wikipedia entry. I went one better, I think...Natalie Glance and I were teaching a seminar in "Analysis of Social Media" this fall, and I assigned the class the task of building a wiki on the subject. During the class I limited wiki access to class members, but it's now open to everyone to read or edit.

It was an interesting experiment. It was only a little extra work in grading and coordinating, but it was worth it for the irony factor alone. The students were mostly positive about it.

The principle content of the wiki is a bunch of paper summaries, not unlike what students would have turned in in a class, but some students came up with some other ideas for contributions. The main change I'd make if I did this again would be switching wiki providers. (Cheapskate that I am, I used a free wiki farm called scribblewiki. They were very helpful early on, and even upgraded me to a "paid" account for free, but took a long time with some other requests later - and never did add support for latex math.) It's not obvious what's the best way to use a wiki in teaching course for the (n+1)th time, but it's a fun twist for the first time you teach a seminar.

Collective Kvetching on The Point

2007-12-28T11:38:00.000-08:00

Over the fall semester, I got too busy to blog, but I bookmarked a couple of items that seemed to be worth thinking about. One was an article on Webware about a site called The Point.

The Point is a new site to help instigators collect the wishes of the masses and to get participants to pledge to take action when a "tipping point" of participation is reached.

For example, if you are upset that Southwest Airlines no longer lets families with small children board first, you can join the pledge to boycott Southwest once 2,000 other people also sign up. As soon as the desired number of people sign on to the campaign, the pledge is activated. But if they don't, you're not left twisting in the wind executing a meaningless protest.
The Point can also work with financial action: You can join a pledge to participate in an event if enough other people sign on, as well. If the pledge goals are met, your credit card (that you're previously submitted) is debited. If not, you're not charged.

I wish I'd thought of this---and I hope that The Point reaches it's own tipping point in use. It's great to see people coming up with new collective action using the web, and it will be interesting to see how/if this catches on once it is integrated with social networks (which is planned).

OneDayPoemPavilion.mov (video/quicktime Object)

2007-11-24T13:47:00.000-08:00

From Mike Berman's blog, the SoCal version of Stonehenge, but with more computers involved in the design. Of course in Pittsburgh's weather it would take all of November to see one cycle of the poem completely - would that make it better or worse ?

StupidFilter :: Main / About

2007-11-13T10:29:00.000-08:00

StupidFilter is "an open-source filter software that can detect rampant stupidity in written English. This will be accomplished with weighted Bayesian or similar analysis and some rules-based processing, similar to spam detection engines. ... To this end, we're collecting a ranked corpus of stupid text, gleaned from user comments on public websites and ranked on a five-point scale."

I wonder, to what extent can stupidity be modeled with a unigram distribution? What is the overall distribution of stupid comments? And how many random stupid comments does the average person look at before moving on?

Does your party affiliation begin with a 'D'?

2007-11-08T17:48:00.000-08:00

Avi Rubin asks: Does your home address begin with a '5'? Interesting story, although it is colored by Avi's unfortunate skepticism about the reliability of modern compu^@^@^@^@^@^@^@ //SYSIN DD* ERROR MISSING DLL

Open Source ML Software Track in JMLR

2007-11-01T09:24:00.000-07:00

What a great idea.

AT&T Slashdotted

2007-10-31T17:14:00.000-07:00

...and also Wired-Blog-Networked (?) with this story about a research paper from 2001 about a language for data-mining, more provocatively called a "Programming Language for Mass Surveillance". While I'm not happy with much of what my former employer seems to have gotten up to since I left, I guess I know too much of the backstory on this to subscribe to this particular round of hysteria. As an employee back in the late 90's and an sometime colleague of the principles on the paper I'd readily believe that the original purpose of "Hancock" was, as claimed, problems more like detecting long-distance fraud, than supporting the NSA and their ilk. (As an aside, at least 2/3 of the authors of this paper are now at Google).

This all points out an intriguing problem, really, for those of us engaged in R&D. Any sufficiently general tool can be used for many purposes, some good and some evil - and as tool creators, we have little control over the eventual effect of what we do. In fact, the better you are as a scientist and engineer, the more general-purpose your results and tools will be - so in CS, at least, it's unlikely that you won't facilitate something unpleasant sometime in your career. Certainly, we can make choices about where we work and how we direct our energies, but the bottom line is that it is now as scientists, but as citizens (and consumers) that we need to decide to what uses technology will be put.

And one man's language for mass surveillance might be another man's language for analyzing protein-protein interactions for look for cancer cures.

Take it back! Take it back!

2007-10-19T14:49:00.000-07:00

The hardest part of having a secret revealed is tracking down everyone that heard it and forcing them to forget it. From a story titled Breathtaking Abuse of the Constitution a confrontation between the Phoenix New Times and one Sheriff Arpaio (in which the Sheriff's home address appeared in an PNT op-ed story) has degenerated to a subpoena asking for, among other things, the domain name and IP address of anyone that accesses the Phoenix New Times website since 2004.

Early on the story says "It is, we fear, the authorities' belief that what you are about to read here is against the law to publish" and indeed the authors spent a night in jail for writing it, so you know it's worth a read.

Another interesting post from Lauren Weinstein

2007-10-10T06:30:00.000-07:00

Can you control what updates happen on your computer, or not? and who should have control? In Lauren's words: "who has the right to ultimately control operations on a system -- the owner of the computer itself, or a software vender?"

I bet you'll never guess what Microsoft's answer to this question is.

Maybe this is why AT&T bricks hacked iPhones?

2007-10-10T06:24:00.000-07:00

From Eweek, via Dave Farber's IP mailing list:

The iPhone has been turned into a "pocket-sized … network-enabled root shell...A rootkit takes on a whole new meaning when the attacker has access to the camera, microphone, contact list and phone hardware. Couple this with 'always-on' Internet access over EDGE and you have a perfect spying device".

A great application for named entity recognition

2007-09-27T17:26:00.000-07:00

1) Locate a highly polar, drippingly opinionated piece of political bloggery.

2) Identify all person names.

3) Randomly replace them with other figures.

4) Repost the story somewhere else, and record whether anyone can detect the change.

If successful it could reduce the time required to generate political blogs by 50%, a huge gain for the economy. I just got this idea in flash while reading this. If you're interested in collaborating on this research, let me know.

Now this is funny

2007-09-22T13:54:00.000-07:00

No longer timely, but catching up on my RSS, here's from an Ars Technica piece Sun to sell Windows Server boxes:

The move comes as a definite surprise, even for veteran Sun-watchers who are used to seeing the company switch strategies on a dime (see the historical chart above).

I always get it backwards

2007-09-22T10:32:00.000-07:00

An interesting Ars Technica post reviews evidence that some of us are susceptible to sales pitches that start by confusing you, and then clear things up. All these years, I've been following a model for talks and papers: always start off with something that everyone can understand, even if you end up presenting some thing dense and hard to follow later on. From now I'll put my hardest math in the abstracts, or maybe even the title.

A puzzle

2007-09-22T10:07:00.000-07:00

Knowledge is power - so why are consumers (and voters) so willing to give up control of information about their own behavior, social life, etc to companies (or the government)? I just followed some links from Fernando's blog and spent an intriguing time reading up on Vendor Relation Management (VRM) and some related projects. Giving consumers more control over their data is an old idea and one that has never really taken off. Maybe the real market niche is for systems like Farecast, that do data-mining of corporate behavior and put the information in the hands of consumers.

(Disclosure: I'm on the advisory board of Farecase, and anyway would be a natural fan of any company that uses "classic rule learning".)

Economics of data-sharing

2007-09-03T17:24:00.000-07:00

Last month I posted some speculations that the limited use of public transit was as much a problem of understanding how to use it as a problem of availability and cost. I got a comment from a legitimate expect on this, Joe Hughes, who pointed out that decent data exchange formats exist but "transit agencies haven't traditionally seen the value of sharing their information with outside developers". (Joe also shared a pointer to his blog, which includes an interesting post on geocoding ideas, btw).

The economics of data exchange is something I've thought about in the past. I think that efficient schemes for sharing information are best thought of as public goods - ie goods that have positive externality - ie things that benefit everyone, but don't benefit any one party enough for them to actually want to pay for it. One classic example of a public good is a road system (which benefits both the merchants and consumers that are connected). Like a road, data sharing schemes in general connect data consumers and potential data providers.

Unlike a road, however, the connected parties don't usually exist until some data-sharing scheme is possible - so you often see "chicken and egg" problems: there's no point establishing a standard until somebody's ready to use the published data, but nobody's going to be ready until the data is there. You can jump start the system from either end (or from a third neutral point, by scraping data sites and standardizing the format - let me tell me about this company I used to work for...) but that is always risky, since it's hard to guess the size of a market that doesn't exist.

Good news for privacy advocates or telcos - I'm not sure which

2007-08-30T10:28:00.000-07:00

Here's a nice article on DCSNet, the FBI's "sophisticated, point-and-click surveillance system that performs instant wiretaps on almost any communications device" (from Lauren Weinstein). From my years at AT&T I'm dubious that the technology works with the 1984-like seamless smoothness the article suggests, but this part sounds accurate to me:

Despite its ease of use, the new technology is proving more expensive than a traditional wiretap. Telecoms charge the government an average of $2,200 for a 30-day CALEA wiretap, while a traditional intercept costs only $250, according to the Justice Department inspector general. A federal wiretap order in 2006 cost taxpayers $67,000 on average, according to the most recent U.S. Court wiretap report.

As summer finally ends...

2007-08-29T08:34:00.000-07:00

Back from vacation, I'm so behind in everything it's amazing...starting with the news....

What's cooler than printing 3-D objects? Maybe printing human organs? This is only a little bit far-out...and even modest success would be hugely interesting, not necessarily to clinicians, but to developmental biologists and others that study how cells interact in tissues.
Some useful warnings about the "cult of FireFox" and the evil of Ad Block Plus. Remember, not reading ads is theft. And not citing my papers - that causes uncontrollable weight gain.
Hal Daume promises to automate the construction of LDA-like statistical models. Well, at least partly. A fascinating idea, although a challenging one...for whatever reason the ML community doesn't seem to take to these sort of high-level tools. AutoBayes and WinBUGS are prior efforts along these lines.
The scary privacy-infringement stories of the week: from Ars Technica, we learn that China is to begin web monitoring with Clippy-style animated police and, if that's not horrifying enough, a confessed movie pirate has been ordered to switch to Windows by the court, so his parole officer can install the appropriate monitoring software. (However, the rumors that the court also ordered a switch from Emacs to Notepad are apparently false.)
In related news, the EFF's suit against AT&T may have gotten stronger: even though National Intelligence Director Mike McConnell has said before that "the disclosure of any information that would tend to confirm or deny... an alleged classified intelligence relationship between the NSA and MCI/Verizon, would cause exceptionally grave harm to the national security" he, oops, confirmed that AT&T was assisting in surveillance: "under ... the terrorist surveillance program, the private sector had assisted us...and they were being sued". And the DoD's official web sites are more than 100x more likely to leak sensitive information than milbloggers.

Please re-calibrate yesterday's posting...

2007-08-17T07:36:00.000-07:00

Today, the same news source has a post on DARPA's bootstrap learning project which, while accurate enough in most of the details, has a pretty high hyperbole factor. (I helped write a grant for this program, which is novel enough, but not exactly "far far far out" - it has very concrete one-year deliverables, and is a pretty typical DARPA 6.1 program in terms of risk and innovation.)

Nothing to hide, and nowhere to hide it

2007-08-16T08:02:00.000-07:00

Shenzhen, China attempts to take the next step in citizen surveillance with 20k cameras "equipped with 'intelligence'".

Famous researchers barking about SEAL

2007-08-14T14:50:00.000-07:00

Richard Wang's SEAL system has gotten a few hits from the curious since my posting. Now it's getting some discussion from Matt Hurst and Fernando Pereira.

Hard to do vs hard to figure out

2007-08-09T06:03:00.000-07:00

I'd like to argue for a second that better websites might do more to stop global warming than hybrid cars....

Sometimes it's easier to just go with what you know than to try something new. Case in point - I was down in DC earlier this week, and even though I had a car and free parking at my final destination, I ended up taking the Metro to my workshop rather than driving - not because it was faster or cheaper, but just because it was easier. I understand the Metro; but I get lost driving around DC on my own. Likewise, when I first moved to the NYC area, I tended to walk or take cabs more frequently from point-to-point in Manhattan...then, after a I'd figured out the baroque and arcane subway system (and no longer found myself mysteriously deposited in Brooklyn at random intervals) I used it instead.

Now I live in Pittsburgh and one of the fringe benefits of my job is a free bus pass. So, I take the bus everywhere, right? Well, not really - I use a few routes I know, and drive most other places. The main obstacle, I think, is just not knowing what bus to take when. It's easier to drive. But that's changing...

Matt Hurst has a nice run-down of mapping systems that give information on public transit (or ways to walk instead of drive). Google transit also has a great system for Pittsburgh (and a handful of other cities). I think all that's really needed to make public transit more widely used is some tools like this, development of the BusML standard for route information, free municipal wifi and something to access it with that fits in my pocket.

Flyer's Rights

2007-07-31T09:39:00.000-07:00

You gotta love anything called a bill of rights for airplane passengers. How do I get them to add the right not to be sold credit cards while on the plane?

A paper emerges from the tunnel of anonymous review

2007-07-31T07:59:00.001-07:00

Overall I'm in favor of anonymous review, but one of the many annoyances of it is that during the review process I always feel reluctant to advertise. But now...

Richard's paper describing his "Set Expander in Any Language" system was accepted to ICDM. Of all my students Richard's the only one who really enjoys building systems...so if you don't have time to read the (preliminary version of the) paper you can play with the demo.

Deep packet inspection meets 'Net neutrality, CALEA: Page 1

2007-07-27T09:47:00.000-07:00

There's a very interesting, in-depth discussion of "deep packet inspection", and some of the implications of it on Ars Technica. DPI is diving into the packets flowing through an ISP, and opening them up to inspect the content - eg, there are commercial tools to identify the type of traffic (e.g., virus vs YouTube video vs iTunes download vs chat vs email vs ...). "Flow analysis" is assembling packets together (e.g., to reconstruct an email message), and that's also commercialized. DPI products that are "CALEA-compliant" can collect and offload a user's datastream (CALEA is the "Communications Assistance to Law Enforcement Act") - usually this stuff is farmed out by an ISP to specialists. Once packets (or flow) is classified it's also possible to impose rules - e.g., squash viruses, eliminate denial-of-service attacks, disallow on-line games for non-premium users, or slow down traffic from, say, YouTube to a crawl unless Google pays up a designated fee.

According to the article, current DPI systems classify packets using signature-based methods, much like anti-virus systems do. This makes a lot of sense if you're only interested in Personally I'm surprised that machine learning isn't used in this step yet - but I suspect that this will happen before long.

CBC.ca Arts - Queen's Brian May to complete astrophysics doctorate

2007-07-26T14:19:00.000-07:00

Brian May, the 60-year old former Queen guitarist just submitted his doctoral thesis in astrophysics on "Radial Velocities in the Zodiacal Dust Cloud" (Imperial College, London). When I retire I think I'll become a rock star.

Data Mining: Text Mining, Visualization and Social Media: LinguisticAgents

2007-07-26T14:13:00.000-07:00

Matt Hurst has been spying out the action at AAAI. Today's post on LinguisticAgents (an Israeli company) is interesting. From a quick read, NanoSyntax is combining morphology with syntax - which makes loads and loads of sense in Hebrew, certainly, and similar methods have been effective in other languages - Klein et al have a very nice paper from CoNLL a few years back on character-level models for NER., to give one example.

I always find it amusin', though, how different industry types and academics pitch their intellectual wares. AI companies are so often based on transformational revolutionary brand-new ideas (if you believe the white papers), whereas the most of us longhair university folks are plugging away with incremental improvements to the big idea from, say, three years ago. Does that seem backwards to anyone else?

What are willing to do - for science?

2007-07-26T06:47:00.000-07:00

Of course you trust your ISP, but next time you're using a friend's, visit http://vancouver.cs.washington.edu

It's, you know, a good cause.

It's a sign!

2007-07-25T18:09:00.000-07:00

While reading the blogs on my front porch, my laptop ran out of power and hibernated while I was reading this. (Discovered from Boingboing).

More email leaks

2007-07-23T07:14:00.000-07:00

For those that are interested in information diffusion processes: I got an update from Vitor on publicity on our paper on "detecting email leaks" - there have been a dozen or so followups to the new stories I mentioned back in June. Interestingly all of these are in Portugese:

Blogs:
http://alexronald.wordpress.com/2007/06/30/brasileiro-cria-sistema-que-evita-envio-de-e-mail-a-destinatario-errado/
http://idgnow.uol.com.br/seguranca/2007/06/29/idgnoticia.2007-06-29.1623119240
http://bloginfodicas.blogspot.com/2007/06/nesta-ltima-dcada-perodo-em-que-os.html
http://www.rogergoncalves.com/blog/?p=21
http://www.piratadarede.com/blog/?p=1031
http://ouhe.blogspot.com/2007/06/novo-sistema-evita-envio-de-e-mail-para.html
http://videobr.pro.br/forum/viewtopic.php?p=37696&sid=e510424e3ba94063a467a5e5e5970cd3
http://inovabrasil.blogspot.com/2007/07/novo-mtodo-previne-vazamento-de.html

http://mundoetecnologia.wordpress.com/2007/06/29/brasileiro-cria-sistema-que-evita-envio-de-e-mail-a-destinatario-errado/

Research, high-tech and general news websites

http://www.acessasp.sp.gov.br/html/modules/news2/article.php?storyid=31
http://www.agencia.fapesp.br/boletim_dentro.php?data%5Bid_materia_boletim%5D=7362
http://g1.globo.com/Noticias/Tecnologia/0,,MUL61241-6174,00.html
http://g1.globo.com/Noticias/Tecnologia/0,,MUL34417-6174,00.html
http://www.adn-negocios.com/adn/index.php?option=com_content&task=view&id=38&Itemid=1
http://pcworld.uol.com.br/noticias/2007/06/29/idgnoticia.2007-06-29.3833885183
http://pe360graus.globo.com/noticias360/matLer.asp?newsId=94336

At least 30 comments on a discussion board, asking the question: "Have you ever sent an email to the wrong person?"
http://g1.globo.com/Noticias/Tecnologia/0,,MUL34362-6174,00.html

And some in news Portugal, pretty much echoing the news from Brazil
http://diariodigital.sapo.pt/news.asp?section_id=44&id_news=283432
http://www.wintech.com.pt/index2.php?option=com_content&do_pdf=1&id=380

I guess there's some element of "local news" with these, since Vitor's Brazilian, but there's nothing particular about the content of the story that seems especially Brazilian - it's just a technique to avoid a class of email-related errors (and was evaluated, in fact, on the Enron corpus, which is all English.) It's interesting that language is as much of a barrier as it appears to be for the spread of high-tech news.

If you don't own an iPhone yet...

2007-07-23T06:15:00.000-07:00

It got easier. Penetrated, completely compromised; the hack validated by Steve Bellovin and Avi Rubin; published in the NYT; and even slashdotted. On the plus side the guy that hacked it is a former employee of the NSA.

Is this the world's most frequently parodied song yet?

2007-07-16T11:57:00.000-07:00

Chortle. (From Lauren Weinstein. Posted, naturally, from my Google-owned blogger account.)

The laughter curver

2007-07-13T14:01:00.000-07:00

Ok, maybe this is politics...but it's also a great example of really bad data analysis. Maybe there is a Laffer curve, but you sure have to look hard to see it here. I wonder how many economists would choose the line in the top graph over the one in the bottom graph, if all they saw were the points, without any labels?

Sad news of the day

2007-07-09T07:43:00.000-07:00

Don Michie and his wife were killed in a car crash.

Personalization and polarity

2007-07-08T09:29:00.000-07:00

In the more than 20 years I've been studying AI I've discovered that every decent knife has two edges. Even those knives that seem like really, really cool ideas when you first grok them. How could giving everyone a free editorial column be a bad thing? and how could improving access to all that new content not be beneficial?

Anyway, I can't resist responding to Fernando's response to Matt's response to my response to Lauren Weinstein's posting (are you following all this?) on search-term polarity.

The original post by Lauren Weinstein that triggered this thread was about the visible global impact of search rankings, but William's discussion suggests a less global but possibly more powerful effect in search personalization, of whether a personalization algorithm could become a strong reinforcer of prejudice without the counter-pressure of critical discussion of globally visible search ranking.

Here's an even broader suggestion: could just having more and better access to more and more diverse content have the same effect - i.e., is the growing blog world "a strong reinforcer of prejudice without the counter-pressure of critical discussion of globally visible" content? It's certainly easy enough to fill time reading political commentary that you can be 99% sure you'll agree with - and look how bitterly partisan the country has become, and how little is now universally accepted as correct.

Maybe Matt or Fernando know whether anyone's ever looked into whether the effect I'm speculating about is real - and if it is, what could we scientists do to create the appropriate "counter-pressure". Ideas, anyone?

iShouldn't have wondered about security

2007-07-06T12:11:00.000-07:00

According to RixStep, the iPhone includes, among its many OS X-based cool features, a root password of 'dottie'.

CKAN

2007-07-06T06:24:00.000-07:00

Here's something interesting, I wonder how far it will go. Wikipedia's been so successful, a lot of people seem to be trying to take a further step in that direction. So far, Freebase is my favorite project of that sort - they have a very precise clear vision that's easy to convey (if not accomplish).

iBuzz

2007-07-05T07:06:00.000-07:00

According to Matt Hurst the iPhone buzz has peaked which means that I'll fashionably late with my contribution...

I don't expect to get an iPhone anytime soon - I'm still perfectly happy with my ancient Samsung i500. But I'm amazed that Apple decided to ~~jump in bed~~ lock in to using AT&T as a provider. (Apparently the first day the iPhone was released AT&T Edge had widespread network outages, affecting non-iPhone users as well as iPhone users.) And I'm thrilled that Apple decided on a closed platform - it will be way amusing to follow the inevitable opening of the iPhone by hackers. The checklist below (from IP) is already yesterday's news:

Break DMG Password *COMPLETE*
Break Activation *COMPLETE*
Unlock Phone
Run Third Party Applications
Allow DUN/Tethering
Remove IMEI Transmitting
Enable Disk Mode

Yes, it would be cool to be able to use an iPhone on whatever network you want - but there's a socially interesting issue here also. My smartphone holds my information, some of which is potentially quite private (e.g., what doctors do I go to? when am I out of my house?), and which we know is not always very secure. I think I know how and where my Palm/phone device keeps this information, but on a networked device with a closed architecture, what guarantees do you have? We know how careful Apple has been with DRM in iTunes - it'll be interesting to see how much effort Apple has put into locking up its customer's information.

Research Publicity

2007-06-29T13:06:00.001-07:00

Last year my student Vitor Carvalho got the clever idea of using machine learning methods to detect when an email is mistakenly sent to the wrong person (an "email leak"). A few months after our publication of some results on this, the idea's getting a little publicity:

An Australian security-consulting firm wrote up an article about our paper...
... and a major Brazilian funding agency wrote another one. In Portugese, so I can't read it, but still.

Search-term Polarity

2007-06-29T06:53:00.001-07:00

Via Farber's "Interesting People" list: Lauren Weinstein has a nice discussion of a well-known bug or feature of Google's ranking method - namely, if you search for "Jew" the top-ranked page is, well, sort of uncomplimentary to us Members of the Tribe. Google's analysis is that "Jew" is used more in an "anti-Semitic context".

Lauren's comments here are interesting, and raise some nice questions. If my search term indicates I'm an anti-Semite, should I get a page that's ranked highly by other anti-Semites, or one that's ranked highly by the (hopefully) larger general community? What if my search term indicates I'm a creationist? a disbeliever in global warming? arguably there's a continuum between Google-bombing (ie, manipulation of search results by a small group) and just exploiting linguistic regularities of a subcommunity to give better search results.

Dangers of eating lunch at home

2007-06-20T12:11:00.000-07:00

Durn, I missed CMU's "Dynamic Balance Festival" and a chance to try out an extreme pogo stick.

AT&T Boing-Boinged

2007-06-19T17:42:00.000-07:00

My old employer AT&T makes the most-read blog, Boing Boing.

But even at $10/month, AT&T DSL should be avoided like the plague. These are the scumbags who illegally wiretapped the entire Internet for the NSA, who broke net-neutrality to find "copyright infringements, and who inspired NBC to call for a law requiring all ISPs to do the same (imagine -- a law forbidding network neutrality!). Seriously: the only day I wouldn't piss on AT&T is if they were on fire.

This isn't politics, btw - it's technology.

The Shape of Things to Come?

2007-06-18T06:29:00.002-07:00

The book The Long Tail is about the way the market for "soft" goods (like music) has changed with the internet. The last chapter discussed some interesting new technologies, including 3D printers - with speculation that they will ultimately force the same sort of changes in the market for real, physical goods. Right now 3D printers are expensive and slow, but they are already making an impact in certain niches--for instance, a talk at CMU a few years ago made a great case for using them to make printing 3D models of proteins. (Believe it or not, holding one for a minute or two beats any CHIME visualization). A good friend of mine recently blogged a visit to Desktop Factory, which makes 3D printers, and posted a bunch of interesting images. Look at the bones, man!

Symbol grounding and relations

2007-06-15T11:38:00.000-07:00

I've been reading a number of Peter Turney's papers and I've lately been catching up on his blog, which has a number of really interesting posts related to his work - and remarkably related to the things I've been pondering about over the last few months.

For instance: machine learning spent years learning how to recognize classes of objects by their attributes; a popular topic now is collective classification, or recognizing the class of an object (in part) by considering how that object is related to other objects. Are attributes only a "convenient fiction" - a useful abstraction that ultimately must be defined in terms of relations? Is an apple intrinsically red, or is "redness" something that describes the interaction of the apple and the sensory system of the observer? likewise is every attribute properly a description of an object and some sensory system or measurement instrument?

This seems like a rather strange and abstract question, but it's intimately connected with the "symbol grounding problem", the subject of another of Peter's posts, which in turn is connected with my long-standing interest in data integration - a very practical real-world problem. To combine data from two heterogeneous knowledge bases is, properly speaking, impossible to do automatically: if they are different formal systems, and there is no surefire way to translate between them. There is no common ground. By the same token, communication between two people is also impossible. How do we know that what I mean by "red" is the same as what you mean?

The solution to the problem, for human communication, appears to be that language is grounded - in part by common perceptual systems. (Goleman's book Social Intelligence is a nice description of some of the ingenious mechanisms that have evolved for establishing this common grounding.) The effect is that I don't really know that what I mean by "fear" is the same as what you mean; an in fact, it may not be the same. But if we're both neurologically typical it is almost certainly highly similar to what you mean.

Back to attributes and relations - I'm not sure, but I think the apparent primacy of relations starts to emerge when you start thinking about these issues. Everything is ultimately defined in terms of relationships with other things, which are finally grounded in our own perceptions--those few things that we don't need words to explain or understand.

Is it real or is it demoware?

2007-06-06T15:02:00.000-07:00

From Lynn Monson. These guys have so been watching Minority Report.

Google Streetview

2007-06-06T06:08:00.000-07:00

There's lots and lots of talk about Google Streetview, including quite a lot of gosh-what-about-privacy? postings: eg there's a rundown of comments in BoingBoing today. One that I resonate with is:

What is the difference between posting a picture of people on a public street on Google Street View or on Flickr?

Obviously - the information is the same, only the information access is different. There's no difference in kind of information - just a qualitatively easier interface to getting it. Anyone can take a plane and a taxi and take a picture of my house - but I don't expect that anyone will bother. So it's jarring to find out suddenly that anyone with a DSL line and 30 sec to spare can get the same effect.

Our expectations of privacy (and privacy laws) are driven by what's easy, and what's likely, not what's theoretically possible.

My personal take on this - it's another great example of how smoothly information can be integrated together, when it's all grounded in the physical world. Entries from maps + business listings + satimages + street view are relatively easy to search together and visualize simultaneously, since they're all tied together by physical location in space.

New Look and New Resolutions

2007-06-06T06:07:00.000-07:00

I'm back to blogging for a while, but I'm staying away from politics. It's just way way too much of a time sink.

Evolution again

2005-05-07T09:02:00.000-07:00

Some scary things are happening again - not here in Pennsylvania, but over in Kansas, where two of my nieces live. They are smart kids and their dad is a biologist at the University of Kansas, so they probably won't be affected much by the latest nonsense. I love this quote:

The hearings in Topeka, scheduled to last several days, are focusing on two proposals. The first recommends that students continue to be taught the theory of evolution because it is key to understanding biology. The other proposes that Kansas alter the definition of science, not limiting it to theories based on natural explanations.

This just so reminds me of that urban legend, where some state legislature defines pi to be exactly 3. And it's even scarier with pictures:

Someone, please tell me this has been photoshopped. Yikes.

On a personal note, my heroic adventurous kid sister is blogging her trip to Darfur. Wow.

It should be in the Onion

2005-02-15T08:38:00.000-08:00

I've been following the Gannongate thing. It's amazing.

Imagine a government that not only pays off pundits to push its own programs, but sticks a few "plants" into press conferences, make absolutely sure that there's no problem staying on message. Imagine that the president is told before a conference that "the bald guy would lob him an easy one."

Imagine that this "plant" was worked for a no-revenue news source, newly created a few days before he applied for a press pass - a news source that was an arm of something called GOPUSA. Imagine that his prior journalistic credentials were attending a single $50 seminar.

Imagine that he operated under an assumed name and had access to not only the president - that rarest of things for a reporter - but also to secret CIA documents. Documents that were leaked to reveal the identity of Undercover CIA Operative Valerie Plame in what Bush 41 called "an act of treason".

Imagine that this plant, given access to secret documents and Bush's press room, a man whose questions are reported in every major newssite in press-conference transcripts, a man who so ardently supports the anti-gay-marriage family-values morality-pushing Red government, also worked literally as a prostitute, a gay prostitute, no less. I do not recommend you follow this link at work, or in front of the kids - they've photoshopped out the full frontal nudity, but it's still pretty raw stuff.

Imagine finally that this plant/"Dominant Top"/gay hooker/"journalist"/Scott McClellan lifeline has written a whole web site to "define himself". It's gone now but there is a cached version of the conservativeguy.com. Some highlights:

- he owns a Baretta 9mm and a Japanese SUV (see the pictures?)

- his mission is, among other things, to "help people to allow their conservative values to govern their actions"

And the punch line:

- his "soon to be published book" is titled "The Hypocrisy Of Liberalism"

LOL! Honestly, this whole thing could be in the Onion.

Take the Offensive on Social Security?

2005-01-22T13:22:00.000-08:00

We know there is no crisis in social security. The bottom line is that the Bush administration wants to spend $2000 billion to phase out the current pay-as-you-go system, and replace it with a system of government-mandated and government-regulated personal retirement accounts.

Progressives are fighting a purely defensive battle, to "save social security". A purely defensive battle is a bad idea - it doesn't communicate any positive values. We need to take the offense on this issue. We need to assemble a solid alternative plan for social security that communicates our values.

Progressives should propose that social security be extended, not phased out - extended to cover anyone involved in childcare. This would include married parents that don't work, or work only part-time. It would also cover anyone working in daycare. By "cover" I mean that parents would have social security eligibility exactly as if they had been working full-time (according to some formula based on full-time wages before and/or after their child-rearing years). Professional caregivers and their employers would be except from social security taxes.

Who benefits? For starters, all families with kids. If one parent takes time off for child-care they are rewarded. If both parents work, then they benefit by an increased availability of potential childcare professionals--a massive problem for suburban soccer moms in many places. The other beneficiaries are childcare professionals--fewer of them need work "under the table", off the official economy, and they are reminded financially that their work is important.

Who pays, and how much? Everyone in social security pays, a little. But it's fair to reward those of us that have kids, sinnce we keep the pipeline full for everyone. And if we start the process now, nobody pays for years---parents that are involved in child-rearing now will not be retiring for a while. This means that the immediate cost is low - and long term, I'm guessing it will cost way, way less than $2000 billion.

What's the strategy? First, it reminds us that social security is NOT all about personal ownership, and that there are other values a government can promote - like encouraging parents to care for their children, and rewarding professionals that do the same. Second, it is a progressive way of promoting "family values". (Why should we give such a nice-sounding phrase away to the Reds? let's make it really mean something, the way Clinton did with "law and order".)

Ok, that's one idea - ready for another? The conservatives love to push private accounts, and they love to say that the trust fund is a bunch of IOUs. So let's make them happy: let's take half of the trust fund - what's there now, and what's coming in later - and distribute it, proportionally, to those that will retire between 2018 and 2042. They can invest it as they choose, and maybe get a boost over T-bills, if they want to take the risk, and if they don't opt in, then the money goes into a government-managed bond account. But let's keep a sunset on this provision--after all, it's simply another way of managing the trust fund for those 25yrs that we need one--and have the system revert to pure pay-as-you-go afterward.

The strategy? putting this in the plan means that we have the same "feature list" the Reds have - private accounts, ownership, etc. But on a smaller scale, and for a reason. And, it keeps Congress out of the trust funds.

I think if we cut out the income caps, cut benefits for the wealthy, we can allow limited private accounts while maintaining a pay-as-go system, and make social security not only fiscally sound, but better - more progressive, and more aligned with the country's values.

Followup to ID post

2004-12-27T07:48:00.000-08:00

For any readers that want to dispute my summary dismissal of intelligent design: the National Academies (of Science, Engineering and Medicine) have been, since they were formed by Lincoln, the main source of technical advice to the government. They have a page on evolution & education, and an book discussing the issue (online, if you want, or else available as a paperbook).

Below is the description of this book from the Nattional Academies Press:

While the mechanisms of evolution are still under investigation, scientists universally accept that the cosmos, our planet, and life evolved and continue to evolve. Yet the teaching of evolution to schoolchildren is still contentious. In Science and Creationism, The National Academy of Sciences states unequivocally that creationism has no place in any science curriculum at any level.

Briefly and clearly, this booklet explores the nature of science, reviews the evidence for the origin of the universe and earth, and explains the current scientific understanding of biological evolution. This edition includes new insights from astronomy and molecular biology.
Attractive in presentation and authoritative in content, Science and Creationism will be useful to anyone concerned about America's scientific literacy: education policymakers, school boards and administrators, curriculum designers, librarians, teachers, parents, and students.

Hooray for the ACLU

2004-12-23T14:58:00.000-08:00

Under the Taliban in Afghanistan, fundamentalist leaders in the government decided what religious beliefs were acceptable. In America, thankfully, the government doesn't tell us what god to believe in: Church and State are separate, and Americans are free to worship as (and if) they please.

One would think that after 200 years, people would, generally speaking, be clear on the basic concepts surrounding freedom of religion. Alas, not so. Not so far from my home, in the Dover, PA school system, science teachers are now required to teach "intelligent design". And let's make no mistake here: the study of "intelligent design" is not about science--it's about twisting science to support a certain literal interpretation of scripture.

(A short summary of ID, from looking at some arguments from proponents: there are some aspects of evolution we don't understand, and therefore G-d may control evolution. I personally don't mind having high school students hear that much, but it's ludicrous to plan to subject them to the actual arguments between ID proponents and opponents, which quickly degenerate into technical discussions of how the expression levels of groups of related genes are regulated in metazoans, the connections between ontogeny and phylogeny, and such. It's also clean that mainstream biologists are not in any way divided on the correctness of the "theory" of evolution.)

Anyway - not surprisingly, the requirement to teach "intelligent design" was not spearheaded by science teachers. The main instigator is a school board member William Buckingham, a self-described born-again Christian and a believer in creationism.

The Pennsylvania branch of the ACLU is now litigating against the Dover school board, and according to my wife (who volunteers and sometimes works there) they've been getting a bunch of letters with varying degrees of hostility. A recent letter in my local paper accused the ACLU of trying to suppress freedom of speech by their lawsuit.

I've sent a shorter version of this blog to my local paper, but that letter-to-the-editor is a classic case of "unclear on the concept": it's not a matter of what gets spoken, but what gets taught to our children. William Buckingham is free to speak as he likes, and I'm sure the ACLU would be all over anyone that said otherwise. But in America, neither he, nor any other religious leader, has the right to use the public schools as a pulpit.

When my family moved to Pittsburgh we immediately chose a synagogue, at which our children now receive religious instruction twice a week. The wonderful thing about that choice is that it was our choice, and ours alone. Americans who sincerely care about religion should be grateful to the ACLU for helping to defend the right of every American to find G-d his or her own way - not through the government.

A few days ago Christmas carollers, I guess as a protest against the godless liberals, came by the ACLU and sang, along with their other tunes, "We can't wish you a Merry Christmas, we can't wish you a Merry Christmas, we can't wish you a Merry Christmas, because the ACLU says no." And the ACLU office workers listened cheerfully, then invited the carollers in for coffee and fudge. (And it was excellent fudge, by the way.) I doubt there will a Christmas-miracle change for heart for anyone, but you never know. After all, Rush Limbaugh has now become an ardent privacy advocate.

Back Again

2004-12-09T16:14:00.000-08:00

Well, it's been weeks since I did this. Blame it on the election - no, I wasn't depressed, particularly. I've been trying to get some work done again! Yes, I'm taking courses this fall, but that doesn't make me a student. Obsessing over news and tramping around Greenfield drumming up Democratic voters is no hobby for an up-and-coming academic.

But I can obsess a little bit, can't I? Friends and family have probably heard my famous airline story, which starts with me (with wife & two kids) grumpily boarding a long-delayed flight to my inlaws in November, 2001 and finding out that pillows and blankets were not available "for security reasons". Wife and I stare at each other with a silent look of "huh? what sense does THAT make?" and I say, with my usual total lack of tact and foresight: "Maybe they want us all to be alert so we can defend the plane if there's a hijacking." The stewardess rushes over - "Shhh! you can't say that word!". Now, I know I can't threaten, or even joke about threatening, or even joke about weapons. But I wasn't. There are words you can't say? George Carlin now has a "seven words you can't say in flight" routine? So anyway, I ask her nicely to tell me what the new rules are about what I can and can't say. There's more to the story, involving being hauled off planes and ID checks and men with guns and army fatigues, but let's not get too involved. Really, I should have just shut up to start with, stopped kvetching, and not caused trouble - did I mention this was Nov 2001?

What bugged me at the time was that I thought they should have at least told me their durn rule. After all, how can I follow it if I don't know what it is? if ignorance of the law is no excuse, and the law is a secret, I mean, what sense does that make? But after all, everyone at the airlines was worked up, about losing their jobs if nothing else. And it was November, 2001.

Turns out that three years later, the situation is still pretty much the same. A conservative congresswoman ran afoul of the TSA last month, when she asked to see the legal authorization for a pat-down search. So, they wouldn't let her fly. The article I cited notes:

in a qualitatively new development in U.S. governance, Americans can now be obligated to comply with legally-binding regulations that are unknown to them, and that indeed they are forbidden to know. This is not some dismal Eastern European allegory. It is part of a continuing transformation of American government that is leaving it less open, less accountable and less susceptible to rational deliberation as a vehicle for change.

Part of a disturbing trend, another part of which is the continuing erosion of any sort of privacy. Laser printers have been quietly augmented so that the government can find out what printer has produced what document:

Next time you make a printout from your color laser printer, shine an LED flashlight beam on it and examine it closely with a magnifying glass. You might be able to see the small, scattered yellow dots printer there that could be used to trace the document back to you...Governments, including the United States, already use the hidden markings to track counterfeiters.

Freedom of the press, right, that's in the Bill of Rights. But the ability to anonymously print a flyer and distribute it seems to be disappearing, if indeed it's not already gone. Say what you like, but remember, they're watching...

And who is being watched? I certainly support tracking counterfeiters. But it appears to me that our ability to observe - to infringe on privacy - is far outpacing our ability to hide - to protect privacy. I would have guessed the opposite would happen, technically. I think the upshot of all this is that we need to really, really watch the watchers. Are they looking out for bad guys? or spying on Quakers and peaceniks? it's up to all of us to make sure.

Two orders of magnitude

2004-11-21T09:21:00.000-08:00

Here's an interesting insight into the situation in Iraq, from a widely-blogged interview of an Israeli interrogator:

How physical are you allowed to get during interrogations, with permission?

Very low levels. It could be two slaps in one interrogation, or to shake him, but not very strongly, or to put a cover on his head to scare him. We have never insulted a person's religion or humiliated them. There is no torture in the security services.

What do you make of the torture and abuse that took place in Abu Ghraib prison in Iraq?

I don't want to judge the Americans. In Gaza we have one security person for every 1000 people. In Iraq they have one for every 100,000. They have no information or intelligence on their detainees. Information is the beginning of interrogation, and if there is none, if there is no language between you and the detainee, sometimes you will use more power. That I presume is what happened in Abu Ghraib.

One for every 1000, versus one for every 100,000. Quite a difference.

Catching up...

2004-11-15T14:15:00.000-08:00

Catching up, as it were - a final political post from Watchblog that's maybe worth sharing with the non-watchblog world. Most of these things are so transient...unfortunately, not all. I also wrote another post, back before 11/2, which was mostly about how Bush has clearly decided to ignore the outrage over torturing accused terrorists. And now Gonzales, the legal eagle behind Abu Ghraib, is being put up for AG - head of the Justice Department. Orwell would love it.

Maybe now that the election's over, I'll finally get back to doing some work. Or maybe not. It's so much more fun to fume about the election results and the news, isn't it?