In which I question the basis of everything

If you've read The Wisdom of Crowds, if you've read A New Kind of Science, if you run SETI @ home, or if you have ever had your mind expanded by any flavor of chaos theory, you, dear reader, are probably a sucker for the idea that new ways of looking at the word--and especially new ways of harnessing the awesome information-processing power that is all around us--is likely to yield spectacular predictive power. I know I am.

And you probably thought Intrade and it's ilk--which purport to predict all kinds of unrelated real-world results based on the mystic power of market forces--are a pretty nifty idea. I know I did.

But is there any evidence whatsoever that predictive markets actually predict future results better than any other kind of analysis? Every election cycle we hear about the predictive markets, but we never hear about the follow up. From the perspective of a media-watcher, this makes it the functional equivalent of some random town in some random state that has a 98 percent success rate of predicting the next president based on their local outcome. And, just like those stories, there is never any follow up.

So this year I've paid marginally (no market pun intended) more attention to the results of the predictive markets, and what have I seen is that the market has followed the trends rather than using the awesome aggregate intellect of their user base to anticipate any developments whatsoever. On Intrade, for example, this example sums it up for me:

Barack Obama’s chances of winning the Democratic nomination have risen sharply in the last four days, with Intrade on Monday giving him a 70 per cent probability of taking the prize against just 30 per cent for Hillary Clinton.

With the exception of a brief leap following his strong victory in the Iowa caucuses on 3 January, Mr Obama’s chances had been rated at substantially below 50 per cent all year.

If I weren't such a sucker, I could probably have predicted their prediction problem. You can follow the links from the wikiepd page for some more detailed and scientifically sound analysis of why predictive markets suck so very badly at what they are set up to do, but permit me oversimplify for rhetorical effect:

Predictive markets fail because, although they use the machinery of actual markets, they aren't actual markets in one crucial respect: There is nothing of actual value being bought or sold. It is simply a complex form of betting on outcomes (no wonder, given that Intrade was founded by Irish bookies). With the actual value of the underlying commodities and securities completely removed, the actual incentives for long-term, reasonable traders are likewise absent. What you are left with are day-traders and other nogoodniks whose activities, while more complex, are really no different than betting on a horse to show down at the track.

The same trouble is plaguing real markets, as well. We haven't managed to actually remove the underlying value of the commodities and companies being traded, but increasingly baroque financial packages have managed to obscure that value to such a degree that even very savvy financial institutions are being caught by surprise when they discover what is actually contained in many of their securities.

For markets to work well, at assigning value or predicting changes in value, there must be incentives for the long game, meaning that there must be real and transparent underlying value.

Not that I'm against gambling! But I do prefer straight up books to the hijinks at Intrade. I hear the odds on Obama are looking pretty good....