A Measurement Challenge

Today I postet the following comment to Douglas Hubbards forum:

Mr. Hubbard,

I’ m a Software Developer from Italy with a passion for the ‘uncertainty sciences’ last century gave us so plentiful. I’ve read both your books and I am about to order the second edition of HTMA. In fact I am so intrigued by AIE methodology that I convinced some quantitatively skilled colleagues to set up a workgroup to apply AIE on some relevant problems to us.

I want to say it’s an honor to confront you with a measurement challenge. Let’s start:

The Swiss bank UBS published an article in its ‘UBS investor’s guide’, special edition April 2010, predicting the outcome of the FIFA 2010 Soccer World Cup. http://www.ubs.com/1/e/bank_for_banks/news/topical_stories/edition_10.html

You will agree this is a relevant problem, as the 'uncertainty reduction' on the game’s outcome will give an advantage in sports-betting.

With hindsight, they failed the prediction miserably, claiming:

(1) Brazil is most probable winner – didn’t reach the semis

(2) Germany and Italy likely to go far – true for Germany (3th in Rank) but Italy didn’t survive the first round.

(3) “Spain – favored by many – will likely not do well, and could exit before the semi-final stage” – Spain won the World Cup.

UBS has now an inglorious record of 1 success in 3 attempts - Wordcup 2006 went good, but European Championship 2008 and Wordcup 2010 failed.

I am inclined to argue that you can’t predict the outcome of the game a priori.

1. UBS likely has built a state of the art econometric model but the conclusive verdict about the rightness of the model can only be “it works”. This show: you certainly can make a sound argument about how you measure it, but still failing miserably.

2. But you cannot know if your model is right or you had luck. This is so because the experiment is not repeatable well. The basic dilemma of social sciences: social systems are complex and adaptive. Using a model: the stochastic process is itself complex, if not random. When we cope with induction we can only believe in the stable nature of the stochastic generator. What UBS’ case tells me: there is anecdotal evidence that the underlying principles of “who wins” are not stable. You cannot say if it will work for the next FIFA world championship or not, making it useless.

3. But probably even if you would know the exogenous factors that influence the game, I suspect the endogenous factors in the system are much more important. Making any reasonable forecast before the games started futile.

Mr. Hubbard: can you measure it?

Sincere Regards,

Roland Kofler