Tetlock said: 'The dominant danger remains hubris, the vice of closed-mindedness, of dismissing dissonant possibilites too quickly.'
Personally, I am fully aware that I am wrong about everything, a posture which, if applied correctly, would make me right 33 per cent of the time in Tetlock's tests and, therefore, a better pundit than the pundits.
A blog about, among other things, imaginary ideas - What ifs? and Imagine thats. What if photographs looked nothing like what we see with our eyes? Imagine that the Berlin Wall had never come down. What if we were the punchline of an interminable joke? All contributions welcome.
I accurately predicted that Milan Baros would win the Golden Boot at Euro 2004 and won two hundred quid (Golden Boot is quite Extremistan, win-lose-draw is more Mediocristan, I think). I was so surprised and terrified by this win that I gave up gambling immediately and am still in a state of funk about what karma has in store.
ReplyDeleteThe only things you can predict with any certainty are that when an RnB singer encourages you to throw your hands in the air, it will be to do it like you just don't care; that when he is saying please baby please, he will very soon be down on his knees; and that the BBC will get it wrong.
Nobody knows anything. Years ago I came across Brian Walden saying that 'We are all duffers' (which I understand he took from Harold Wilson). Years ago this line of thinking would disturb me, but the older I get the more consoling it is.
ReplyDeleteTetlock obviously never knew Fred Irving who, in the more than ten years that I knew him would forecast the outcome of horse races with unerring accuracy, a nineteen eighties Fred Goodwin you might or might not say.
ReplyDeleteI hope you're watching Fred, in that big tote in the sky.
Your complexes emerge into the sunlight more and more every day Bryan, personally I am right about absolutely everything, as I zoomed backwards into a ditch at 80mph I accurately predicted that the vehicle would grind to halt.
I got the 1992 election dead on.
ReplyDelete21 seats and I even knew that the 21 count would not be confirmed till the Friday morning.
I won a fair bit of cash at the bookies, Kinnock was well ahead, which I gave it to St Lukes hospice, I have never before that placed a bet, and have never since ( my grandad was an addict)
I had a premonition a few weeks before, and its still spooking me.
I think similar studies have been done with stock pickers. Folks asked at random on the street have produced better-performing stock portfolios than "professional" analysts. The folks on the street have a better sense of which outfits are up or down because they have more experience of using their goods and services.
ReplyDeleteOh well. If experts, authorities, pundits and analysts aren't up to the job, maybe it's time to bring back the humble prophet.
Their problem being that they couldn't self-correct, presumably because they'd invested so much of their personality and self-esteem in a specific view.
ReplyDeletePertinent to yesterday's bloggery: Eric Hobsbawm.
My missus has been a pretty good stock picker over the years. She can't read a balance sheet. I ask her how she does it and she says she looks around and sees what people are buying. She reads up on the company to satisfy herself it's not a total basket case and takes the plunge. In the past few years she's done very nicely out of electronic gaming and debt management companies (though she got out of them two years ago). By contrast, I used to follow the broker's tips - it was disastrous.
ReplyDeleteSocrates was the wisest man in Greece, because he was the only one who knew that he knew nothing.
ReplyDeletethe future has only three possible answers?! now you tell me!!
ReplyDeleteIs it only personality and self-esteem? Isn't it also career, worldy status and therefore ultimately money?
ReplyDeleteIsn't this the case for intellectuals being amateur? Which you can be either because you have independent wealth (Plato), or because you have a humdrum job and do your thinking beneath open skies, on the side (Elberry).
Well, unless you have already established for yourself a status as a free thinking columnist, such as your lucky self.
(p.s. by 'luck' I am not implying that you don't deserve it, just to be clear!)
Ian -
ReplyDeletebetter; worse; much the same.
WJ, Its hard work knowing that you know nothing its easier to pretend you know something but you are not quite sure what it is.
ReplyDeleteBut what a game Socrates had in the 82 world cup final against Italy, how could he be on the losing side?
Smoker, drinker, doctor and philosopher, but not a world cup winner, what a tragic end?
Don't forget the fourth Brit and Ian, none at all.
ReplyDeleteI have that on good authority from a Pundit of the Jehovah's Witnesses persuasion.
None at all counts as worse. (Or it does if you're Frank Wilson. If you're B Appleyard it counts as better and also erotic).
ReplyDeletethat's all very well, Brit, but who for?
ReplyDeleteI'm confused now. Does that mean we have to take your chum Jim Lovelock seriously or not?
ReplyDeletethink science is different from punditry
ReplyDeleteThat's a great article. Is the wise man who can't pick his cereal more self-aware and does this depend on his route to the indecision. I have recently accepted the yank Dan Dennet's position (pardon the big mucky quote)
ReplyDelete"I have ventured (1) the empirical hypothesis that our capacity to relive or rekindle contentful events is the most important feature of consciousness-- indeed, as close to a defining feature of consciousness as we will ever find; and (2) the empirical hypothesis that this echoic capacity is due in large part to habits of self-stimulation that we pick up from human culture, that the Joycean machine in our brains is a virtual machine made of memes”. Leher's stuff seems to put a different spin on that. So does highly evolved consciousness imply indecisive schizoid meltdown? Bring on the Soma..
Thanks, Roge.
ReplyDeleteWhatever the future holds, or whether it holds anything at all, I think, on balance, I'd rather not go there. One way or the other, it probably won't get much better for the likes of me. For that reason, I'm investing heavily in the present. You can say what you like about it, but at least the present is moderately predictable (human nature being what it is).
ReplyDeleteNeil! It's been so long, hasn't it?
ReplyDeleteJohn asked whether we need to take Lovelock seriously. Strangely, without aiming to I ended up last night in a January interview with him, quite possibly the first words of his I've ever read. And I'd suggest taking him totally seriously in this:
ReplyDeleteMost of the "green" stuff is verging on a gigantic scam. Carbon trading, with its huge government subsidies, is just what finance and industry wanted. It's not going to do a damn thing about climate change, but it'll make a lot of money for a lot of people and postpone the moment of reckoning
Extremely well said. As you'd expect from one of Appleyard's chums.
On the other hand he's certain CO2 is a big problem and mean temperature increases of 2degC and maybe 4degC are a real threat. I don't think that is science, Bryan, in that it comes from models or theories that have not be confirmed by the real world. So it is punditry and the question is fair.
But this also reminds me that pundits get it right and wrong at the same time. Think Churchill in the 30s, on Germany and India. And where he was right he was crucially right. There should have been no half-measures in following, saving the world from millions of horribly violent deaths. As long as you knew which was which.
I'm well impressed by that statement about carbon trading. That needs to be said. (How I got to read it is also interesting, to me, but I'll leave that for now.)
Yes, Bryan, it has been a while. My life has taken some twists and turns. But I have been looking in regularly but just not commenting.
ReplyDeleteMaybe the reason pundits are wrong is because they fail to take into account the reaction to their own advice.
ReplyDeleteFor example if I say, "You're heading over a cliff!" you realize your predicament and swerve to avoid going over the cliff.
Likewise if the pundits say, "Stock A is undervalued!" then everyone rushes to buy Stock A so that it becomes overpriced.
The only way to avoid this contrarian effect is to be a voice in the wilderness that nobody listens to. But once you gain a reputation for sagacity and people start paying attention to you, then your predictive power ceases.
Shrew, Anon, a version of the self-correction issue I think.
ReplyDeleteInteresting post! Tetlock's work is pretty cool, but this point has been made before, over 100 years ago, in the great paper "The Method of Multiple Working Hypotheses" by TC Chamberlin:
ReplyDelete"The moment one has offered an original explaniation for a phenomenon which seems satisfactory, that moment affection for his intellectual child springs into existence.... There is an unconscious selection and magnifying of the phenomena that fall into harmony with the theory and support it, and an unconscious neglect of those that fail of coincidence... There springs up, also, an unconscious pressing of the facts to make them fit the theory.... The search for facts, the observation of phenomena and their interpretation are all dominated by affection for the favored theory...."
Chamberlin understood that his theory of multiple working hypotheses, where a thinker is constantly examining problems with multiple lenses, could counteract the biases related to "intellectual parentage" through an "effort to bring into view every rational explanation of new phenomena, and to develop every tenable hypothesis respecting their cause and history." His target audience where the fellow hard scientists who read Science, the journal in which his article was published, but his theory can be applied to the social sciences just as much as biology or physics.
We've all assumed, and now Tetlock has proven, that political pundits are usually full of hot air, and Tetlock is right on target when he blames closed-minded hubris. Hubris and bias toward "the facts that fall happily into the embrace of theory, [but] a natural coldness toward those that seem refractory" don't mix well with rational scientific thought and progress. Pundits (and politicians) today find success only if they can carve out a deep niche in which they can reliably spew out comfortable and predictable opinions that are easily washed down by a cool glass of pre-favored theories, rather than a hard-to-swallow shot of openness toward multiple theories and other possible explanations. This is why they are so often wrong: pundits today don't actually give the problem much thought in its own broader context. They think and write, but more about how to fit a problem into their well-established box than about how one would address the problem in a rational and creative way.
John Platt updated and reiterated Chamberlin's theory in another great paper in Science in 1964. By then, multiple working hypotheses had technically become the norm, but Platt felt that the scientific community needed a reminder that science progresses best by "strong inference," (also the title of his paper) which he sums up in 3 steps (taken verbatim):
1) Devising alternative hypotheses;
2) Devising a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypothesis;
3) Carrying out the experiment so as to get a clean result.
This all seems pretty basic, but Platt points out that nobody really does it anymore. Oftentimes steps 1 and 2, those that "require intellectual inventions, which must be cleverly chosen so that hypothesis, experiment , outcome and exclusion will be related in a rigorous syllogism" are passed up entirely, and the previously proposed theories are lazily accepted as the touchstone for everything that the world throws at them. Pundits have it tough; political or economic science doesn't afford the same opportunities for experimentation, but this doesn't mean that pundits and politicians can still skip steps 1 and 2; even in a thought experiment, multiple outcomes and multiple hypotheses must be considered and eliminated using whatever data is available and a hearty helping of reason. Platt goes on to praise figures like Louis Pasteur, who "every two or three years...moved to one biological problem from another," bringing to each "a completely different method of reasoning." Today's pundits, as well as many academic political scientists and economists, become too well known as "the guy who's against free trade," or "the fervent neoconservative," for example; many may earn some fame for a new theory, but then settle into polishing their gem and using sloppy reason to bend it to answer every problem, and every circumstance. Since Reagan, the GOP has proposed tax cuts as the proper and appropriate reaction to every economic change: the economy is expanding? Tax cuts! The economy is marginally shrinking? Tax cuts! The economy is in a recession? Tax cuts! At no juncture do they entertain any other hypothesis, nor the possibility of any other outcome.
Platt also was aiming his paper at hard scientists, but social scientists (and pundits and journalists) ought to give it a good hard read. At the very least, I have confidence that the Obama administration has embraced strong inference, at least more than any other administration in the recent past. Obama has already admitted mistakes, and realized that every theory, and every idea may not be the right one. Rigorous evaluation of multiple proposals and theories (including those that may not fit the historical American definitions of liberal) to fix the economy and run the government seems to be taking place more with Obama, and this is certainly a good thing. Let's just hope that he too doesn't settle in with what's familiar, and continues to be a strong thinker throughout his time in office.
If anyone is interested:
Chamberlin, T.C., "The Method of Multiple Working Hypotheses" -- http://arti.vub.ac.be/cursus/2005-2006/mwo/chamberlin1890science.pdf
Platt, John, "Strong Inference," -- http://pages.cs.wisc.edu/~markhill/science64_strong_inference.pdf
Hang on, if you say that you're wrong about everything, wouldn't you be right 66% of the time? Or always? Or... wait a minute... never? Um... is this like the one where one village always tells the truth and the other one always lies?
ReplyDeleteYou have opened a philosophical can of worms, sir. Maybe it would be simpler if you just stuck with thinking you're right and being wrong.
Good post! Bless you for reminding me of Tetlock! It seems to me pundits are in the business of trying to persuade people to adopt or support various views and policies. That can be significantly different from the business of providing analysis and insight into what's going on. The latter appears to be just a Trojan Horse for the pundits to hide their persuasive work in.
ReplyDeleteI'm considering that, Anon, I feel it may earn me the admiration of beautiful women. The loser thing doesn't. And my pleasure, Paul.
ReplyDelete