MORE ON THE WISDOM OF CROWDS

The book begins with a taxonomy of three types of problems that individuals and groups try to solve:

• Cognition problems: Problems with one definitive answer that we try to accurately assess after considering available and missing information (e.g. what’s a stock worth, who will win an election, or what caused a disaster),
• Coordination problems: Where an optimal combined solution is needed for a problem that affects a whole group, and where this optimal solution is usually sought by having each individual act in personal self-interest (e.g. finding buyers and sellers for products, or determining the best route to work in traffic), and
• Cooperation problems: Where an optimal combined solution is needed for a problem that affects a whole group, and where this optimal solution usually depends on individuals trusting each other and acting fairly and in what they perceive to be collective self-interest rather than just their own (e.g. how to deal with pollution, devise a tax system, or remunerate employees).

One of Surowiecki’s principal arguments is that all three types of problems are best solved by canvassing groups (the larger the better) of reasonably informed and engaged people. The group’s answer, he shows, is almost invariably better than any expert’s answer, even better than the best answer of the experts in the group. The group’s answer is the collective answer, a term Surowiecki prefers to ‘average’ or ‘consensus’ answer, which aren’t always the same thing. And the superiority of the collective answer depends importantly on the group’s members having three qualities:

• Intellectual diversity: Different opinions and perspectives (unlike most management teams and boards, who tend to select others who think the same way they do),
• Independence: Freedom from the tendency to want to agree automatically with what one or more other group members says, and
• Decentralization with Aggregation: Individual access to different, specialized knowledge, and a mechanism for effectively sharing that knowledge with the rest of the group.

Much of the book describes the three types of problems, with copious examples (some of them quite entertaining), justifying the three conditions for collective wisdom, and drawing some remarkable inferences about what all this means to some important decision-making processes in our modern world. Surowiecki argues that too much communication can actually render a group dysfunctional, victims of ‘analysis paralysis’. Likewise, he demonstrates, too much consensus or compromise produces weak, suboptimal solutions, since opposing views, which must have had some basis for belief, are not adequately aired and weighed.

Surowiecki drops his first bombshell early: “There is no evidence that one can become expert in something as broad as decision-making, policy, or strategy…or perhaps even management. … Large groups of diverse individuals will make more intelligent decisions than even the most skilled decision-maker.” The implication of this is that business executives, expert consultants, investment analysts, egomaniacal doctors and heads of state are not competent to make important decisions related to cognitive, coordination or cooperation problems, and should always defer to the collective wisdom of large diverse groups when such problems arise. In fact, the implication is that these individuals (who Surowiecki argues are grossly overpaid) have far too much power in our society, and may be doing us far more harm than good. Instinctively we have always known that more egalitarian social structures produce better answers; now we have some empirical evidence to support this ‘uncommon’ wisdom. Our individual perception and intuition has been denigrated in the civilized world to the point where the personal judgements of those in high-ranking positions in our hierarchies unreasonably carry more weight than group judgement. Get a second opinion, and a third and a fourth… and you’ll get a better answer and make a more intelligent decision. These overpaid individuals would be much better employed using their expertise to implement the group’s decision instead of making it for them.

The book describes at length the phenomenon of groupthink and how it biases groups’ decisions and gives collective wisdom a bad name. In fact there are four phenomena at work: The tendency of groups to excessively rationalize away minority views as improbable, the shyness of individuals to voice the first opposing view in the face of an apparent consensus, the tendency to accept consensus of a small number as inherent ‘proof’ of that consensus’ validity, and the bandwagon tendency of groups to be infected by what Gladwell in The Tipping Point called an ‘epidemic’. These are all subtly different phenomena, and they’re natural behaviours, but they’re irrational, and have led to great skepticism about collective wisdom. These phenomena show up in wild swings in popular opinion, in the inexplicable and transient popularity of crazes, in market ‘bubbles’, in mob violence, in our willingness to let one person dominate the discussion and bully us into accepting his view (even if he’s not the most senior or eloquent person in the group), and in our ability to be brainwashed and engage in barbaric and irrational behaviours. No wonder, then, that so many view the wisdom of crowds as an oxymoron, and are so easily seduced by leaders and experts even when their records of decision-making have been deplorable.

I recently attended a conference that had great promise, but which in my view wasn’t anywhere near living up to that promise. I spoke to a few attendees individually and was astonished when they nodded their heads in strong agreement after I said how disappointed I was. They encouraged me to speak up, to rescue the event. But as a junior attendee, I rationalized that it wasn’t my place to hijack the meeting, and said nothing. Truth is, I simply lacked the courage to do so.

Perhaps because he’s an American, living in the land where selfishness is seen as a virtue and ‘natural’ behaviour, Surowiecki agonizes at some length over what he sees as the ‘irrationality’ of subordinating personal self-interest to collective self-interest in Cooperation Problem situations. But while he argues that this is a learned behaviour, I would argue that it’s an instinctive one. It has its roots, I believe, in the subordination of the individual to the local community, something innate in all animal species and evident in tribal human communities as well. It’s in our DNA. The reason it appears irrational or counter-intuitive is that we’ve lost this sense of community, and instead we have nuclear families and huge impersonal political entities. Neither of these are communities, or even fundamentally democratic structures as true communities are. But we’ve got this sense that ‘community comes first’ in our genes, and it’s not surprising at all to me that we exhibit that instinctive altruism, even with strangers: Giving to charity, caring about what’s happening in Sudan, taking ‘irrational’ responsibility. Our instincts often quietly trump our reason.

This book forces you to rethink almost everything you believe. Here are a few of its implications for the things I particularly care about:

• How to Save the World: Despite Margaret Mead’s famous saying about the impact of a few caring people, maybe instead of my proposed ‘think tank’ of experts working on solutions to our intractable problems, we need to ask a lot more people for their take on what the coming scenarios might be, and how to avoid the worst of them. At the very least, the think-tank cannot be self-selecting, and its membership will need some careful and objective crafting to ensure it meets Surowiecki’s three qualifications.
• Innovation: My article earlier this week reflects an amendment to my earlier thinking on optimal innovation processes, lessening reliance on a few unrepresentative ‘pathfinder’ customers and drawing more expansively on collective wisdom, both to find and qualify innovative ideas.
• The Blogosphere: Shirky’s Law tells us that we tend to read a few, early-launched blogs on any subject, and largely ignore the late-comers. We might be better informed, and in a better position to make judgements and decisions on these subjects, if we instead diversified our reading, selecting which bloggers to read largely at random (Technorati has a tool that allows you to find every blog that has written about a subject, by typing in keywords). And perhaps Google, by pointing us to the most popular sites on a subject rather than listing them randomly, may be doing us an intellectual disservice, reinforcing conventional wisdom instead of helping us find collective wisdom. That pretty well rules out the blogosphere and its corners as true communities. But maybe it means the fact my blogroll is so large and unwieldy, and the fact I don’t read any one blog with great regularity, is a good thing.
• Liberal vs. Conservative: The polarization we are seeing in America may be more illusion than fact. Perhaps what is happening is that a minority of both liberals and conservatives are becoming (thanks to the Internet etc.) more informed, and that is entrenching their beliefs, but the vast majority of people are largely uninformed and are drifting with the tide, subject to all the vagaries of Groupthink. That suggests that, despite the pollsters’ claims that very few US voters are undecided, a very large number may still be up for grabs, and could swing en masse, perhaps irrationally. It also suggests that the decision may not be as cut-and-dried as we would like to think. The fact that there is no collective wisdom on which is the better candidate may indicate that the election result isn’t as important as we all seem to believe. That’s a scary thought, to both sides. In the recent Canadian election, the collective wisdom of voters was ‘we don’t trust any of you, so we’re going to elect a minority government to keep you all in check’. And they did just that, despite enormous pressure to vote ‘strategically’.
• Expertise Finders: Obviously, if you buy Surowiecki’s argument that experts aren’t worth their price tag, you have to wonder whether technology that will help us find experts is really all that important, and whether perhaps we should instead be looking at ways to create and tap into large, diverse impromptu groups that can better help us decide what’s really going to happen in our businesses and what to do about it.
• Knowledge Management: The wisdom of crowds belies the assessment that ‘best practices’ will or can be identified and volunteered by individuals in the organization. And when he points out the importance of ‘Decentralization with Aggregation’ as a condition of high quality group judgement, Surowiecki puts his finger on the greatest challenge of KM: How to aggregate each individual’s knowledge so that it can be effectively shared with the entire organization. He even suggests some answers to this challenge, which imply that neither ‘filtered’ knowledge (edited and culled by an expert group) nor ‘unfiltered’ knowledge (simply collected with no editing, rating or comment) is the optimal answer.

Ultimately, though, it’s the well-picked and cleverly-told illustrations and examples, not the theory, that makes The Wisdom of Crowds such a joy to read. When he’s describing the ability of a group to accurately pick the number of jellybeans in a jar, or explaining the instinctive coordination ‘rules’ that allow huge flocks of birds to sweep through the sky as if they were one huge organism, or describing examples of how large groups solve problems brilliantly even when they are not individually or collectively aware of the knowledge they are bringing to bear in doing so, or explaining why we begrudgingly pay taxes until we think the number of cheaters is too high, or why television shows are so awful and why traffic jams occur and how they might be eliminated without reducing the number of cars on the road, Surowiecki is at his best. Buy it, read it more than once, cherish it, think about all it means. This is a profoundly important book.