James Surowiecki writes the financial column in The New Yorker, so even before his book The Wisdom of Crowds came out I knew he did good research, and wrote well. But the book exceeded my high expectations: It is the best-written book I’ve read in years — clear, clever, accessible, and jammed full of brilliant, original thinking and compelling, supporting stories and reference. The only disappointment is that, following an annoying recent tradition in non-fiction, the book has no index. If you haven’t read it already, put it on your list, and on your gift list — I can’t imagine anyone not learning something important from it, and it’s fun to read.
The book begins with a taxonomy of three types of problems that individuals and groups try to solve:
One of Surowiecki’s principal arguments is that all three types of problems are best solved by canvassing groups (the larger the better) of reasonably informed and engaged people. The group’s answer, he shows, is almost invariably better than any expert’s answer, even better than the best answer of the experts in the group. The group’s answer is the collective answer, a term Surowiecki prefers to ‘average’ or ‘consensus’ answer, which aren’t always the same thing. And the superiority of the collective answer depends importantly on the group’s members having three qualities:
Much of the book describes the three types of problems, with copious examples (some of them quite entertaining), justifying the three conditions for collective wisdom, and drawing some remarkable inferences about what all this means to some important decision-making processes in our modern world. Surowiecki argues that too much communication can actually render a group dysfunctional, victims of ‘analysis paralysis’. Likewise, he demonstrates, too much consensus or compromise produces weak, suboptimal solutions, since opposing views, which must have had some basis for belief, are not adequately aired and weighed.
Surowiecki drops his first bombshell early: “There is no evidence that one can become expert in something as broad as decision-making, policy, or strategy…or perhaps even management. … Large groups of diverse individuals will make more intelligent decisions than even the most skilled decision-maker.” The implication of this is that business executives, expert consultants, investment analysts, egomaniacal doctors and heads of state are not competent to make important decisions related to cognitive, coordination or cooperation problems, and should always defer to the collective wisdom of large diverse groups when such problems arise. In fact, the implication is that these individuals (who Surowiecki argues are grossly overpaid) have far too much power in our society, and may be doing us far more harm than good. Instinctively we have always known that more egalitarian social structures produce better answers; now we have some empirical evidence to support this ‘uncommon’ wisdom. Our individual perception and intuition has been denigrated in the civilized world to the point where the personal judgements of those in high-ranking positions in our hierarchies unreasonably carry more weight than group judgement. Get a second opinion, and a third and a fourth… and you’ll get a better answer and make a more intelligent decision. These overpaid individuals would be much better employed using their expertise to implement the group’s decision instead of making it for them.
The book describes at length the phenomenon of groupthink and how it biases groups’ decisions and gives collective wisdom a bad name. In fact there are four phenomena at work: The tendency of groups to excessively rationalize away minority views as improbable, the shyness of individuals to voice the first opposing view in the face of an apparent consensus, the tendency to accept consensus of a small number as inherent ‘proof’ of that consensus’ validity, and the bandwagon tendency of groups to be infected by what Gladwell in The Tipping Point called an ‘epidemic’. These are all subtly different phenomena, and they’re natural behaviours, but they’re irrational, and have led to great skepticism about collective wisdom. These phenomena show up in wild swings in popular opinion, in the inexplicable and transient popularity of crazes, in market ‘bubbles’, in mob violence, in our willingness to let one person dominate the discussion and bully us into accepting his view (even if he’s not the most senior or eloquent person in the group), and in our ability to be brainwashed and engage in barbaric and irrational behaviours. No wonder, then, that so many view the wisdom of crowds as an oxymoron, and are so easily seduced by leaders and experts even when their records of decision-making have been deplorable.
I recently attended a conference that had great promise, but which in my view wasn’t anywhere near living up to that promise. I spoke to a few attendees individually and was astonished when they nodded their heads in strong agreement after I said how disappointed I was. They encouraged me to speak up, to rescue the event. But as a junior attendee, I rationalized that it wasn’t my place to hijack the meeting, and said nothing. Truth is, I simply lacked the courage to do so.
Perhaps because he’s an American, living in the land where selfishness is seen as a virtue and ‘natural’ behaviour, Surowiecki agonizes at some length over what he sees as the ‘irrationality’ of subordinating personal self-interest to collective self-interest in Cooperation Problem situations. But while he argues that this is a learned behaviour, I would argue that it’s an instinctive one. It has its roots, I believe, in the subordination of the individual to the local community, something innate in all animal species and evident in tribal human communities as well. It’s in our DNA. The reason it appears irrational or counter-intuitive is that we’ve lost this sense of community, and instead we have nuclear families and huge impersonal political entities. Neither of these are communities, or even fundamentally democratic structures as true communities are. But we’ve got this sense that ‘community comes first’ in our genes, and it’s not surprising at all to me that we exhibit that instinctive altruism, even with strangers: Giving to charity, caring about what’s happening in Sudan, taking ‘irrational’ responsibility. Our instincts often quietly trump our reason.
This book forces you to rethink almost everything you believe. Here are a few of its implications for the things I particularly care about:
Ultimately, though, it’s the well-picked and cleverly-told illustrations and examples, not the theory, that makes The Wisdom of Crowds such a joy to read. When he’s describing the ability of a group to accurately pick the number of jellybeans in a jar, or explaining the instinctive coordination ‘rules’ that allow huge flocks of birds to sweep through the sky as if they were one huge organism, or describing examples of how large groups solve problems brilliantly even when they are not individually or collectively aware of the knowledge they are bringing to bear in doing so, or explaining why we begrudgingly pay taxes until we think the number of cheaters is too high, or why television shows are so awful and why traffic jams occur and how they might be eliminated without reducing the number of cars on the road, Surowiecki is at his best. Buy it, read it more than once, cherish it, think about all it means. This is a profoundly important book.