James Surowiecki writes the financial column in The New Yorker, so even before his book The Wisdom of Crowds came out I knew he did good research, and wrote well. But the book exceeded my high expectations: It is the best-written book I’ve read in years — clear, clever, accessible, and jammed full of brilliant, original thinking and compelling, supporting stories and reference. The only disappointment is that, following an annoying recent tradition in non-fiction, the book has no index. If you haven’t read it already, put it on your list, and on your gift list — I can’t imagine anyone not learning something important from it, and it’s fun to read. The book begins with a taxonomy of three types of problems that individuals and groups try to solve:
One of Surowiecki’s principal arguments is that all three types of problems are best solved by canvassing groups (the larger the better) of reasonably informed and engaged people. The group’s answer, he shows, is almost invariably better than any expert’s answer, even better than the best answer of the experts in the group. The group’s answer is the collective answer, a term Surowiecki prefers to ‘average’ or ‘consensus’ answer, which aren’t always the same thing. And the superiority of the collective answer depends importantly on the group’s members having three qualities:
Much of the book describes the three types of problems, with copious examples (some of them quite entertaining), justifying the three conditions for collective wisdom, and drawing some remarkable inferences about what all this means to some important decision-making processes in our modern world. Surowiecki argues that too much communication can actually render a group dysfunctional, victims of ‘analysis paralysis’. Likewise, he demonstrates, too much consensus or compromise produces weak, suboptimal solutions, since opposing views, which must have had some basis for belief, are not adequately aired and weighed. Surowiecki drops his first bombshell early: “There is no evidence that one can become expert in something as broad as decision-making, policy, or strategy…or perhaps even management. … Large groups of diverse individuals will make more intelligent decisions than even the most skilled decision-maker.” The implication of this is that business executives, expert consultants, investment analysts, egomaniacal doctors and heads of state are not competent to make important decisions related to cognitive, coordination or cooperation problems, and should always defer to the collective wisdom of large diverse groups when such problems arise. In fact, the implication is that these individuals (who Surowiecki argues are grossly overpaid) have far too much power in our society, and may be doing us far more harm than good. Instinctively we have always known that more egalitarian social structures produce better answers; now we have some empirical evidence to support this ‘uncommon’ wisdom. Our individual perception and intuition has been denigrated in the civilized world to the point where the personal judgements of those in high-ranking positions in our hierarchies unreasonably carry more weight than group judgement. Get a second opinion, and a third and a fourth… and you’ll get a better answer and make a more intelligent decision. These overpaid individuals would be much better employed using their expertise to implement the group’s decision instead of making it for them. The book describes at length the phenomenon of groupthink and how it biases groups’ decisions and gives collective wisdom a bad name. In fact there are four phenomena at work: The tendency of groups to excessively rationalize away minority views as improbable, the shyness of individuals to voice the first opposing view in the face of an apparent consensus, the tendency to accept consensus of a small number as inherent ‘proof’ of that consensus’ validity, and the bandwagon tendency of groups to be infected by what Gladwell in The Tipping Point called an ‘epidemic’. These are all subtly different phenomena, and they’re natural behaviours, but they’re irrational, and have led to great skepticism about collective wisdom. These phenomena show up in wild swings in popular opinion, in the inexplicable and transient popularity of crazes, in market ‘bubbles’, in mob violence, in our willingness to let one person dominate the discussion and bully us into accepting his view (even if he’s not the most senior or eloquent person in the group), and in our ability to be brainwashed and engage in barbaric and irrational behaviours. No wonder, then, that so many view the wisdom of crowds as an oxymoron, and are so easily seduced by leaders and experts even when their records of decision-making have been deplorable. I recently attended a conference that had great promise, but which in my view wasn’t anywhere near living up to that promise. I spoke to a few attendees individually and was astonished when they nodded their heads in strong agreement after I said how disappointed I was. They encouraged me to speak up, to rescue the event. But as a junior attendee, I rationalized that it wasn’t my place to hijack the meeting, and said nothing. Truth is, I simply lacked the courage to do so. Perhaps because he’s an American, living in the land where selfishness is seen as a virtue and ‘natural’ behaviour, Surowiecki agonizes at some length over what he sees as the ‘irrationality’ of subordinating personal self-interest to collective self-interest in Cooperation Problem situations. But while he argues that this is a learned behaviour, I would argue that it’s an instinctive one. It has its roots, I believe, in the subordination of the individual to the local community, something innate in all animal species and evident in tribal human communities as well. It’s in our DNA. The reason it appears irrational or counter-intuitive is that we’ve lost this sense of community, and instead we have nuclear families and huge impersonal political entities. Neither of these are communities, or even fundamentally democratic structures as true communities are. But we’ve got this sense that ‘community comes first’ in our genes, and it’s not surprising at all to me that we exhibit that instinctive altruism, even with strangers: Giving to charity, caring about what’s happening in Sudan, taking ‘irrational’ responsibility. Our instincts often quietly trump our reason. This book forces you to rethink almost everything you believe. Here are a few of its implications for the things I particularly care about:
Ultimately, though, it’s the well-picked and cleverly-told illustrations and examples, not the theory, that makes The Wisdom of Crowds such a joy to read. When he’s describing the ability of a group to accurately pick the number of jellybeans in a jar, or explaining the instinctive coordination ‘rules’ that allow huge flocks of birds to sweep through the sky as if they were one huge organism, or describing examples of how large groups solve problems brilliantly even when they are not individually or collectively aware of the knowledge they are bringing to bear in doing so, or explaining why we begrudgingly pay taxes until we think the number of cheaters is too high, or why television shows are so awful and why traffic jams occur and how they might be eliminated without reducing the number of cars on the road, Surowiecki is at his best. Buy it, read it more than once, cherish it, think about all it means. This is a profoundly important book. |
This collective thinking business certainly explains MY gut feeling, which is that while I definitely DO want to give input into decisions that need to be made (call me opinionated – many do!), I dislike making unilateral decisions. And I also dislike making “consensus” decisions where someone says “I think we should do X” and then a bunch of people just agree without even thinking. Well-focused meetings by well-prepared individuals almost can’t go wrong. I may point my co-workers to this post tomorrow – so I suppose I shouldn’t have posted this in the comments, eh?It is often said that women in general prefer coming to consensus on issues, and dislike making decisions without buy-in. Does this mean that women are “wiser” than men? :-)
Catnmus: I think that goes without saying ;-) But I also think women are more easily bullied than men in meetings, and prone to suppress their opposing views, which contributes often to the poverty of the group’s ideas and decisions. But then I find that meetings are rarely about making decisions — they’re usually for the boss to collect information and to communicate his/her predetermined decisions.
Regarding your analysis of the Blogosphere as not supporting true community: What if we had hub sites founded in specific offline projects and organisations which displayed aggregated content from the blogs of the people involved, and at the same time provided feeds for that aggregated content? That way, physical community, rooted in and nourished by its place and real-life relationships, finds an expression and forum online, which enables it to transcend place and join the global web. What would also be great would be a search function that one could factor place and social group (explicit “friends” and organic linkedness) so as to “scope” the search more intimately than the globalised Google.
One of the (many) problems I had with Surowiecki’s book is that he stops short of drawing out his ideas into action. He doesn’t immediately attack the obscene cult of the individual or say that much about the rise of the CEO as hero and superhuman leader. As you say Dave, this may be a cognitive dissonance with the author’s American origins. Also he says very little about the rise of the expert. The reason we have “experts” is nothing to do how good their decisons are- it because we demand responsibility (ie someone to punish) and leadership (someone to follow). Leaders, if you like, respresent our dark side. If you want to read a new (and very funny) tale about the idiocy of crowds look at http://www.join-me.co.uk/story.html
Luke: This technology has intriguing possibilities, but technology capability does not in itself make community. There needs to be trust and caring and some skin in the game, and I don’t know how (or if) the blogosphere can generate those, beyond tiny circles of motivated people. Some people blog precisely because it’s anonymous, and community is not what they’re looking for (they’ll take adulation, though). Ahmed: Maybe he’ll suggest actions in his next book. I actually liked the fact he didn’t this time because this book had me thinking about implications and actions for myself. He might also suggest that it’s the crowd’s collective wisdom about what the actions should be that counts, not his ‘expert’ opinion. And his point is that we shouldn’t hold leaders so responsible (or pay them so much or give them so much authority) — we need to take, individually and collectively, more responsibility ourselves.
Dave: I agree that technology does not make community in itself
I think all of this comes down to two things. We need to promote, teach, encourage and reward independent thought and we also need to build into the decision making process the gathering of a variety of these indepenent thoughts, not necessarily to create a consensus, but to create a thorough, workable, and hopefully successful solution. Too often independent thought is punished in our society today. Politicians who don’t tow the party line get punished. Middle management who don’t succomb to the will of upper management don’t get promotions and are often the first to be let go in downsizing. This needs to change especially in our political landscape.
From your description Surowiecki’s wisdom about the wisdom of crowds seems to boil down to the principle that collective decisions are better than individual ones, as long as you have enough (but not too much) intellectual diversity, independence and decentralization of information-gathering and analysis. Which leaves us with the questions, how much of these things are enough, and how much too much.
The mention of Margaret Mead’s saying about the impact of a few caring people, and the discussions about experts and leaders (not the same thing, by the way!), leads me to wonder if perhaps a wise crowd also needs some of each of these, as well as the aspects Surowiecki mentions.One characteristic of genuine experts is that they’re “meme generators”, and some of the generated memes may catch on and spread.A genuine leader can make the crowd more effective, recognizing the power and enabling it, cutting through unproductive thrashing, etc.A group of “a few caring people” can provide a kind of exemplar of a possibility currently unexamined by the majority, a seed that may sprout if/when the conditions are right.That doesn’t diminish Surowiecki’s argument at all; without the “wise crowd”, these entities would be worthless, and with a “dumb crowd”, they can be dangerous. (For example, the neocons are a group of a few caring people…)
In the world of Internet collective decision-making, Presidents could do more presiding and more executing of policies proposed, refined and decided for and by the governed. But in an electronic democracy, the ultimate problems are no different than among the Athenians. They revolve around non-voters and uninformed voters. We may actually create laws that penalize non-voters (And some constituencies have done just that.) But how to deal with the ignorant voter? And how to build into our educational system — some fundamental learning of the realities of political judging.
Thanks for your great articles,To your success alwaysTracy Howww.wisdomgettingloaded.com/