Malcolm Gladwell on Neural Networks That ‘Solve’ Complex Problems

neural netIn last week’s New Yorker, Malcolm Gladwell describes the neural network systems that have been designed, by two independent groups, to address two complex commercial problems: predicting the success of a new popular song, and predicting the success of a new movie. The article is called The Formula.

Although neural networks (collectors of massive amounts of data that then seek ‘meaningful’ patterns in that data that can be used to infer causality or at least correlation) have been around for years, most students of complex adaptive systems believe that complex problems (like global poverty, global warming, or lack of innovation in big business) can never be ‘solved’ because there are simply too many variables (perhaps an infinite number) to allow any kind of exhaustive correlation or useful predictive models to be built. The closest we can hope to get, most complexity theorists would tell us, is interventions that would have a positive impact.

The complex problems that Gladwell’s subjects have addressed do have a lot of variables — the factors that determine popular opinion on a song or movie are myriad and often seemingly unfathomable — but the number of variables is finite. What’s more, the problem-solvers believed that, in the matters they were concerned with, a reasonably small number of variables (certainly a number manageable by today’s computer systems) were disproportionately responsible for a song or movie’s success or failure. And because we’re talking about a product, something that can be ‘put in a box’, the challenge of identifying these variables is less problematic than the challenge of, say, identifying all the variables needed to predict when and where the next major hurricane or pandemic disease will hit.

One could say there are complex problems and there are complex problems, and some are more complex than others. Despite this, the designers of the ‘expert systems’ that are now being used to predict song and movie success — with remarkable precision — faced ridicule and rejection from sponsors and customers because of the prevailing belief that, when it comes to predicting these things, as movie mogul William Goldman put it, “nobody knows anything”.

The key to neural network analysis, besides the computing power to do a lot of iterations with a lot of variables to look for patterns, is patience — when the first few hundred variables don’t pan out, you set them aside and look at a few hundred new ones, and keep adding until some pattern finally emerges.

The variables used in the song success predictor involve the song’s structural components: melody, harmony, rhythm, beat, tempo, octave, pitch, chord progression, cadence, sonic brilliance, frequency and so on. The predictor, called Platinum Blue, has analyzed a huge number of songs of many different genres and found the same patterns that resonate with us in popular music can be found in classical and folk music from all different eras. Popular songs, it finds, fall into clusters — the variables of those songs, taken in aggregate, exhibit similar sets of patterns. The details, of course, are confidential.

If a song falls outside these clusters, the designers of Platinum Blue can tell you which variables need to be changed, and roughly in what way, to get it back into the success strike zone, but beyond that it is up to the artist — how to change the composition is beyond the capacity of the predictor. But once the artist has made the changes, the predictor can tell whether the result is within one of the clusters, and how much the changes mean to the expected revenues from the song. The predictor’s greatest claim to fame is its prediction of enormous success for the CD “Come Away With Me” by then-little-known Norah Jones.

What is most remarkable about Platinum Blue is its wonderful vindication of the talents of writers and composers: It predicts the success of the song regardless of its lyrics or performer, based only on its compositional qualities, its mathematical structure.

A different group, Epagogix, has found the same thing applies to Hollywood film releases — the ultimate popularity and success of a film depends on the qualities of its composition, not on the ‘stars’ attached to it who get paid all the money. Plot lines, the ingredients of particular scenes, characters, and settings matter. That’s not to say that stars don’t get people into theatres, at least until word of mouth begins to prevail over advance billing. But Epagogix can tell you that if the stars want $40 million and the rest of the movie costs another $20 million, whether the $60 million investment will be a winning or a losing one.

Epagogix analyzed the 2005 movie The Interpreter, which went through several massive and well-documented changes before it was finally released. It concluded that the changes had made a probable $33 million film into a $69 million film (within $4 million of the actual revenues), and then pointed out the ways in which it could, with relatively few changes and for very little additional investment, have become a $150 million-plus blockbuster. Epagogix analyzed another (unidentified) film and predicted would make $47 million as scripted and $72 million of three minor changes were made — the changes were not made and the film grossed $50 million. If you’ve seen The Interpreter (I haven’t) you can probably assess better than I can whether the changes they proposed would have made it a better film or not — and the authors of both Platinum Blue and Epigogix both assert that it’s not enough to have the right ingredients — the product has to work artistically as a whole as well, and no neural network system will tell you how to do that.

What the neural network systems can’t do is identify and parse the variables to consider. That takes a combination of a knowledge of the art form, and how it’s appreciated by audiences, and also a great deal of imagination, to keep honing and refining and trying new variables until the ones that really matter start to emerge from the pattern matching. This is in some ways as much an art as the writing and making of a song or film itself.

The Wisdom of Crowds is another type of neural network system, the difference being that the ‘crowd’ doesn’t explicitly identify the relevant variables it considers in rendering its judgement. The point is that the group using that wisdom is concerned with coming up with the ‘right’ answer (prediction, choice, critical information etc.) and not overly concerned with how the crowd came up with it. One WoC application, the Iowa Electronic Markets, predicted that Bush would win the 2004 election while most of the other expert predictors were saying Kerry would win. How did they know? Among the crowd they had knowledge of thousands of subtle, unidentified variables that individual experts could never know.

We would be right to be skeptical of what we’re told about Platinum Blue and Epigogix. Despite what we’re told about their success, there is enough secretiveness about both projects that it is possible that both products are hoaxes, or at least much less accurate than their authors allege.

But suppose they are the real deal? What other complex problems could similar neural network systems be applied to ‘solve’? As Gladwell points out, you could apply them to win at the racetrack, and perhaps even in the stock market. You could probably use them to devise the best possible new product design process. But could you use them to analyze all the education programs in the world and come up with the ideal curriculum for self-sufficiency and critical skills learning? Or all the health systems in the world to design a hybrid system that offered the best of all worlds? Or how about using them to develop a revenue-neutral tax system that would actually change behaviours to reduce greenhouse gases sufficient to end the threat of global warming?

I’m not so sure, even if neural networks could solve some straightforward complex problems, that they’re up to the challenge of helping us grapple with the ‘wicked’ ones that have defeated us for centuries, and which even threaten our civilization’s demise. Even if it were possible to employ them to such ends, it would take a great deal of passion, patience, commitment and imagination to go along withan astonishingly sophisticated and massive pattern-seeking technology.

But as one more tool in the Coping with Complexity toolkit, it sure couldn’t hurt.

Image: NASA’s depiction of the neural network of a single plant.

This entry was posted in Working Smarter. Bookmark the permalink.

3 Responses to Malcolm Gladwell on Neural Networks That ‘Solve’ Complex Problems

  1. PaulSweeney says:

    I think I saw this gladwell article reviewed elsewhere and the key missing factor was the “outcome measures” of success. These were said to be very fuzzy indeed. If I find the article I will link it back to you.

  2. andrew says:

    Even if it were possible to employ them to such ends, it would take a great deal of passion, patience, commitment and imagination to go along with an astonishingly sophisticated and massive pattern-seeking technology.Dave, we could all try expanding the capacity of ”wet ware” aka the human mind ;-)aka learning authentically instead of through machines and bunkum theories and the Gladwells of this world who parse science and pseudo science and sell it on like candy in a WalMart ;-) andrew

  3. Dave Pollard says:

    Paul: Thanks. I’d like to see that. Andrew: I think that’s a tad harsh. Gladwell has done a lot to bring some very innovative ideas into public discourse. To anyone who can make money popularizing the concept of Tipping Points in change processes, I say “Good on yer, mate”. As you know I’m a techno-skeptic, but I’ve learned enough about neural nets to respect what they are good for (which is finding correlations to help address somewhat complex, not very complex, problems). But what if we could get all those thousands of PCs futilely looking for SETI, instead working on finding patterns that would help us address global warming or world poverty?

Comments are closed.