|In last week’s New Yorker, Malcolm Gladwell describes the neural network systems that have been designed, by two independent groups, to address two complex commercial problems: predicting the success of a new popular song, and predicting the success of a new movie. The article is called The Formula.
Although neural networks (collectors of massive amounts of data that then seek ‘meaningful’ patterns in that data that can be used to infer causality or at least correlation) have been around for years, most students of complex adaptive systems believe that complex problems (like global poverty, global warming, or lack of innovation in big business) can never be ‘solved’ because there are simply too many variables (perhaps an infinite number) to allow any kind of exhaustive correlation or useful predictive models to be built. The closest we can hope to get, most complexity theorists would tell us, is interventions that would have a positive impact.
The complex problems that Gladwell’s subjects have addressed do have a lot of variables — the factors that determine popular opinion on a song or movie are myriad and often seemingly unfathomable — but the number of variables is finite. What’s more, the problem-solvers believed that, in the matters they were concerned with, a reasonably small number of variables (certainly a number manageable by today’s computer systems) were disproportionately responsible for a song or movie’s success or failure. And because we’re talking about a product, something that can be ‘put in a box’, the challenge of identifying these variables is less problematic than the challenge of, say, identifying all the variables needed to predict when and where the next major hurricane or pandemic disease will hit.
One could say there are complex problems and there are complex problems, and some are more complex than others. Despite this, the designers of the ‘expert systems’ that are now being used to predict song and movie success — with remarkable precision — faced ridicule and rejection from sponsors and customers because of the prevailing belief that, when it comes to predicting these things, as movie mogul William Goldman put it, “nobody knows anything”.
The key to neural network analysis, besides the computing power to do a lot of iterations with a lot of variables to look for patterns, is patience — when the first few hundred variables don’t pan out, you set them aside and look at a few hundred new ones, and keep adding until some pattern finally emerges.
The variables used in the song success predictor involve the song’s structural components: melody, harmony, rhythm, beat, tempo, octave, pitch, chord progression, cadence, sonic brilliance, frequency and so on. The predictor, called Platinum Blue, has analyzed a huge number of songs of many different genres and found the same patterns that resonate with us in popular music can be found in classical and folk music from all different eras. Popular songs, it finds, fall into clusters — the variables of those songs, taken in aggregate, exhibit similar sets of patterns. The details, of course, are confidential.
If a song falls outside these clusters, the designers of Platinum Blue can tell you which variables need to be changed, and roughly in what way, to get it back into the success strike zone, but beyond that it is up to the artist — how to change the composition is beyond the capacity of the predictor. But once the artist has made the changes, the predictor can tell whether the result is within one of the clusters, and how much the changes mean to the expected revenues from the song. The predictor’s greatest claim to fame is its prediction of enormous success for the CD “Come Away With Me” by then-little-known Norah Jones.
What is most remarkable about Platinum Blue is its wonderful vindication of the talents of writers and composers: It predicts the success of the song regardless of its lyrics or performer, based only on its compositional qualities, its mathematical structure.
A different group, Epagogix, has found the same thing applies to Hollywood film releases — the ultimate popularity and success of a film depends on the qualities of its composition, not on the ‘stars’ attached to it who get paid all the money. Plot lines, the ingredients of particular scenes, characters, and settings matter. That’s not to say that stars don’t get people into theatres, at least until word of mouth begins to prevail over advance billing. But Epagogix can tell you that if the stars want $40 million and the rest of the movie costs another $20 million, whether the $60 million investment will be a winning or a losing one.
Epagogix analyzed the 2005 movie The Interpreter, which went through several massive and well-documented changes before it was finally released. It concluded that the changes had made a probable $33 million film into a $69 million film (within $4 million of the actual revenues), and then pointed out the ways in which it could, with relatively few changes and for very little additional investment, have become a $150 million-plus blockbuster. Epagogix analyzed another (unidentified) film and predicted would make $47 million as scripted and $72 million of three minor changes were made — the changes were not made and the film grossed $50 million. If you’ve seen The Interpreter (I haven’t) you can probably assess better than I can whether the changes they proposed would have made it a better film or not — and the authors of both Platinum Blue and Epigogix both assert that it’s not enough to have the right ingredients — the product has to work artistically as a whole as well, and no neural network system will tell you how to do that.
What the neural network systems can’t do is identify and parse the variables to consider. That takes a combination of a knowledge of the art form, and how it’s appreciated by audiences, and also a great deal of imagination, to keep honing and refining and trying new variables until the ones that really matter start to emerge from the pattern matching. This is in some ways as much an art as the writing and making of a song or film itself.
The Wisdom of Crowds is another type of neural network system, the difference being that the ‘crowd’ doesn’t explicitly identify the relevant variables it considers in rendering its judgement. The point is that the group using that wisdom is concerned with coming up with the ‘right’ answer (prediction, choice, critical information etc.) and not overly concerned with how the crowd came up with it. One WoC application, the Iowa Electronic Markets, predicted that Bush would win the 2004 election while most of the other expert predictors were saying Kerry would win. How did they know? Among the crowd they had knowledge of thousands of subtle, unidentified variables that individual experts could never know.
We would be right to be skeptical of what we’re told about Platinum Blue and Epigogix. Despite what we’re told about their success, there is enough secretiveness about both projects that it is possible that both products are hoaxes, or at least much less accurate than their authors allege.
But suppose they are the real deal? What other complex problems could similar neural network systems be applied to ‘solve’? As Gladwell points out, you could apply them to win at the racetrack, and perhaps even in the stock market. You could probably use them to devise the best possible new product design process. But could you use them to analyze all the education programs in the world and come up with the ideal curriculum for self-sufficiency and critical skills learning? Or all the health systems in the world to design a hybrid system that offered the best of all worlds? Or how about using them to develop a revenue-neutral tax system that would actually change behaviours to reduce greenhouse gases sufficient to end the threat of global warming?
I’m not so sure, even if neural networks could solve some straightforward complex problems, that they’re up to the challenge of helping us grapple with the ‘wicked’ ones that have defeated us for centuries, and which even threaten our civilization’s demise. Even if it were possible to employ them to such ends, it would take a great deal of passion, patience, commitment and imagination to go along withan astonishingly sophisticated and massive pattern-seeking technology.
But as one more tool in the Coping with Complexity toolkit, it sure couldn’t hurt.
Image: NASA’s depiction of the neural network of a single plant.
Other Writers About CollapseAlbert Bates (US)
Andrew Nikiforuk (CA)
Carolyn Baker (US)*
Catherine Ingram (US)
Chris Hedges (US)
Dahr Jamail (US)
Dark Matter Women Witnessing (CA)
David Petraitis (US)
David Wallace-Wells (US)
Dean Spillane-Walker (US)*
Deena Metzger (US)
Derrick Jensen (US)
Doing It Ourselves (AU)
Dougald & Paul (UK)*
Gail Tverberg (US)
Guy McPherson (US)
Jan Wyllie (UK)
Janaia & Robin (US)*
Jem Bendell (US)
Jonathan Franzen (US)
Kari McGregor (AU)
Keith Farnish (UK)
Kristinha Anding (US)
NTHE Love (UK)
Paul Chefurka (CA)
Paul Heft (US)*
Post Carbon Inst. (US)
Richard Heinberg (US)
Robert Jensen (US)
Roy Scranton (US)
Sam Mitchell (US)
Sam Rose (US)*
Tim Bennett (US)
Tim Garrett (US)
Umair Haque (US)
William Rees (CA)
Archive by Category
My Bio, Contact Info, Signature PostsAbout the Author (2016)
--- My Best 100 Posts --
Preparing for Civilization's End:
What Would Net-Zero Emissions Look Like?
Why Economic Collapse Will Precede Climate Collapse
Being Adaptable: A Reminder List
A Culture of Fear
What Will It Take?
A Future Without Us
Dean Walker Interview (video)
The Mushroom at the End of the World
What Would It Take To Live Sustainably?
The New Political Map (Poster)
Complexity and Collapse
Save the World Reading List
What a Desolated Earth Looks Like
Giving Up on Environmentalism
The Dark & Gathering Sameness of the World
The End of Philosophy
The Boiling Frog
What to Believe Now?
Conversation & Silence
The Language of Our Eyes
Cultural Acedia: When We Can No Longer Care
Several Short Sentences About Learning
Why I Don't Want to Hear Your Story
A Harvest of Myths
The Qualities of a Great Story
The Trouble With Stories
A Model of Identity & Community
Not Ready to Do What's Needed
A Culture of Dependence
So What's Next
Ten Things to Do When You're Feeling Hopeless
No Use to the World Broken
Living in Another World
Does Language Restrict What We Can Think?
The Value of Conversation Manifesto Nobody Knows Anything
If I Only Had 37 Days
The Only Life We Know
A Long Way Down
No Noble Savages
Figments of Reality
Too Far Ahead
The Rogue Animal
How the World Really Works:
If You Wanted to Sabotage the Elections
Collective Intelligence & Complexity
Ten Things I Wish I'd Learned Earlier
The Problem With Systems
Against Hope (Video)
The Admission of Necessary Ignorance
Several Short Sentences About Jellyfish
A Synopsis of 'Finding the Sweet Spot'
Learning from Indigenous Cultures
The Gift Economy
The Job of the Media
The Wal-Mart Dilemma
The Illusion of the Separate Self:
Did Early Humans Have Selves?
Nothing On Offer Here
Even Simpler and More Hopeless Than That
What Happens in Vagus
We Have No Choice
Never Comfortable in the Skin of Self
Letting Go of the Story of Me
All There Is, Is This
A Theory of No Mind
The Ever-Stranger (Poem)
The Fortune Teller (Short Story)
Non-Duality Dude (Play)
Your Self: An Owner's Manual (Satire)
All the Things I Thought I Knew (Short Story)
On the Shoulders of Giants (Short Story)
Calling the Cage Freedom (Short Story)
Only This (Poem)
The Other Extinction (Short Story)
Disruption (Short Story)
A Thought-Less Experiment (Poem)
Speaking Grosbeak (Short Story)
The Only Way There (Short Story)
The Wild Man (Short Story)
Flywheel (Short Story)
The Opposite of Presence (Satire)
How to Make Love Last (Poem)
The Horses' Bodies (Poem)
Distracted (Short Story)
Worse, Still (Poem)
A Conversation (Short Story)
Farewell to Albion (Poem)
My Other Sites
This work is licensed under a
Creative Commons License.