In last week’s New Yorker, Malcolm Gladwell describes the neural network systems that have been designed, by two independent groups, to address two complex commercial problems: predicting the success of a new popular song, and predicting the success of a new movie. The article is called The Formula.
Although neural networks (collectors of massive amounts of data that then seek ‘meaningful’ patterns in that data that can be used to infer causality or at least correlation) have been around for years, most students of complex adaptive systems believe that complex problems (like global poverty, global warming, or lack of innovation in big business) can never be ‘solved’ because there are simply too many variables (perhaps an infinite number) to allow any kind of exhaustive correlation or useful predictive models to be built. The closest we can hope to get, most complexity theorists would tell us, is interventions that would have a positive impact. The complex problems that Gladwell’s subjects have addressed do have a lot of variables — the factors that determine popular opinion on a song or movie are myriad and often seemingly unfathomable — but the number of variables is finite. What’s more, the problem-solvers believed that, in the matters they were concerned with, a reasonably small number of variables (certainly a number manageable by today’s computer systems) were disproportionately responsible for a song or movie’s success or failure. And because we’re talking about a product, something that can be ‘put in a box’, the challenge of identifying these variables is less problematic than the challenge of, say, identifying all the variables needed to predict when and where the next major hurricane or pandemic disease will hit. One could say there are complex problems and there are complex problems, and some are more complex than others. Despite this, the designers of the ‘expert systems’ that are now being used to predict song and movie success — with remarkable precision — faced ridicule and rejection from sponsors and customers because of the prevailing belief that, when it comes to predicting these things, as movie mogul William Goldman put it, “nobody knows anything”. The key to neural network analysis, besides the computing power to do a lot of iterations with a lot of variables to look for patterns, is patience — when the first few hundred variables don’t pan out, you set them aside and look at a few hundred new ones, and keep adding until some pattern finally emerges. The variables used in the song success predictor involve the song’s structural components: melody, harmony, rhythm, beat, tempo, octave, pitch, chord progression, cadence, sonic brilliance, frequency and so on. The predictor, called Platinum Blue, has analyzed a huge number of songs of many different genres and found the same patterns that resonate with us in popular music can be found in classical and folk music from all different eras. Popular songs, it finds, fall into clusters — the variables of those songs, taken in aggregate, exhibit similar sets of patterns. The details, of course, are confidential. If a song falls outside these clusters, the designers of Platinum Blue can tell you which variables need to be changed, and roughly in what way, to get it back into the success strike zone, but beyond that it is up to the artist — how to change the composition is beyond the capacity of the predictor. But once the artist has made the changes, the predictor can tell whether the result is within one of the clusters, and how much the changes mean to the expected revenues from the song. The predictor’s greatest claim to fame is its prediction of enormous success for the CD “Come Away With Me” by then-little-known Norah Jones. What is most remarkable about Platinum Blue is its wonderful vindication of the talents of writers and composers: It predicts the success of the song regardless of its lyrics or performer, based only on its compositional qualities, its mathematical structure. A different group, Epagogix, has found the same thing applies to Hollywood film releases — the ultimate popularity and success of a film depends on the qualities of its composition, not on the ‘stars’ attached to it who get paid all the money. Plot lines, the ingredients of particular scenes, characters, and settings matter. That’s not to say that stars don’t get people into theatres, at least until word of mouth begins to prevail over advance billing. But Epagogix can tell you that if the stars want $40 million and the rest of the movie costs another $20 million, whether the $60 million investment will be a winning or a losing one. Epagogix analyzed the 2005 movie The Interpreter, which went through several massive and well-documented changes before it was finally released. It concluded that the changes had made a probable $33 million film into a $69 million film (within $4 million of the actual revenues), and then pointed out the ways in which it could, with relatively few changes and for very little additional investment, have become a $150 million-plus blockbuster. Epagogix analyzed another (unidentified) film and predicted would make $47 million as scripted and $72 million of three minor changes were made — the changes were not made and the film grossed $50 million. If you’ve seen The Interpreter (I haven’t) you can probably assess better than I can whether the changes they proposed would have made it a better film or not — and the authors of both Platinum Blue and Epigogix both assert that it’s not enough to have the right ingredients — the product has to work artistically as a whole as well, and no neural network system will tell you how to do that. What the neural network systems can’t do is identify and parse the variables to consider. That takes a combination of a knowledge of the art form, and how it’s appreciated by audiences, and also a great deal of imagination, to keep honing and refining and trying new variables until the ones that really matter start to emerge from the pattern matching. This is in some ways as much an art as the writing and making of a song or film itself. The Wisdom of Crowds is another type of neural network system, the difference being that the ‘crowd’ doesn’t explicitly identify the relevant variables it considers in rendering its judgement. The point is that the group using that wisdom is concerned with coming up with the ‘right’ answer (prediction, choice, critical information etc.) and not overly concerned with how the crowd came up with it. One WoC application, the Iowa Electronic Markets, predicted that Bush would win the 2004 election while most of the other expert predictors were saying Kerry would win. How did they know? Among the crowd they had knowledge of thousands of subtle, unidentified variables that individual experts could never know. We would be right to be skeptical of what we’re told about Platinum Blue and Epigogix. Despite what we’re told about their success, there is enough secretiveness about both projects that it is possible that both products are hoaxes, or at least much less accurate than their authors allege. But suppose they are the real deal? What other complex problems could similar neural network systems be applied to ‘solve’? As Gladwell points out, you could apply them to win at the racetrack, and perhaps even in the stock market. You could probably use them to devise the best possible new product design process. But could you use them to analyze all the education programs in the world and come up with the ideal curriculum for self-sufficiency and critical skills learning? Or all the health systems in the world to design a hybrid system that offered the best of all worlds? Or how about using them to develop a revenue-neutral tax system that would actually change behaviours to reduce greenhouse gases sufficient to end the threat of global warming? I’m not so sure, even if neural networks could solve some straightforward complex problems, that they’re up to the challenge of helping us grapple with the ‘wicked’ ones that have defeated us for centuries, and which even threaten our civilization’s demise. Even if it were possible to employ them to such ends, it would take a great deal of passion, patience, commitment and imagination to go along withan astonishingly sophisticated and massive pattern-seeking technology. But as one more tool in the Coping with Complexity toolkit, it sure couldn’t hurt. Image: NASA’s depiction of the neural network of a single plant. |
Navigation
Collapsniks
Albert Bates (US)
Andrew Nikiforuk (CA)
Brutus (US)
Carolyn Baker (US)*
Catherine Ingram (US)
Chris Hedges (US)
Dahr Jamail (US)
Dean Spillane-Walker (US)*
Derrick Jensen (US)
Dougald & Paul (IE/SE)*
Erik Michaels (US)
Gail Tverberg (US)
Guy McPherson (US)
Honest Sorcerer
Janaia & Robin (US)*
Jem Bendell (UK)
Mari Werner
Michael Dowd (US)*
Nate Hagens (US)
Paul Heft (US)*
Post Carbon Inst. (US)
Resilience (US)
Richard Heinberg (US)
Robert Jensen (US)
Roy Scranton (US)
Sam Mitchell (US)
Tim Morgan (UK)
Tim Watkins (UK)
Umair Haque (UK)
William Rees (CA)
XrayMike (AU)
Radical Non-Duality
Tony Parsons
Jim Newman
Tim Cliss
Andreas Müller
Kenneth Madden
Emerson Lim
Nancy Neithercut
Rosemarijn Roes
Frank McCaughey
Clare Cherikoff
Ere Parek, Izzy Cloke, Zabi AmaniEssential Reading
Archive by Category
My Bio, Contact Info, Signature Posts
About the Author (2023)
My Circles
E-mail me
--- My Best 200 Posts, 2003-22 by category, from newest to oldest ---
Collapse Watch:
Hope — On the Balance of Probabilities
The Caste War for the Dregs
Recuperation, Accommodation, Resilience
How Do We Teach the Critical Skills
Collapse Not Apocalypse
Effective Activism
'Making Sense of the World' Reading List
Notes From the Rising Dark
What is Exponential Decay
Collapse: Slowly Then Suddenly
Slouching Towards Bethlehem
Making Sense of Who We Are
What Would Net-Zero Emissions Look Like?
Post Collapse with Michael Dowd (video)
Why Economic Collapse Will Precede Climate Collapse
Being Adaptable: A Reminder List
A Culture of Fear
What Will It Take?
A Future Without Us
Dean Walker Interview (video)
The Mushroom at the End of the World
What Would It Take To Live Sustainably?
The New Political Map (Poster)
Beyond Belief
Complexity and Collapse
Requiem for a Species
Civilization Disease
What a Desolated Earth Looks Like
If We Had a Better Story...
Giving Up on Environmentalism
The Hard Part is Finding People Who Care
Going Vegan
The Dark & Gathering Sameness of the World
The End of Philosophy
A Short History of Progress
The Boiling Frog
Our Culture / Ourselves:
A CoVid-19 Recap
What It Means to be Human
A Culture Built on Wrong Models
Understanding Conservatives
Our Unique Capacity for Hatred
Not Meant to Govern Each Other
The Humanist Trap
Credulous
Amazing What People Get Used To
My Reluctant Misanthropy
The Dawn of Everything
Species Shame
Why Misinformation Doesn't Work
The Lab-Leak Hypothesis
The Right to Die
CoVid-19: Go for Zero
Pollard's Laws
On Caste
The Process of Self-Organization
The Tragic Spread of Misinformation
A Better Way to Work
The Needs of the Moment
Ask Yourself This
What to Believe Now?
Rogue Primate
Conversation & Silence
The Language of Our Eyes
True Story
May I Ask a Question?
Cultural Acedia: When We Can No Longer Care
Useless Advice
Several Short Sentences About Learning
Why I Don't Want to Hear Your Story
A Harvest of Myths
The Qualities of a Great Story
The Trouble With Stories
A Model of Identity & Community
Not Ready to Do What's Needed
A Culture of Dependence
So What's Next
Ten Things to Do When You're Feeling Hopeless
No Use to the World Broken
Living in Another World
Does Language Restrict What We Can Think?
The Value of Conversation Manifesto Nobody Knows Anything
If I Only Had 37 Days
The Only Life We Know
A Long Way Down
No Noble Savages
Figments of Reality
Too Far Ahead
Learning From Nature
The Rogue Animal
How the World Really Works:
Making Sense of Scents
An Age of Wonder
The Truth About Ukraine
Navigating Complexity
The Supply Chain Problem
The Promise of Dialogue
Too Dumb to Take Care of Ourselves
Extinction Capitalism
Homeless
Republicans Slide Into Fascism
All the Things I Was Wrong About
Several Short Sentences About Sharks
How Change Happens
What's the Best Possible Outcome?
The Perpetual Growth Machine
We Make Zero
How Long We've Been Around (graphic)
If You Wanted to Sabotage the Elections
Collective Intelligence & Complexity
Ten Things I Wish I'd Learned Earlier
The Problem With Systems
Against Hope (Video)
The Admission of Necessary Ignorance
Several Short Sentences About Jellyfish
Loren Eiseley, in Verse
A Synopsis of 'Finding the Sweet Spot'
Learning from Indigenous Cultures
The Gift Economy
The Job of the Media
The Wal-Mart Dilemma
The Illusion of the Separate Self, and Free Will:
No Free Will, No Freedom
The Other Side of 'No Me'
This Body Takes Me For a Walk
The Only One Who Really Knew Me
No Free Will — Fightin' Words
The Paradox of the Self
A Radical Non-Duality FAQ
What We Think We Know
Bark Bark Bark Bark Bark Bark Bark
Healing From Ourselves
The Entanglement Hypothesis
Nothing Needs to Happen
Nothing to Say About This
What I Wanted to Believe
A Continuous Reassemblage of Meaning
No Choice But to Misbehave
What's Apparently Happening
A Different Kind of Animal
Happy Now?
This Creature
Did Early Humans Have Selves?
Nothing On Offer Here
Even Simpler and More Hopeless Than That
Glimpses
How Our Bodies Sense the World
Fragments
What Happens in Vagus
We Have No Choice
Never Comfortable in the Skin of Self
Letting Go of the Story of Me
All There Is, Is This
A Theory of No Mind
Creative Works:
Mindful Wanderings (Reflections) (Archive)
A Prayer to No One
Frogs' Hollow (Short Story)
We Do What We Do (Poem)
Negative Assertions (Poem)
Reminder (Short Story)
A Canadian Sorry (Satire)
Under No Illusions (Short Story)
The Ever-Stranger (Poem)
The Fortune Teller (Short Story)
Non-Duality Dude (Play)
Your Self: An Owner's Manual (Satire)
All the Things I Thought I Knew (Short Story)
On the Shoulders of Giants (Short Story)
Improv (Poem)
Calling the Cage Freedom (Short Story)
Rune (Poem)
Only This (Poem)
The Other Extinction (Short Story)
Invisible (Poem)
Disruption (Short Story)
A Thought-Less Experiment (Poem)
Speaking Grosbeak (Short Story)
The Only Way There (Short Story)
The Wild Man (Short Story)
Flywheel (Short Story)
The Opposite of Presence (Satire)
How to Make Love Last (Poem)
The Horses' Bodies (Poem)
Enough (Lament)
Distracted (Short Story)
Worse, Still (Poem)
Conjurer (Satire)
A Conversation (Short Story)
Farewell to Albion (Poem)
My Other Sites
I think I saw this gladwell article reviewed elsewhere and the key missing factor was the “outcome measures” of success. These were said to be very fuzzy indeed. If I find the article I will link it back to you.
Even if it were possible to employ them to such ends, it would take a great deal of passion, patience, commitment and imagination to go along with an astonishingly sophisticated and massive pattern-seeking technology.Dave, we could all try expanding the capacity of ”wet ware” aka the human mind ;-)aka learning authentically instead of through machines and bunkum theories and the Gladwells of this world who parse science and pseudo science and sell it on like candy in a WalMart ;-) andrew
Paul: Thanks. I’d like to see that. Andrew: I think that’s a tad harsh. Gladwell has done a lot to bring some very innovative ideas into public discourse. To anyone who can make money popularizing the concept of Tipping Points in change processes, I say “Good on yer, mate”. As you know I’m a techno-skeptic, but I’ve learned enough about neural nets to respect what they are good for (which is finding correlations to help address somewhat complex, not very complex, problems). But what if we could get all those thousands of PCs futilely looking for SETI, instead working on finding patterns that would help us address global warming or world poverty?