As a scenario strategist and futurist, I start with the principle that no one can reliably and consistently predict the future. The world is simply too complex, has too many interlinked unknowns and volatile variables to be able to be modelled and therefore predicted. This has been the basis of my work for over 30 years. However, with the latest, astonishingly rapid development in GenAI and large language models (LLMs), will this always remain true? Is AI on the ultimate path to be able to "know the unknowable"?
Given the fundamental underlying challenges this entails, perhaps a better and fairer way to reframe this, is as a relative question; is AI better than humans at predicting the future? This is a hot topic of debate, not just amongst business leaders, decision-makers and technologists but also academics. A number of papers have been published in recent months by the academic community, with predictably varying and evolving views.
Two academics who have been prolific in addressing this question, are Philipp Schoenegger and Peter S. Park, of the LSE and MIT respectively. Their initial exploration, captured in an LSE blog entitled, "Who is the better forecaster: humans or generative AI?", was published last November. Their experiment focused on questions that were genuinely unknown at the time of asking, spanning everything from Big Tech developments, to geopolitics, to viral outbreaks and US politics.
Using the Metaculus platform, 843 humans competed against ChatGPT-4 to try and predict these uncertain future events. The results showed ChatGPT not only significantly underperformed their human counterparts but its forecasts were barely better than guessing 50% for each answer. As the blog highlights, "in simple terms, GPT-4 was no Nostradamus". Schoenegger and Park suggest that a key factor in GenAI's underperformance was its inability to keep up with new information and current events, something the human forecasters used extensively in making their predictions.
Last month however, Schoenegger and Park partnered with Philip Tetlock from the University of Pennsylvania (he of the "superforecaster" fame) to publish the latest update of their "Wisdom of the Silicon Crowd" paper. This also explores the capabilities of LLMs to forecast versus their human counterparts, however, this time they pitched 31 binary questions to both 925 human forecasters and a combination 12 LLMs. The conclusion (spoiler alert), was that the LLMs were as good as and in some cases better that the humans.
This outcome shouldn't be surprising, given that the questions were both specific and closed (there was no attempt to predict the future of the global economy or how the geopolitical environment will be 10 years from now) . The forecasts were largely created by projecting forward historical trends. This is something that LLMs excel at. The quality of outcome will of course depend on the quality of data the models were trained on. However, by definition, the application of this approach for decision / policy makers will be limited.
For now, at least, GenAI does not offer a panacea to forecasting or prediction, just a useful aid to existing, albeit limited, capabilities. What about the application of AI in the service of another popular decision-making uncertainty tool; scenario planning? Here I can write from my own experience but also reflect on the thoughts of others, including academic Professor Stefan Michel of IMD and the foresight practitioners Daniel J. Finkenstadt, Tojin T. Eapen, Jake Sotiriadis, and Peter Guinto who published "Use GenAI to Improve Scenario Planning" in HBR in November 2023.
As their cunning "the clue is in the title" HBR article suggests, the team of practicitioners argue that GenAI can be effectively deployed to enhance the efficiency and effectiveness of scenario planning, most notably in the areas of:
Scenario creation: highlighting the most important strategic issues and game-changing trends, before the planning team drafts preliminary scenarios.
Narrative exploration: using GenAI to develop a more comprehensive narrative, making it more memorable, palatable and obtaining buy-in from other team members.
Strategy generation: The planning team can prompt GenAI to propose strategies tailored to address the specific challenges (and opportunities) presented by each scenario.
From my own practice and 25 years of scenario planning experience, I can only partly agree with these conclusions. GenAI can be helpful in expediting and enhancing the research phase but much of the value from scenario planning is derived from human discussion, challenge and insight that comes from the ebb and flow of a "diverge and converge" dialogue. If too much of this is automated / outsourced to AI, what you may gain in efficiency and completeness, you will lose in relevance and meaning. Scenario planning is not an academic exercise in creating perfect narratives but a visceral, emotional experience, designed to challenge pre-conceived ideas and open the mind of decision-makers to new possibilities.
I find it even easier to disagree with Professor Michel. In his article "Why the future of AI marks the end of scenario planning", he argues scenario planning is no longer relevant to today's world. He cites Shell's use of scenarios, starting over 50 years ago, and their focus on single variable (oil price) as an indication that scenario planning cannot handle the complexity nor the interaction of multiple uncertainties. This in my view, shows a complete lack of understanding of Shell's work and how scenario planning, correctly executed, actually works. It is the very ability to understand the causal relationship between multiple variables and uncertainties that made Shell so successful in the 1970's and 1980's and allows scenario planning to remain such a powerful tool today.
Applied judiciously, AI can enhance the power of foresight generally and scenario planning specifically. But as with the application of Artificial Intelligence more broadly, that success will depend on skilled and appropriate application, blending with and enhancing human capabilities.
留言