Why Do Financial Forecasts Persist Despite Their Failures?
Because they are stories. And we need stories to cope with uncertainty.
As touched upon in the earlier post about the Efficient Market Hypothesis, which—to remind us—states that markets are informationally efficient, meaning stock prices quickly reflect all publicly available information. Hence, there is no room for financial analysts to forecast stock price developments in the long run. This is the stance of neoclassical economic thought.
You might have heard of Burton Malkiel and his book A Random Walk Down Wall Street. He is in the same camp as Eugene Fama, the neoclassical one. In the book, Malkiel provocatively writes the following to illustrate the random-walk hypothesis, taken to its theoretical extreme:
“A blindfolded monkey throwing darts at a newspaper’s financial pages could select a portfolio that would do just as well as one carefully selected by experts.” (Malkiel, 1973)
In other words, stock-picking, active fund management, and forecasting perform no better than pure chance. The only way, according to neoclassical economics, to predict stock price developments would be if a market actor had access to private information that is not available to any other market participant. And if that is the case, one would assume that a financial analyst would not be able to obtain such information on a continuous basis in the long run. Hence, predicting market developments or individual stock movements is, by these assumptions, not possible.
And really, there is not even a need to bring up academic research articles that have tested this empirically. Because it is quite intuitive—or at least I would say so—that one cannot predict the future. Most of us who have followed economic or financial news for some time can probably recall several instances where “they” got it wrong.
But just for the sake of it, let us mention a few.
Stefan Leins, in his book Stories of Capitalism, mentions Working (1934), Kendall (1953), and Osborne (1959). They all concluded that, on average, financial analysts are largely unable to outperform the market. And, as I mentioned in an earlier post—Acting Alpha—only 23% of active funds outperformed their passive counterparts over the period from 2010 to 2020. There are, of course, many other studies to be found that reach the same conclusion.
So market forecasting is problematic—both from an economic theory perspective and from an empirically tested perspective.
Over time, criticism against the neoclassical school of thought grew into the creation of the field of Behavioral Economics (think Kahneman and Tversky). This field examines the heuristics and cognitive biases that prevent economic actors from behaving in a fully rational way. I touched on some of the most common mental shortcuts that get in the way of making rational, or “good,” decisions in the prior post What Makes a Good Decision?. From this foundation, the field (or subfield) of Behavioral Finance developed.
Behavioral Finance, then, looks specifically at economic actors' behavior in a financial market context. The same biases—overconfidence, overoptimism, confirmation bias, cognitive dissonance, to mention a few—can of course also be observed when studying financial analysts and their attempts to forecast market developments.
But being aware that these biases exist does not help legitimize the market practice of forecasting. Simply because, in a biased market—where everyone is biased—it is not really helpful to be the only unbiased one. Market prices will reflect the biases of the crowd, not the objectivity of the individual. Right?
So the idea of knowing the biases, adjusting for them, and then forecasting some “unbiased” future price becomes a bit illogical. You are starting from the assumption that current prices are distorted by psychology and noise—but then expect that future prices will somehow align with your clean, rational estimate? Why would that be the case?
It is a bit of a conundrum. Financial analysts will often say: current prices are driven by manipulation, herding, fear, and psychology. But they can still forecast future prices based only on fact-based estimations?
So again, the field of Behavior Finance does also not, unfortunately, justify the existence of financial analysis. Perhaps, there are professions in which expertise is not possible? Stock picking might be a good example, where experts are no better than a dice-throwing monkey (not my words, but Kahneman’s, cited in Leins, 2018).
This brings us to another alternative to neoclassical economics—new institutional economics. At its core, this approach argues that real-world markets are not made up of perfectly rational individuals with full access to cost-free information. Instead, it emphasizes that our actions are shaped by institutions—by which economists mean the rules, norms, and social conventions that structure how people behave and interact. Institutions emerge as a response to limited access to information and the inherent uncertainty of future events.
New institutional economics also brings attention to something often ignored in traditional theory: that gathering and interpreting information costs time and money, known as “transaction costs.” These costs mean that access to market information is not equal for everyone, and that outcomes are shaped by power dynamics as much as by supply and demand. This also brushes up against the information paradox, famously articulated by Grossman and Stiglitz—though from a slightly different angle. I explain that paradox in more detail in the earlier post The Efficient Market Hypothesis Eats Itself.
So, could this perspective justify the practice of market forecasting? Not really. While it helps explain why financial analysts exist—as intermediaries who reduce information costs for others—it still does not offer a way for them to predict the future. Financial analysis itself rests on a set of conventions, such as the belief that future values can be estimated based on company earnings, sales, and other economic indicators. From an institutional perspective, these conventions function as stabilizing tools that enhance the analyst’s position. But even with more information, analysts still face the same fundamental problem: they do not know which information will matter, or how markets will react. In the end, uncertainty remains a constant feature of financial markets. And so, like neoclassical and behavioral economics, institutional economics also fails to offer a solid theoretical foundation for forecasting.
So if forecasting is not grounded in theory, and consistently fails in practice, why does it persist? Forecasts, it turns out, are not just about predicting the future. They are about making sense of uncertainty. They provide structure, language, and direction in an environment where outcomes are unknown and unknowable. Forecasts are not merely calculations—they are stories. And stories help stabilize expectations, build confidence, and coordinate action.
From this perspective, financial analysts do not simply predict—they narrate. Their role is less about uncovering what the future will be, and more about constructing a version of it that others can act upon. Their value lies less in theoretical or empirical precision, and more in how well they organize information into a compelling and actionable story. In a world that demands decisions under uncertainty, storytelling becomes a way to cope, to communicate, and to construct a shared sense of what might come next—even if that imagined future never arrives.
Thank you for reading this far. I hope you found the post interesting. Feel free to share it or follow along on the socials linked below. Till next time, my friend.
Instagram; Podcast on Spotify and Apple Podcast; YouTube; Website
References
Leins, S. (2018) Stories of Capitalism: Inside the Role of Financial Analysts. Chicago: University of Chicago Press.
Working, H. (1934) ‘A random-difference series for use in the analysis of time series’, Journal of the American Statistical Association, 29(185), pp. 11–24. Cited in Leins (2018).
Kendall, M.G. (1953) ‘The analysis of economic time series’, Journal of the Royal Statistical Society. Series A (General), 116(1), pp. 11–34. Cited in Leins (2018).
Osborne, M.F.M. (1959) ‘Brownian motion in the stock market’, Operations Research, 7(2), pp. 145–173. Cited in Leins (2018).