In this interview with <i>CFA Institute Magazine</i>, Andrew Lo explains the adaptive markets hypothesis, its application in building portfolio tools and financial regulation, and why a decision-making process needs to consider the human element.
• “The main idea behind the adaptive markets hypothesis is that financial markets are governed more by the laws of biology than the laws of physics.”
• “Looking at financial markets as an ecosystem allows us to understand the relation between investment performance and the interactions of various types of investors.”
• “There’s a logic to crises, and we can understand that logic. It’s not necessarily mathematically precise, but it is biologically precise.”
Introduction
How should we view markets? Are they efficient, irrational, or slightly biased? A recent, more integrative theory is the adaptive markets hypothesis (AMH), proposed by Andrew W. Lo, the Charles E. and Susan T. Harris Professor at the MIT Sloan School of Management and director of the MIT Laboratory for Financial Engineering.
His new book is Adaptive Markets: Financial Evolution at the Speed of Thought. In this interview with CFA Institute Magazine, Lo explains the inspiration behind AMH theory, its application in building portfolio tools and financial regulation, and why a decision-making process needs to consider the human element.
The main idea behind the adaptive markets hypothesis is that financial markets are governed more by the laws of biology than the laws of physics. There are five basic tenets of adaptive markets: (1) People act in their own self-interest; (2) people make mistakes; (3) from those mistakes, they learn, adapt, and innovate; (4) as they experiment and fail or succeed, the process of natural selection operates on individuals, institutions, and markets just as it operates on bacteria, sea slugs, and chimpanzees; and (5) this evolutionary process is what determines financial market dynamics.
The AMH applies the framework of evolutionary biology to specific financial contexts. If you follow that perspective to its logical conclusions for any given issue in finance, you’ll get answers that are quite different than what you’d get from either an efficient markets hypothesis (EMH) or [a] behavioral finance perspective.
Here’s an example: How should you determine your asset allocation between stocks and bonds? The EMH says that prices fully reflect all available information, so there’s no use trying to pick winners or losers or timing the market. You should just consider your own risk preferences, your age, your income, and the kind of retirement you’d like to have and then formulate your asset allocation to stocks and bonds to maximize your chances of achieving these goals.
The AMH starts with the observation that there’s no guaranteed return on equities or bonds. Their performance depends on particular market conditions, and those conditions evolve over time. In other words, there are periods where equities will do well, and there are periods where equities won’t do well. So if your goal is to retire with a particular level of wealth, you need to manage your asset allocation dynamically. When equity markets have a higher expected return, you’ll want to tilt more towards equity markets; when equity markets have a lower expected return, you’ll tilt more towards bonds.
How can you tell what expected returns will look like? By monitoring the entire financial ecosystem—the number of individuals and institutions that are investing in equities looking to pull money out and put money into bonds. Looking at financial markets as an ecosystem allows us to understand the relation between investment performance and the interactions of various types of investors. You may not be able to time the markets day to day, but you can certainly see trends over longer holding periods.
We do have some tools, but they’re not ideal because we haven’t collected all the necessary data to use them. For example, over the course of the past 10 years, there have been tremendous amounts of assets flowing into passive index funds and [exchange-traded funds]. Not surprisingly, you’ve seen those passive products earning positive expected returns. That kind of trend tells you that equity markets are going to continue to do reasonably well—until we hit some kind of market disruption. What if we could measure such disruption as it starts developing? By measuring the market interactions of investors and traders at higher frequencies and on a micro scale, we can develop much better projections of what’s going to happen.
But we’re not very good at this yet because we’re not actually looking at financial markets as the kind of system that I just described. We’re not collecting the right kinds of data. We’re measuring prices and other economic fundamentals, which may not be the only factors or even the most important factors that are driving markets.
Imagine if we were ecologists trying to study the ecosystem of the Amazon rainforest. How would we begin? The approach to studying financial ecosystems is much the same.
I would start by tracking the different species of financial market participants. When I say “species,” I mean it much in the same way that a biologist does. A species is a collection of animals that share certain traits and behave in a similar manner. One example is pension funds, which seem to behave in a similar manner due to commonalities in their legal and financial functions. Hedge funds also behave in a similar manner, even though they may differ in their investment styles. So the first thing I would do is to identify and catalog the different financial species—pension funds, hedge funds, mutual funds, banks, broker/dealers, insurance companies, and so on—and take an inventory of the size, growth rates, and other characteristics of each.
Once we've catalogued and measured the different species, we need to understand how they behave. How often do pension funds make investment decisions? How frequently do they revise them? What are their risk tolerances? What are their financial objectives? Are there certain assets that they can invest in [and] others that are forbidden?
If we did this, we’d discover that pension funds can invest in most publicly traded securities, but there are certain constraints that they have to satisfy. They have fiduciary obligations, so they can’t invest in below-investment-grade debt. Their liquidity needs will prevent them from investing in a large fraction of their holdings in illiquid hedge funds and so on and so forth.
By studying each of these financial market species, we can get a sense of how they’re likely to behave in different market circumstances. If we then aggregate across all these species, we begin to get a clear picture of how financial markets are trending and how they’re likely to respond to market shocks.
Most of it is available, here and there. But no one central repository collects and maintains all of the data. That’s really the challenge. In fact, in some cases the data aren’t even being saved. They’re being generated and kept for a short time and then eliminated simply to save on storage costs.
For example, some financial institutions only keep records on certain business transactions for five years. So if eight years ago, they conducted business with a certain counterparty, some potentially valuable information regarding the behavior of that counterparty during negotiations or after a deal was struck would be lost. If we were able to sift through those records systematically, we might be able to come up with some very valuable insights and algorithms for improving our business processes.
Absolutely. When you have to articulate an idea in non-technical language, it forces you to understand it much more deeply than ever before. While I obviously had a pretty clear idea of what I wanted to say, the process of writing it down was tremendously helpful in crystallizing connections between the AMH and other disciplines. I also gained many insights into the application of the AMH to many contexts that I hadn’t considered before, several of which go far beyond finance.
One of the things I discovered was the fact that human intelligence works very much the way internet search engines work. This idea—which comes from the neuroscience and artificial intelligence (AI) literature—has surprisingly broad implications, not just for finance but for life in general.
In the 1970s and 1980s, when the field of AI was just getting off the ground, the big idea was the “expert system,” a piece of software that would mimic human intelligence. Whether it was doing high-school algebra, steering a guided missile to its target, or making a robotic arm play ping pong, these expert systems were complex pieces of software that implemented sophisticated mathematical algorithms to anticipate and provide optimal responses to every possible contingency.
Expert systems made very little use of data, because in those days storage was actually quite expensive; we didn’t have anything like today’s “big data.” The software was highly complex, but the use of data was relatively limited. Today’s expert systems have completely reversed this trend. The algorithms that we use today are by comparison relatively simple, and the amount of data that we process is extraordinarily large.
After studying AI and attempting to model various types of financial decision making algorithmically, I realized that humans make decisions very much the same way that modern search engines do. We have vast stores of data—the experiences we’ve encountered in our lives—and we use very simple algorithms to make predictions and decide on actions. I recollect what happened in my past circumstances, and based upon that history of evidence, I’m going to extrapolate the likely outcome of the current situation and choose the best course of action.
This is how we adapt to various circumstances. It allows us to make very quick decisions. If I tried to analyze every situation and optimize every possible outcome the way that the rational optimizing Homo economicus might, by the time we’re done optimizing, we’d have been eaten by a lion.
In fact, we don’t do that. What we do is rely on big data to make predictions. A large portion of our brains is devoted to memory, and we draw upon our past experiences to make a very, very quick extrapolation—in many cases without any information or detailed analytical deliberation. This is what I mean by the subtitle “Financial evolution at the speed of thought”: Instead of evolving one generation at a time, a single human can go through many generations of ideas and pick the one that seems best, based on her past experiences.
The problem is this: While this mechanism works for keeping us alive, it’s not always the ideal decision-making mechanism for determining our asset allocation. Therein lies the challenge in understanding the limitations of human cognition and developing systems to improve that process.
Focusing on human behavior allowed me to bring it all together. I realized there are many different disciplines that have one thing in common—us, Homo sapiens. The common denominator among anthropology, sociology, economics, psychology, and biology is human behavior. When I realized that financial decision making was simply one aspect of human decision making, I [started] thinking more broadly about how people make decisions and how we model them, both analytically and biologically.
We can improve by first acknowledging that in many cases we make decisions emotionally, not rationally. We need to take into account the emotional reactions that ultimately drive our behavior during certain circumstances. Any financial product that we design—or any investment plan that we decide to implement—needs to take into account the human element.
If an individual realizes she’s likely to “freak out” (that’s a technical term) if she loses more than 10% of her portfolio, she needs to incorporate that into her planning. She needs to consider what kind of assets she’ll hold—that either won’t lose more than 10% over a given period or if they do, she’ll have a contingency plan to deal with the freak-out factor. That plan may be as simple as “When I freak out, I’m going to convert my investments into cash,” but if so, an even more important part of that plan is “After a fixed period of time, I’m going to put my cash back into these assets.”
We need to create investment plans that adapt to market conditions and also take into account our own personal frailties and emotions. Instead, what people do now is they’re told by their financial advisers to buy and hold for the long run and criticized as “short-term investors” or “hot money” when they freak out. That might be appropriate for an automaton, but that’s not helpful advice for a human, because we’re not going to be able to take that advice. When the stock market drops by 25%, we will react one way or the other; that’s the reality of human nature.
Both. Personal and professional investors are both Homo sapiens. Now, professional investors have many more tools at their disposal to deal with some of these challenges, but they still face challenges of their own. Human challenges are just as difficult to deal with for institutional investors as they are for individual investors. It just takes a different form.
I don’t use the word successor because it implies a kind of a critique of both. I actually think that the AMH reconciles and integrates the EMH with behavioral finance. Behavioral anomalies and efficient markets are opposite sides of the same coin: They reflect the dual nature of human behavior. The fact is sometimes we’re rational and sometimes we’re emotional. Usually we’re a bit of both.
The AMH reconciles efficient markets with behavioral finance in an internally consistent and intellectually satisfying way, creating a more holistic view of markets. So maybe in that way, it’s a successor. But it’s a successor that takes the two theories and creates a more complete perspective; it doesn’t say these theories are wrong. They’re not wrong. They’re just incomplete. They don’t apply all the time. The AMH shows how they can happily and productively co-exist when you look at human behavior from a biological perspective.
The responses have differed across different audiences. Industry has responded favorably, mainly because anybody who’s been involved in business understands that adaptation is the key to survival. They see evolution happening before their very eyes, day to day and year after year.
When they learn about the AMH, everybody who’s ever traded for a living—or has run a hedge fund—immediately responds, “Yeah, exactly. That’s exactly what happens.” The theory seems much better able to predict the outcomes of various kinds of market circumstances and environments than either behavioral finance or efficient markets.
However, the academic side is much more skeptical—mainly because the theory hasn’t been presented in a purely mathematical form and finance academics tend to be highly quantitative. So the CAPM, the Black–Scholes/Merton option-pricing model, and the EMH all have formal mathematical expression, whereas the theory of evolution wasn’t mathematically precise when it was first proposed by Darwin (I don’t think there was a single equation in On the Origin of Species).
Now, you can certainly quantify evolution, as many evolutionary biologists and ecologists have done since Darwin, but the statement of the theory is actually deceptively simple and intuitive: The forces of competition, innovation, and natural selection dictate the dynamics of the population. Academics have been less ready to adopt the AMH simply because it’s so early that we haven’t formulated a lot of the mathematical implications.
Instead of telling investors they shouldn’t freak out—which is basically fighting their own hard-wired instincts—why not build portfolio tools that will help them navigate these very, very difficult and challenging periods? Let’s help them deal with the freak-out factor more productively.
These tools include adaptive risk management protocols that measure and manage risk much more dynamically, scaling portfolios down during high-volatility periods and scaling them back up when volatility spikes subside. They also include dynamic factor models for measuring common exposures among investors’ holdings, where the factor loadings are time varying and capture shifts in the relative importance of factors over time. Finally, these tools also include interactive software platforms for measuring the preferences of investors on a regular basis, monitoring changes in their goals, desires, and constraints as their lives unfold and they change.
Regulators are human as well; this means they’re also subject to the influences of market conditions. When markets are going strong and everything is looking stable, it’s very hard to get regulators to “take the punchbowl away.” There’s no easy way to motivate this kind of self-correcting behavior because when things are going so well, who wants to spoil the party?
But it’s exactly during those times when everybody is making money and we haven’t experienced any bank failures or big losses that we do need to consider taking the punchbowl away. So if we understand human nature and incorporate it into the regulatory process, we might consider introducing adaptive regulations—that is, policies that adapt to changing market conditions.
One example is countercyclical capital buffers. When times are good, we increase capital requirements, and after things have blown up and we’re sifting through the wreckage, that’s the time to lower capital requirements. So instead of fixing capital ratios at 2% or 10% or whatever they are now, we ought to adjust capital ratios as a function of market conditions so as to produce a stable probability of loss.
Exactly. It would amount to an automatic stabilizer that would help to control these kinds of risks, much like how the body regulates our temperature by causing us to shiver in the winter and perspire during the summer.
My book on the AMH doesn’t have a single equation, and I did that quite deliberately. (And it wasn’t easy!) I did it because I wanted to reach a broader audience—particularly people who don’t necessarily have the mathematical background that financial quants do. You don’t need mathematical formalism to be able to use and benefit from the AMH framework. If you simply recognize the fact that markets adapt and investors adapt to changing market conditions, you’ll be able to start thinking more flexibly and intelligently about your portfolio.
For example, the starting point for applying the AMH to one’s portfolio is to first recognize that there’s a logic to market behavior, and it’s more biology than physics. Rather than using simple rules of thumb or complex mathematical equations, we can develop a more nuanced view of how markets change, how risks can change, and what we ought to do in response to those changes.
Financial crises do happen from time to time, because when we all start becoming irrationally exuberant at the same time or we all start freaking out at the same time, that’s going to create market bubbles and crashes. There’s a logic to crises, and we can understand that logic. It’s not necessarily mathematically precise, but it is biologically precise. The AMH tells us we have to start learning more about human biology and about ourselves.
These are still early days for the AMH; it’s not, by any means, a finished product. When Darwin published On the Origin of Species, that wasn’t a complete theory either, and I’m no Charles Darwin! It was just the beginning. It provided us with a roadmap for the different theoretical developments to come and the various applications of evolution.
My hope is that this book will introduce the field to my colleagues so they can start developing their own versions, their own models and empirical investigations. The book lays out an ambitious research program for putting these ideas into practice as well as doing more research to flesh out the details, and only time will tell whether the AMH survives.