There is a lot of talk these days in financial circles about artificial intelligence and machine learning. Most of the discussion has focused on how powerful these approaches are and how they may transform the industry in the next five years.
We have seen that machine learning offers potentially powerful solutions to problems in many domains across the economy. While machine learning might transform the investment management industry, the path by which we get there is uncertain. As is often the case with potentially disruptive technologies, the obstacles seem as much cultural as technical. In a word, the issue is transparency.
Investors require transparency in how and why investment decisions are made. To the extent that machine learning is used to make investment decisions (rather than as an ancillary element of the investment process), transparency might be drastically reduced. Here is the rub: To obtain potentially better investment results driven by a full-blown machine-learning model, investors might need to accept less transparency regarding why a model did what it did. That cultural shift will take time.
The primary branch of ML is call supervised machine learning, and is used to predict something — e.g., whether an email is spam or not. In this case, the ML algorithm learns the key characteristics of spam (words used, formatting, etc.) by looking at perhaps hundreds of thousands or even millions of emails that are labeled as either spam or not spam. It can then predict fairly accurately which emails to filter out.
In the world of investing, an oversimplified example of the use of supervised ML might be the case of predicting whether stocks will go up or down the following day. A data scientist would need to make many critical model construction decisions, including the exact algorithm to use in the ML framework (something simple, such as logistic regression, or something more involved, such as a random forest), and the appropriate data set. The data would include many columns of time series market data (such as a variety of price momentum and valuation measures) where each row of data (or time period) is labeled something like, "Stocks went up next day" or "Stocks went down next day." The model is then trained on this historical data to determine characteristics of periods associated with future rising or falling stock prices. The model (the machine) is continually trying to learn — to figure out the best way to weight and process the many data items in order to best predict the direction of the market.
The issue is that, when a model is implemented and predicts that "stocks will go up tomorrow," it might not be clear why the model is making this particular recommendation. How this huge mass of data worked its way through very complex algorithms to come up with the "buy" answer is not always obvious — not even to the data scientist who created the model. By contrast, with a more traditional quantitative approach, the reason a particular model has put on a particular trade is often more clear. In a trend-following strategy, for example, if an investor asks the manager why the strategy was significantly long equities last February, it is easy to point to the strong positive trend in equities over the prior 12 months to explain the long positioning. Similar intuitions can be made quite easily with regard to carry, curve, value and other traditional risk premium strategies captured through quantitative approaches. But again, this will not necessarily be the case with a large-scale machine learning approach.
The real issue comes when there is sustained underperformance. When the reason a model put on money-losing positions is not obvious (i.e., why it was long bunds, short Japanese yen, or long the FTSE 100 index), investors begin to raise larger questions, and perhaps rightly so. Is the model broken? How do you know it's not broken? Did you overfit the model?
It is perhaps at this point that ML fails — not because it doesn't work, but because investors become more reticent with their investment dollars in the face of this lack of transparency.
The question as to whether there is something inherently wrong with the model is an important one. Given the power of ML methods, it is indeed possible that a given model has been overfit. Overfitting occurs when the model incorporates random and noisy characteristics of the training data and treats them as though they actually provide good, predictive information. Generally, the larger the number of data fields that go into the model and the more detailed the algorithm, the higher the probability the algorithm will identify a "pattern" that is really just random noise.
Cognizant of these modeling concerns, researchers have created a host of methods for limiting the propensity of these models to overfit data, including being very thoughtful about which data to start with. Similarly, there is a greater awareness of the need for data scientists who have strong financial backgrounds and can work collaboratively with those who are more pure data scientists. All the same, when a model underperforms, potential overfitting can be a legitimate concern, and one that can be difficult to ascertain, even using state-of-the-art techniques.
Having focused on the most intensive use of machine learning in investment management above, it is worth noting that ML also can be used indirectly. A quantitative manager might use it in order to better select model parameters. If, for example, a model uses an estimate of a stock's volatility, one might use ML for ferreting out the best combination of a multitude of methods for calculating volatility. This can be a very powerful use of machine learning in the current environment, as it is one that generally does not compromise the transparency of the investment process.
ML can also be used within a discretionary investment process. A value manager, for example, might use it to screen the universe of stocks for those with the best prospects based on valuation. The data set could include a very large number of traditional and non-traditional valuation measures. Stocks that the model identifies can then be subject to a deeper bottom-up analysis before any final buy or sell determination. Nevertheless, we might run into similar cultural issues in this use of ML. The lack of transparency afforded by these models may produce results that are inconsistent with the discretionary manager's intuition and thereby induce questions as to whether the stock screening process is broken.
In short, the main advantage and the main drawback of machine learning are one and the same: these algorithms care less about the intuition behind the model and more about the power to predict. This means that, if done well, ML may produce better results than more traditional approaches, but explanations for the outperformance or underperformance of a particular model are more difficult to produce.
As machine learning is used more frequently in investment management, investors may increasingly need to rely on due diligence regarding the model construction and data science process, and they might also need to accept a less than transparent explanation and perhaps unintuitive reason for specific periods of outperformance or underperformance. This is effectively the cultural shift that must take place with regard to investor acceptance of these strategies before machine learning can take a more central role in investment management.