Machine learning research began over 50 years ago and evolved steadily until the emergence of modern artificial intelligence. Techniques such as principal component analysis, support vector machines, random forests, and even artificial neural networks are all considered traditional ML approaches. Modern AI builds upon these traditional ML methods and leverages increased computational power, enabling it to demonstrate remarkable capabilities across a wide range of applications, from route navigation and protein design to, most notably, the development of advanced conversational models like ChatGPT.
Can AI 'count the cards'?
Investing is a forecasting business. All investors need to forecast future returns to generate alpha. The stock market is considered random and unpredictable, making forecasting accuracy low due to noise. Can modern AI forecast returns better than humans? We make the analogy that blackjack is like the stock market. The only way to win at blackjack is to "count the cards" because the house has an edge. The market can be thought of as the world’s largest blackjack game, and modern AI can help hedge funds "count the cards" to achieve outperforming returns. Traditional ML often identifies associations in data, which can mistakenly imply causal relationships. For example, we might observe a 100% association between a wet lawn and rainfall, but traditional ML could erroneously suggest that the wet lawn causes the rain — a clear misinterpretation. Such a model might build confidence in this faulty assumption over successive rainy days. However, on a sunny day when a gardener waters the lawn, traditional ML could wrongly predict rain with high confidence. If investments were made based on that prediction, significant losses could follow —highlighting the limitations of traditional ML in financial decision-making.
While it's clear that a wet lawn does not cause rain, signals in financial markets are much less straightforward. Modern AI, utilizing advanced nonlinear models and a broader range of input data, can identify leading indicators and uncover genuine causal relationships. In the previous example, modern AI would understand the phenomena of rainfall and a wet lawn, then determine that it is the rainfall that wets the lawn. This capability can help prevent potential losses and turn them into gains.
Essentially, modern AI discovers the causes behind signals (beyond just association), providing a more reliable basis for identifying investment opportunities. It acts like a noise-canceling system, filtering out misleading correlations and uncovering true causal links. Though even modern AI cannot identify causal relationships with 100% certainty, it offers a slight edge over traditional ML. All that is needed is to "count 1–3 cards" which leads to a 2% edge, similar to a casino's edge. Hedge funds leverage this edge by making a large number of trades, relying on the law of large numbers to amplify their advantage and generate consistent positive returns. Owning the "math" is the edge every hedge fund strives to achieve.
One often-overlooked strength for hedge funds — sizing
Traditional ML often results in a constant bet size or leverage, as these models cannot predict how much the market will move in their favor. However, what differentiates modern AI is its forward-looking approach, constantly searching for leading indicators in the market. This allows the AI to increase its bets when it identifies more alpha opportunities and reduce them when it sees less, just like a card player has to forecast how much to bet on the next hand. Sizing alpha is an under-appreciated component of modern AI and allows an investor to produce a convex/asymmetric payoff — resulting in positive skewness (or convexity), uncommon among hedge funds.
Modern AI is also revolutionizing intraday trading by processing vast amounts of real-time data — including price movements, trading volumes, and news sentiment — to uncover patterns and predictive signals that are beyond humans’ ability to process; it can adjust trading strategies on the fly, dynamically altering the sizing of trades based on predicted price movements and volume shifts. This capability enables more precise entry and exit points, optimizing trade execution throughout the day. By continuously adapting to evolving market conditions, modern AI provides a critical edge in capturing short-term market opportunities.
AI’s adoption is challenging but inevitable
First, a manager needs an edge and a time horizon to minimize risk/variance. AI, like any forecasting technology, will have periods of underperformance until skill emerges. While modern AI is undoubtedly a powerful tool, specialized skills are required to effectively apply it to investment; if modern AI is akin to a magic wand, wielding it skillfully involves understanding and innovating intricate "spells." This is why opinions vary on AI's potential in investment. Some are able to design models that deliver exceptional results; while others may struggle to outperform traditional methods. The ability to design high-performing AI models is the competitive edge that sets leading hedge funds apart.
Modern AI has already impressed the world with its capabilities in generating text (such as ChatGPT and Gemini), images (like DALL-E), and even videos (such as Sora). We believe AI is poised to revolutionize hedge fund strategies by effectively "counting the cards" and driving exceptional performance. As we stand at the forefront of this unfolding revolution, it is clear that the advantage of modern AI today may not yet be fully apparent, but we are confident that its adoption will be inevitable in the decade ahead.
Zhilei Xu and Yangfan Li are quantitative researchers and Renee Yao is the chief investment officer at Neo Ivy Capital. They are all based in New York. This content represents the views of the authors. It was submitted and edited under Pensions & Investments guidelines but is not a product of P&I's editorial team.
