Skip to main content
MENU
Subscribe
  • Subscribe
  • Account
  • LOGIN
  • Topics
    • Alternatives
    • Consultants
    • Coronavirus
    • Courts
    • Defined Contribution
    • ESG
    • ETFs
    • Hedge Funds
    • Industry Voices
    • Investing
    • Money Management
    • Opinion
    • Partner Content
    • Pension Funds
    • Private Equity
    • Real Estate
    • Russia-Ukraine War
    • SECURE Act 2.0
    • Special Reports
    • White Papers
  • Rankings & Awards
    • 1,000 Largest Retirement Plans
    • Top-Performing Managers
    • Largest Money Managers
    • DC Money Managers
    • DC Record Keepers
    • Largest Hedge Fund Managers
    • World's Largest Retirement Funds
    • Best Places to Work in Money Management
    • Excellence & Innovation Awards
    • WPS Innovation Awards
    • Eddy Awards
  • ETFs
    • Latest ETF News
    • Fund Screener
    • Education Center
    • Equities
    • Fixed Income
    • Commodities
    • Actively Managed
    • Alternatives
    • ESG Rated
  • ESG
    • Latest ESG News
    • The Institutional Investor’s Guide to ESG Investing
    • ESG Sustainability - Gaining Momentum
    • Climate Change: The Inescapable Opportunity
    • Impact Investing
    • 2022 ESG Investing Conference
    • ESG Rated ETFs
  • Defined Contribution
    • Latest DC News
    • DC Money Manager Rankings
    • DC Record Keeper Rankings
    • Innovations in DC
    • Trends in DC: Focus on Retirement Income
    • 2022 Defined Contribution East Conference
    • 2022 DC Investment Lineup Conference
  • Searches & Hires
    • Latest Searches & Hires News
    • Searches & Hires Database
    • RFPs
  • Performance Data
    • P&I Research Center
    • Earnings Tracker
    • Endowment Returns Tracker
    • Corporate Pension Contribution Tracker
    • Pension Fund Returns Tracker
    • Pension Risk Transfer Database
    • Future of Investments Research Series
    • Charts & Infographics
    • Polls
  • Careers
  • Events
    • View All Conferences
    • View All Webinars
    • 2022 Retirement Income Conference
    • 2022 Managing Pension Risk & Liabilities
    • 2022 WorldPensionSummit
Breadcrumb
  1. Home
  2. ALTERNATIVES
July 27, 2017 01:00 AM

Glass box vs. black box

Joshua Vogelstein and Jonathan Caplis
  • Tweet
  • Share
  • Share
  • Email
  • More
    Reprints Print

    Sixty-two percent of systematic hedge fund managers are using machine learning techniques within the investment process, a recent Barclays report shows. Given the significant increase to computing power (and reduced cost), along with the wide availability of off-the-shelf statistical computing packages, the barriers to implementing machine learning techniques have been significantly mitigated, while the difficulty in using these techniques successfully remains. Indeed, fund managers that don't fully grasp the upside potential and risk parameters of their strategy can quickly implode, even if the investment thesis is fundamentally sound; the problem becomes further magnified within the expanding complexity of a machine learning framework.

    Machine learning, at its core, is a misnomer. The term suggests one can throw data at a machine, and out pops accurate predictions. In fact, to set machine learning up to solve a problem, a team of humans is required to choose the data sources, the algorithms, the hardware and the parameters. Each of these choices requires a complementary set of skills, backgrounds and intuition, ranging from finance, to statistics, to computer science.

    In our experience, the teams that most successfully utilize machine learning are the teams that treat it like a glass box, rather than a black box. Whereas a black box provides no knowledge of what occurs between input and output, an intelligently constructed glass box offers a level of insight and the ability to test a clear hypothesis. As is always a good policy, if someone can't clearly explain to you what they are doing, they probably don't understand it themselves.

    How can an investor separate the wheat from the chaff?

    When evaluating machine learning strategies, here are some of the questions investors should be asking to help separate black boxes from glass boxes.

    1. What is your investment thesis? Don't let sales people switch the topic by citing sophisticated algorithms in response to this question. If they do, ask the following simple questions: Why do you make money? What is the market inefficiency in which you are capitalizing? If they can't answer these basic questions, or rather the team prefers to advertise the value of their tool and its ability to identify patterns — over their understanding of the fundamental principles underlying statistical science — there is cause for concern.

    2. Are you a data analyst using a machine learning engineer's product or are you a machine learning engineer? Just as an investor scrutinizes the qualifications and experiences of an investment team to determine whether they are appropriately skilled, the same level of scrutiny should apply to the quantitative team. Are they:

    • MBAs with a concentration in statistics;

    • professionals with graduate-level degrees in data science; or

    • Ph.D.s in the applied sciences such as biomedical engineering or applied neuroscience?

    You should look for the latter.

    In order to fully understand the nuances of implementing machine learning techniques, one must have the proper training and experience to effectively use these techniques, where borrowing tools from machine learning packages is insufficient.

    3. How do you handle the trade-off between leveraging more powerful tools and the ability to understand the model? In data science speak, this concept is known as bias-variance trade-off. More parameters introduce greater variance and less bias, increasing the likelihood of overfitting, or describing noise in the data, rather than identifying underlying relationships and predictive patterns. For example, linear regressions come with significant bias (they work well when the relationship is linear), but the variance is small because they cannot estimate any non-linear function such as seasonal variability.

    One must always consider whether the additional variance is too large and worth including. This is a subtle point that was lost on the much of the neural network community for decades, so it is little wonder that it is still absent in large communities applying mathematical tools to forecast financial markets. The more variables and parameters (degrees of freedom), you feed into the model, the more one loses the ability to reason what happens between the input and the output. This is directly related to the next question.

    4. How do you determine when a model is broken before it is too late? A primary difficulty in utilizing machine learning algorithms is the increasing power of those tools for finance. Modern algorithms can find patterns in almost any dataset, much like a novice stargazer can find patterns looking up at the stars. But finding an historical pattern is not the same as being able to forecast a profitable trade, just as seeing a star formation of a lion does not necessarily foretell a day of courage.

    Without significant expertise in the subtleties of how to evaluate these powerful algorithms, being able to detect when and why a model loses its predictive ability and making the necessary changes required becomes almost impossible.

    The most successful financial teams will continue to be those that a have a deep understanding of the kinds of things that can go awry, how to avoid them, and a plan for how to cope with them when they happen, no matter which tools they have at their disposal.

    The increasing ubiquity of powerful computing technologies will likely continue to increase the number of users in the financial community. Ultimately, machine learning is far from a blanket description but something for investors to question especially given the ongoing hype and media attention. As they would in other strategies across their portfolios, investors should look for the best investment teams with deep domain knowledge about the markets they trade and the underlying data.

    Joshua Vogelstein is an assistant professor in biomedical medical engneering at Johns Hopkins University, Baltimore, and Jonathan Caplis is a managing principal at PivotalPath, New York. This article represents the views of the authors. It was submitted and edited under Pensions & Investments guidelines, but is not a product of P&I's editorial team.

    Related Articles
    Machine learning will transform investment management
    AI? It's not sci-fi anymore
    AI key for third wave of hedge fund managers
    Recommended for You
    Partners Group names head of private equity for Japan
    H.I.G. Capital names head of U.S. private equity business development
    mark_carney_feb_2020_1550-main_i.jpg
    Brookfield names leaders of soon-to-be spun off asset management unit

    Reader Poll

    August 10, 2022
    SEE MORE POLLS >
    Sponsored
    White Papers
    Gaining Momentum: Where Next for Trend-Following?
    How Has 2022's Carnage Reshaped Global Stock and Bond Markets?
    Can Sustainable Labeling of Financial Products Prevent Greenwashing?
    Hedge Funds 2.0: Back to the future
    Is there a mid-cap gap in your DC plan?
    Why pursue direct lending in the core middle market?
    View More
    Sponsored Content
    Partner Content
    The Industrialization of ESG Investment
    For institutional investors, ETFs can make meeting liquidity needs easier
    Gold: the most effective commodity investment
    2021 Investment Outlook | Investing Beyond the Pandemic: A Reset for Portfolios
    Ten ways retirement plan professionals add value to plan sponsors
    Gold: an efficient hedge
    View More
    E-MAIL NEWSLETTERS

    Sign up and get the best of News delivered straight to your email inbox, free of charge. Choose your news – we will deliver.

    Subscribe Today
    August 1, 2022 page one

    Get access to the news, research and analysis of events affecting the retirement and institutional money management businesses from a worldwide network of reporters and editors.

    Subscribe
    Connect With Us
    • RSS
    • Twitter
    • Facebook
    • LinkedIn

    Our Mission

    To consistently deliver news, research and analysis to the executives who manage the flow of funds in the institutional investment market.

    About Us

    Main Office
    685 Third Avenue
    Tenth Floor
    New York, NY 10017-4036

    Chicago Office
    130 E. Randolph St.
    Suite 3200
    Chicago, IL 60601

    Contact Us

    Careers at Crain

    About Pensions & Investments

     

    Advertising
    • Media Kit
    • P&I Content Solutions
    • P&I Careers | Post a Job
    • Reprints & Permissions
    Resources
    • Subscribe
    • Newsletters
    • FAQ
    • P&I Research Center
    • Site map
    • Staff Directory
    Legal
    • Privacy Policy
    • Terms and Conditions
    • Privacy Request
    Pensions & Investments
    Copyright © 1996-2022. Crain Communications, Inc. All Rights Reserved.
    • Topics
      • Alternatives
      • Consultants
      • Coronavirus
      • Courts
      • Defined Contribution
      • ESG
      • ETFs
      • Hedge Funds
      • Industry Voices
      • Investing
      • Money Management
      • Opinion
      • Partner Content
      • Pension Funds
      • Private Equity
      • Real Estate
      • Russia-Ukraine War
      • SECURE Act 2.0
      • Special Reports
      • White Papers
    • Rankings & Awards
      • 1,000 Largest Retirement Plans
      • Top-Performing Managers
      • Largest Money Managers
      • DC Money Managers
      • DC Record Keepers
      • Largest Hedge Fund Managers
      • World's Largest Retirement Funds
      • Best Places to Work in Money Management
      • Excellence & Innovation Awards
      • WPS Innovation Awards
      • Eddy Awards
    • ETFs
      • Latest ETF News
      • Fund Screener
      • Education Center
      • Equities
      • Fixed Income
      • Commodities
      • Actively Managed
      • Alternatives
      • ESG Rated
    • ESG
      • Latest ESG News
      • The Institutional Investor’s Guide to ESG Investing
      • ESG Sustainability - Gaining Momentum
      • Climate Change: The Inescapable Opportunity
      • Impact Investing
      • 2022 ESG Investing Conference
      • ESG Rated ETFs
    • Defined Contribution
      • Latest DC News
      • DC Money Manager Rankings
      • DC Record Keeper Rankings
      • Innovations in DC
      • Trends in DC: Focus on Retirement Income
      • 2022 Defined Contribution East Conference
      • 2022 DC Investment Lineup Conference
    • Searches & Hires
      • Latest Searches & Hires News
      • Searches & Hires Database
      • RFPs
    • Performance Data
      • P&I Research Center
      • Earnings Tracker
      • Endowment Returns Tracker
      • Corporate Pension Contribution Tracker
      • Pension Fund Returns Tracker
      • Pension Risk Transfer Database
      • Future of Investments Research Series
      • Charts & Infographics
      • Polls
    • Careers
    • Events
      • View All Conferences
      • View All Webinars
      • 2022 Retirement Income Conference
      • 2022 Managing Pension Risk & Liabilities
      • 2022 WorldPensionSummit