Identifying realistic estimates of future investment strategy performance.

Past performance is no guarantee of future results.

We see this statement everywhere financial performance is advertised. No one expects the future to be exactly like the past, but isn’t it reasonable to expect them to be similar? After all, history may not repeat itself exactly, but to paraphrase Mark Twain, it often “rhymes.” Wouldn’t it be nice to know if the past performance of a specific strategy has a good chance of continuing?

Answering this question effectively is one of the key reasons my partners and I formed our company. In brief, we work with third-party asset managers to estimate the chances an active investment strategy will continue to perform as it has in the past. Using statistically sound methods, we perform a high level of due diligence on active strategies.

So why should a financial advisor care? Because clients count on advisors to provide realistic estimates of future performance despite the ubiquitous “past performance” warning. But how can we do that? Since the past is not completely reliable, how can we generate realistic expectations for the future?

Before discussing the answer to that question, we need to understand why, in some cases, future investment strategy performance can be radically different from the past, even when the market conditions seem historically similar. One major culprit is the data mining bias (DMB).

Haven’t heard of the data mining bias? You’re not alone. Despite the impact it can have on future investment strategy performance, DMB remains relatively unknown, misunderstood, and in some cases, outright ignored. DMB is sometimes referred to by other names, including curve-fitting, overfitting, data-snooping, or overoptimization. Regardless of what we call it, the presence of DMB can fool an asset manager into believing a worthless investment strategy has the ability to produce excellent returns.

History may not repeat itself exactly, but to paraphrase Mark Twain, it often “rhymes.”

We can understand how DMB happens using a popular metaphor. Imagine we give a billion monkeys computer keyboards, provide them rewards for typing, and let them bang away for days, weeks, months, even years. Given enough time, eventually one of them will produce a line or two of Shakespearean prose. Does that mean the monkey who quoted Shakespeare is likely to continue creating literary masterpieces in the future? Of course not.

The Shakespearian monkey example shows that given enough time and resources, it is not only possible, but highly likely that luck will impersonate mastery. Yet, the exact same “luck” effect is present in many human-created investment strategies, but is less easily recognized. Why? Because there are thousands of people creating investment strategies using resources (computing power that was unimaginable 25 years ago) who in aggregate try millions—even billions—of strategy combinations and pick only the best ones to trade through a process called optimization. Using this type of selection process to build investment strategies is exactly the same kind of process that allowed our monkey friends to create prose from random key banging.

Data mining bias can fool an asset manager into believing a worthless investment strategy has the ability to produce excellent returns.

All historical investment strategy results are a combination of both a market edge and luck, although the balance between the two varies considerably from strategy to strategy. But how can we tell the difference? In a moment, we’ll discuss a method, called system parameter permutation (SPP), designed to help identify the real “edge” in a given investment strategy.

But first, let’s discuss another issue with how investment strategy results are typically described. Performance results are usually presented like this: 14% annual return with a maximum drawdown of 20% over the last 10 years. So what’s the problem? These single numbers for return and drawdown—also known as “point” estimates—mask the variability we are likely to see while running the strategy. So, if the strategy returns only 2% one year, is the strategy broken? These so-called point estimates don’t provide much help in answering that question.

Wouldn’t it be much more useful to understand the range of likely outcomes? How about the probability of achieving a certain annual return? Armed with this additional information, advisors can make more informed decisions about the suitability of a given strategy for a client and maybe avoid a few panicked phone calls.

Is there a better way? In my opinion, there is. To arm asset managers with a simple, yet powerful, way to address these issues, I developed a technique called system parameter permutation (SPP). SPP has been well-received, and in May 2014, I was presented the National Association of Active Investment Managers’ prestigious Wagner Award for a paper describing the method.

Wouldn’t it be much more useful to understand the range of likely outcomes? How about the probability of achieving a certain annual return?

SPP offers a practical way of measuring the range of expected system performance, rather than providing a potentially misleading “point” estimate. And remember the optimization processes those investment asset managers used to create a huge number of strategy combinations? The beauty of SPP is that it leverages that optimization data to cut through the DMB and help identify the real investment strategy “edge.”

As shown in the figure, SPP takes all of the optimization data used to build an investment strategy and shows where the “point” estimate (which is what most investment asset managers provide) falls within the range of likely performance. Relatively easy-to-apply statistical techniques can be used to evaluate the likelihood of achieving a specific performance target, and the chance performance will fall within a given range (e.g., a 90% chance of an annual return between -1% and 11%).

In brief, the SPP process involves the following steps:

• A range of investment strategy parameters and an evaluation time period are selected.
• All combinations of the selected parameters are simulated individually (as would be done in exhaustive optimization) using a realistic, portfolio-based backtest engine.
• The results of each simulation are combined to create a range of values for each performance metric of interest.

In the figure, we used compound annual return (CAR) as the metric of interest, but SPP allows us to estimate similar ranges of performance for other metrics, including the maximum drawdown and Sharpe ratio.

In summary, clients rely on advisors to select the most suitable active strategies, but suitability requires an understanding of strategy reliability and realistic ranges for future system performance. SPP provides a simple method asset managers can use to determine both. In turn, this helps advisors using third-party strategies to have greater confidence in the strategies used to achieve their clients’ objectives.

The opinions expressed in this article are those of the author and do not necessarily represent the views of Proactive Advisor Magazine. These opinions are presented for educational purposes only.