Is your trading strategy truly profitable or just random?
One common question we often receive from our readers is: how do you evaluate the effectiveness of a trading strategy?
In this post, we explore two fundamental techniques used in quantitative research to assess whether a trading strategy genuinely offers an advantage or whether its performance is likely due to random chance. These techniques are p-values from statistical tests and bootstrapping methods. We break them down with simple examples so they remain accessible even without a strong mathematical background.
Understanding the Basics: What Are We Evaluating?
Imagine you have backtested two trading strategies, Strategy A and Strategy B, over 500 days.
Both strategies have the same average daily return of 0.10%.
Strategy A has low volatility of 0.90% per day.
Strategy B has higher volatility of 2.50% per day.
Since both strategies share the same average return, they reach the same cumulative return over time. However, the path they take is very different due to their volatility.
The chart above shows Strategy A in blue and Strategy B in red. Strategy A grows steadily with relatively small fluctuations. Strategy B, on the other hand, experiences larger swings, both upward and downward, before reaching the same final return.
Even though both strategies end up at the same level, Strategy B exposes the investor to significantly more variability along the way.
Statistical testing and the role of the p-value
To determine whether the observed returns are statistically significant or could have occurred by chance, we use a one-sided t-test.
The idea is simple. We want to measure the probability of observing a similar or higher average return even if the strategy has no real edge. This probability is what we call the p-value, a standard concept in statistical hypothesis testing.
Formally, we test:
- Null hypothesis (H0): the strategy has no positive edge
- Alternative hypothesis (H1): the strategy has a positive edge
After performing the test:
- Strategy A has a p-value of 0.90%
- Strategy B has a p-value of 18.50%
A low p-value means that the observed performance is unlikely to be the result of randomness. This increases our confidence that the strategy has a genuine edge.
A higher p-value suggests that the observed performance could easily occur by chance, as discussed in our analysis of randomness in trading strategies.
In practice, we often use a threshold of 1%. If the p-value is below this level, we consider the result statistically significant.
Connection to False Positives:
The p-value is directly linked to false positives, also known as Type I errors.
A false positive occurs when a strategy appears profitable but in reality has no edge. By setting a significance level, for example 1%, we control the probability of making this mistake.
If the p-value is below this threshold, we reject the null hypothesis while accepting a small risk of being wrong.
Monte Carlo Simulation: Visualizing p-Values
To better understand what a p-value of 18.5% means for Strategy B, we can simulate random return paths.
We assume the strategy has no edge, meaning the average return is zero, but we keep the same volatility of 2.50%.
We then:
Simulate 1,000 return paths
Compute the cumulative return for each path
Compare how many paths achieve a return equal to or higher than Strategy B
The left chart shows all simulated paths, while the red line represents Strategy B.
The right chart shows the distribution of outcomes. Around 18.7% of simulated paths outperform Strategy B. This is very close to the analytical p-value and helps visualize what it represents.
In simple terms, this means there is a meaningful probability that Strategy B’s performance could be explained purely by chance.
Bootstrapping methods
Another approach to evaluating a trading strategy is bootstrapping. Instead of assuming a distribution, we resample the actual observed returns.
The goal is to generate new return paths that preserve the statistical properties of the original data.
The process is straightforward:
- We resample daily returns with replacement
- We generate 1,000 new paths of 252 days
- We compute the cumulative return for each path
- We measure how often the strategy is profitable
The figure above shows both the simulated paths and the distribution of outcomes.
In this case, 74.4% of the paths are profitable, while 25.6% result in a loss.
This 25.6% represents the probability of observing a negative outcome even if the strategy has an edge. This is known as a false negative or Type II error.
It reflects the impact of volatility rather than the absence of an edge.
Why this matters
Bootstrapping helps us understand what kind of performance we should expect going forward.
It allows us to quantify the probability of losses, understand variability, and detect when future performance deviates from what was observed during the backtest.
One of the key advantages of bootstrapping is that it does not rely on assumptions about the underlying distribution of returns.
This is particularly important in financial markets, where returns are often skewed or exhibit fat tails. By relying on empirical data from the backtest, bootstrapping provides a robust way to evaluate strategy performance without imposing unrealistic distributional assumptions.
Key Takeaways
When evaluating a trading strategy, two aspects are particularly important.
p-value (Type I error):
The probability of identifying a strategy as profitable when it actually has no real edge. Lower values increase confidence that the results are not driven by randomness.
Bootstrapping (Type II error):
The probability of experiencing negative performance due to variability, even when the strategy has an edge. A higher proportion of profitable paths indicates a more robust strategy.
In summary:
A low p-value increases confidence that the observed performance is not due to chance.
A high bootstrapping success rate indicates that the strategy is resilient to variability in returns.
In this example, Strategy A is clearly the stronger candidate, combining a low p-value of 0.90% with a very low probability of negative outcomes over a one-year horizon.
In future discussions, we will explore the Bonferroni correction, a statistical method used to reduce the risk of false positives when evaluating multiple trading strategies simultaneously. This becomes increasingly important as the number of tested strategies grows.
Understanding and applying these statistical techniques can significantly improve the reliability of trading strategies and help ensure that decisions are based on solid statistical evidence rather than chance.
In case you have any questions or comments, do not hesitate to write DM or reach me out at [email protected]
