Research · Concept note
Backtest Overfitting
A condition in which a strategy fits accidental noise in historical data more than a repeatable market principle.
Definition
Backtest overfitting occurs when repeated tests of many rules and parameters select a strategy that excels on historical data but fails on new data. The more combinations explored relative to the number of observations, the easier it is to find a lucky result.
Why it matters
Selecting only the best Sharpe ratio or return hides the many failed experiments. Trading costs, poor data, and look-ahead leakage can widen the gap between backtest and live performance.
Calculation and example
There is no single formula: assess experiment count, degrees of freedom, out-of-sample decay, and parameter stability together.
Testing two moving-average periods from 1 to 200 creates about 40,000 combinations. Reporting only the most profitable pair makes it likely that random sample fit is mistaken for an economic edge.
What to check when interpreting it
- Keep final evaluation data separate from model selection.
- Check whether nearby parameter values produce similar results.
- Use walk-forward validation with realistic fees and slippage.
Common misconception
Passing one out-of-sample test does not remove overfitting. If the strategy is revised after seeing that result, the sample has become development data.