Guide · Backtesting
The short answer
Write the rule so a computer cannot misread it, hold back data you never look at, and expect most versions to fail.
A backtest does not tell you whether a strategy will work. It tells you whether a strategy would have worked, on one sample, under assumptions you chose — and its real value is in making those assumptions explicit enough that someone can disagree with them.
Treated that way it is the most useful tool available. Treated as a forecast it is worse than nothing, because a bad backtest produces confidence rather than doubt.
A subtle case of look-ahead: including the bar currently in progress. Its close is not yet known, so any rule that reads it is reading the future. On a daily series this is a whole session of information.
This is not hypothetical here. The measured run this site was preparing to publish included the forming bar in its candlestick rows, and the run was withdrawn rather than published with a caveat. The definitions, thresholds and detection rules were unaffected — they describe what is tested, not what was measured — but the hit rates went, and they stay gone until the re-run completes.
Worth knowing
The cost of catching this late is one withdrawn study. The cost of not catching it is a permanent, quotable, wrong number. That trade is not close.
The last one is the test of the other four. A rule you can publish as an executable condition is a rule whose result can be checked; anything less is an assertion. Every formation on this site publishes its thresholds for that reason — see the candlestick patterns index.
The single most effective defence against overfitting is data you never looked at. It works because it cannot be gamed by intention — once you have seen a period, you cannot un-see it when choosing parameters.
Where this goes wrong
If the out-of-sample result is much worse than the in-sample one, that is the honest finding. The temptation at that point is to adjust and re-test, which destroys the only unbiased estimate you had.
A single split assumes one parameter set holds for the whole period. Walk-forward tests that assumption instead of making it: fit on a window, trade the next window, roll forward, repeat.
It is more work and it answers a more useful question: whether the strategy survives being re-fitted the way you would actually have to re-fit it. A strategy whose optimal parameters swing wildly between windows is telling you something a single backtest would have hidden.
Most of them. A research process that produces a working strategy from most attempts is not finding edges, it is fitting noise — and the rate at which ideas survive is itself a diagnostic.
If nine in ten of your tested rules fail, the process is probably honest. If most pass, the tests are too loose, the sample is being reused, or the parameters are being chosen after seeing results.
Worth knowing
Keeping a record of the ideas that failed is what makes the correction possible later. A family you cannot count is a family you cannot correct across.