Method · Method
The short answer
Detection rules, six exit rules, the conditioning grid, the sample floor, and what the test cannot tell you.
More than it appears. A formation is not a strategy — it has no entry price, no exit and no position size — so a backtest has to supply all three, and every choice changes the result.
A hit rate on its own is unquotable. The question is never "did it close higher 58% of the time" but "did it close higher more often than an arbitrary entry on the same bars, in the same period, on the same instrument".
Without that benchmark you are measuring the instrument's drift and attributing it to the pattern. On an instrument that rose through the sample, every bullish formation looks predictive.
Overlapping samples. Formations cluster. Twelve detections inside a twenty-bar stretch share most of their forward windows, so they are not twelve independent observations. Treating them as independent narrows the confidence interval by roughly the square root of the overlap, and turns noise into significance.
The family. Backtesting 107 formations is 107 tests. The threshold has to be corrected across all of them, and the correction has to be applied to the whole family — including the ones that did not work, which is what makes it a correction rather than a selection.
Where this goes wrong
Both errors push in the same direction: toward finding an edge that is not there. That is why a study reporting no edge is more likely to be sound than one reporting several.
The run for these 107 formations has been withdrawn — superseded engine, forming bar included — and is being recomputed. What is published in the meantime is the part a backtest depends on and rarely discloses: the exact rule, the thresholds, and the strict and loose variants that show how sensitive the count is to those thresholds.
See how to backtest a trading strategy for the general discipline, or the backtesting page for how it applies here.
The obvious next step after a null result is to slice: maybe the formation works in uptrends, or on daily bars, or in high volatility. Each slice is a reasonable hypothesis, and together they are a statistical trap.
Slicing is legitimate, but every slice enters the family the correction is applied across. A result reported from one slice, with the correction computed only over the 107 unsliced tests, is reporting a false positive with a confidence interval attached.
Formations cluster. In a choppy stretch the same rule fires repeatedly within a few bars, and those detections share most of their forward window — so they are not independent observations of the formation, they are one observation counted several times.
Treating overlapping detections as independent narrows the interval by about the square root of the overlap factor, and that is usually the difference between a significant result and a null one. Any published study that reports a raw occurrence count without addressing overlap has probably understated its own uncertainty.
Most published candlestick backtests can be assessed quickly, because the disclosures that matter are short and their absence is conspicuous.
| Disclosed | If missing |
|---|---|
| Exact thresholds | The count cannot be reproduced or compared |
| Benchmark on the same bars | The figure includes the market's drift |
| Correction across the family | Roughly 5% of reported findings are noise |
| Effective vs raw sample | The interval is narrower than the data supports |
A study missing all four is not weak evidence, it is an unfalsifiable claim with numbers attached.