Backtesting
Direct answer
Write the entry and exit as exact rules, run them over historical data with realistic costs, and report the result alongside the sample size and what could bias it. A backtest is only as trustworthy as its stated limitations.
Using information that would not have been available at the time of the trade.
Testing only on instruments that still exist today, which excludes every failure.
Tuning a rule's parameters until it fits the past perfectly and the future not at all.
Spread, commission and slippage that would have erased the edge in live trading.
A result from a few dozen trades is closer to noise than to evidence.
Trying many rules and reporting only the one that happened to work.
This is the same discipline behind the candlestick work: 125 formations are written as executable rules with a stated trend filter and published thresholds, every rule is checked for look-ahead, and the engine is run against a browser mirror until the two agree on every detection. When a measured run is found to have been computed on a superseded engine, it is withdrawn rather than published with a caveat — which is what happened to the current one. Read that study →
The five steps above are the general discipline; the guide covers the failure modes in detail. This is what the engine behind the formation pages does specifically, because a method described in prose is not checkable and one described exactly is.
| Decision | What was chosen |
|---|---|
| Entry | The open of the bar after the formation completes — the earliest price actually available |
| Exit | The close of the Nth bar. No stop, no target: the raw forward move, not a system |
| Direction | Returns signed to the formation’s own claim, so a bearish rule that falls scores positive |
| Benchmark | An arbitrary entry on the same instrument, timeframe and period, sample size matched |
| Correction | Benjamini-Hochberg across the whole family, not per formation |
| Sample | De-overlapped, because consecutive detections share forward windows |
A result is only reproducible if a second implementation agrees. The Python engine and the browser mirror were run against each other across D1, H4 and H1: 81,301 detections, 0 differing. Every rule was checked for look-ahead, and the swing primitive carries a negative test — that a pivot is invisible one bar before it confirms.
The most useful thing on this page is not the method, it is what happened when the method caught something.
A completed run measuring all 107 candlestick formations was ready to publish. It was withdrawn instead: the candlestick rows had been computed with the still-forming bar included, and the engine fingerprint had moved when eighteen chart formations were added. The formation definitions, thresholds and invalidation conditions were unaffected — those describe what is tested, not what was measured — so they remain published. The hit rates did not.
Where this goes wrong
The cost of catching a forming-bar leak late is one withdrawn study. The cost of not catching it is a wrong number that gets quoted for years. A site that reports a figure for every formation regardless of whether it survived scrutiny is easier to read and worth less.
When the re-run completes, every figure appears with its sample size, a benchmark measured on the same bars, a confidence interval, the correction applied across the family of 125, and the rule that produced it — precisely enough that someone else can run it and get the same number. Anything that cannot carry all six is not published.