Five Ways a Trading Backtest Lies to You (2026)
A backtest is the only evidence most signal services offer, and it is the easiest evidence to manufacture without meaning to. Five specific things make a curve look better than the strategy is: fitting the rules to the history you tested on, using information that was not available at the time, testing only the instruments that survived, assuming fills you would not have got, and quietly running the test many times until one worked. None of these require dishonesty. All of them produce a chart that goes up and an account that does not.
Disclosure: FuturesAI is our own product and we publish backtest figures too. The checks below apply to ours — we point out where they bite in the last section rather than exempting ourselves.
Jump to a Section
- Fitting the Rules to the Answer
- Using Information You Did Not Have
- Testing Only the Survivors
- Assuming Fills You Would Not Get
- Running It Until It Works
- The Questions That Separate Them
- Where This Bites Our Own Numbers
Fitting the Rules to the Answer
The most common failure is also the most innocent. You have an idea, you test it, results are mediocre, so you adjust a threshold. Better. Adjust another. Better again. Ship it.
What happened is that you used the test period to choose the rules, and then reported performance on that same period. The strategy has been shaped to the particular noise of those specific years. On new data the noise is different and the shaping does not transfer.
The tell is parameter precision. A rule that works at a 14-period lookback and fails at 13 and 15 has found an artefact of that history, not a property of markets. Robust rules degrade gracefully as you nudge their settings; fitted ones fall off a cliff.
The fix is boring and non-negotiable: fix the rules on one stretch of history, then test once on a stretch you have never looked at. If you go back and adjust after seeing the second result, you no longer have an untouched test period — you have two fitted ones.
Using Information You Did Not Have
Look-ahead bias is when the test quietly consumes data that would not have existed at the moment of the decision. It produces the most spectacular equity curves and it is the hardest to spot by eye, because nothing looks wrong.
The usual sources are mundane. A daily indicator computed from the closing price but used to enter during that same day. Fundamental or index-membership data stamped with the date it describes rather than the date it was published. A revised figure used where only the original release was available. Resampling that lets a later bar leak into an earlier one.
The diagnostic is a sanity check on the result itself. An equity curve with very high win rate, very shallow drawdowns and remarkably even returns is more likely to be a leak than an edge. Real strategies are lumpy. When something looks too clean, the first hypothesis should be that the test is seeing the future, not that the idea is brilliant.
Testing Only the Survivors
If you build a universe from the instruments that exist today and test it over the past five years, you have restricted the test to things that did not fail, delist, get acquired or collapse. The losers are missing by construction.
This matters more in crypto than in equities because the failure rate is higher and faster. Tokens that went to zero, exchanges that closed, pairs that lost all liquidity — a universe assembled from today’s listings excludes every one of them, and those are exactly the positions that would have hurt.
The result is a backtest of a strategy applied to a world where nothing dies. Ask whether the instrument universe was reconstructed as it stood at each point in time, or assembled once from the present. The second is far more common and much easier to build.
Assuming Fills You Would Not Get
Most backtests execute at a price that existed and assume you would have got it. Two ways that goes wrong.
Limit orders assumed filled. If the test fills a limit order whenever price touched the level, it is claiming fills on moves that traded through instantly with no depth at your price. Live, some of those never fill — and the ones that do not fill are disproportionately the strongest moves, meaning the test keeps winners you would have missed.
Costs understated. A test with no fees is obviously wrong, but a test with fees and no slippage is the more common and more persuasive error. Slippage is paid twice per round trip and scales with size and thinness.
The sensitivity test is quick and brutal. Re-run the backtest with the cost assumption doubled. If the strategy survives, costs are not carrying the result. If it collapses, the edge was an artefact of an optimistic fill model and the honest conclusion is that there is no strategy here.
Running It Until It Works
Test twenty variants and keep the best one, and you have not found an edge — you have run a search. Some variant was always going to look good, because with enough attempts chance supplies one.
What makes this hard to police is that the failed attempts leave no trace. The published result shows one strategy and one curve. Nothing in the chart reveals that nineteen others were discarded, and the person presenting it may not think of those as attempts at all.
The question that surfaces it: how many variations were tested before this one? A single honest answer is more informative than the entire equity curve. If the number is large, the reported performance needs to clear a much higher bar than if the rules were specified once in advance.
The Questions That Separate Them
Six questions. Any backtest that cannot answer them is a picture, not evidence.
| Ask | What a bad answer looks like |
|---|---|
| Were the rules fixed before this period? | “We refined them as we went” |
| How many trades are in it? | A percentage with no n |
| What cost and slippage were assumed? | Fees only, or nothing |
| How was the universe selected? | Today’s listings, applied backwards |
| How many variants were tried? | No answer, or “a few” |
| What does buy-and-hold return over the same window? | Not shown |
The last row does the most work for the least effort. A strategy that wins 70% of the time and still finishes behind simply holding the asset has not been shown to add anything, and the benchmark is the one number that makes that visible immediately.
Where This Bites Our Own Numbers
We publish backtest figures, so the same questions apply.
Our stated per-trade averages — +0.41% at a one-day horizon and +2.83% at five days — include a 20 basis point round-trip fee. That is better than no cost assumption and it is still a fee assumption rather than a measured fill: real slippage on a thin book would push both numbers down, and the one-day figure has the least room to absorb it.
That backtest is also a single ticker. It is directionally consistent with the broader scoring data, and it is not a market-wide result. Anyone treating it as one is over-reading it, and we would rather say that than let the number travel further than it should.
And the benchmark row applies to us too. Over the same window, simply holding the asset returned substantially more than the strategy did. The win rate was higher; the outcome was not. We publish that alongside rather than behind the accuracy figure, because a backtest without its benchmark is the half of the picture that flatters.
The underlying data is public at FuturesAI if you want to apply the six questions yourself.
🔗 Related
FAQ
Is a longer backtest always better?
Longer is better for sample size and worse if market structure changed within it. A decade spanning several regimes can hide that the strategy only worked in one of them — ask for performance broken out by period, not just the total.
What is a realistic slippage assumption?
It depends on the instrument, your size and the book depth, which is why no single number is right. Measure it on your own fills instead, then use that figure. Doubling whatever you assume is a useful stress test either way.
Does forward testing solve this?
It removes fitting and look-ahead, which is a lot. It does not remove the sample-size problem — a few months of live results on a modest edge still cannot distinguish it from chance.
If every backtest has these problems, why look at them?
Because a backtest that survives the six questions is meaningfully stronger evidence than one that does not, and because the questions are quick. The goal is not a perfect test, it is knowing which way the errors point.
Sources
- FuturesAI public backtest endpoint — single-ticker runs at 1-day and 5-day horizons with a 20 basis point round-trip fee, and the buy-and-hold benchmark over the same window. Retrieved September 25, 2026.
- FuturesAI public scoring endpoint — 26,630 samples across 59 tickers. Retrieved September 25, 2026.
- futuresai.io — public scoring dashboard.
This article is for informational purposes only and is not investment advice. Trading carries risk, and leveraged trading in particular can result in the loss of your entire principal. Backtested results are hypothetical and do not represent actual trading.
