Almost every honest claim about a strategy comes down to two words: in-sample and out-of-sample. Get the distinction clear and most of backtesting's traps become obvious. Blur it and you will keep mistaking a good fit for a good strategy.
In-sample: the data you tuned on
In-sample data is the history you used to choose your parameters. By definition the strategy has seen it, so it is allowed to look good there — that is what tuning does. A strong in-sample result tells you the optimiser did its job. It does not tell you the strategy will work on anything else.
Out-of-sample: the data you held back
Out-of-sample data is history the strategy never touched during tuning. It stands in for the future: prices the model has no way to have fitted itself to. Performance here is the first real evidence that an edge exists rather than a memory of past noise.
The gap is the signal
The number that matters most is not either result on its own — it is the gap between them. A small gap means the strategy generalises: it behaves on unseen data roughly as it did on tuned data. A large gap means it memorised the past, and the friendly in-sample figure was an illusion. The wider the gap, the louder the overfitting.
Why walk-forward exists
A single in-sample/out-of-sample split is a start, but one out-of-sample window can still be lucky. Walk-forward testing repeats the split again and again — tune, validate on the next unseen window, roll forward — so the out-of-sample evidence is a track record, not a coin flip. And full costs are charged inside every fold, so a strategy must beat its real friction to pass.
To see the rolling version in full, read walk-forward testing explained, then how to spot overfitting in a strategy for the warning signs of a wide gap. When a strategy finally earns trust, the Kraken pillar covers where it eventually runs. Backtests do not guarantee future results.