In-sample, out-of-sample and walk-forward testing

Split the history before you start. The in-sample period is where you design and tune the rules; the out-of-sample period is held back and used once, with the rules frozen, to see whether the result survives on data the tuning never saw. A common split is about 70% in-sample and 30% out-of-sample, with the out-of-sample period last in time. If you look at the out-of-sample result, change a rule and test again, that period has become in-sample too.

Walk-forward analysis repeats the split on a rolling window: tune on, say, four years, test the chosen settings unchanged on the next year, roll forward one year and repeat. Joining the test years gives a long out-of-sample record produced the way you would really trade: re-tuning from time to time, always on the past. Some practitioners compare the annualized out-of-sample result with the in-sample one, a ratio called walk-forward efficiency, and distrust a system that keeps much less than about half of its in-sample performance; treat that as a rough guide, not a law.

Turn your phone sideways to read the image. Tap to close.

Six rows of rolling windows over the years 2016 to 2025: in each row, four years hatched gray (in-sample: tune the settings) followed by one green year (out-of-sample: run them unchanged), each row shifted one year to the right so the green test years cover 2020 to 2025. Underneath, a green line joins the test years, labeled stitched test results.

Too many parameters, data snooping and parameter stability

Every parameter, filter and exception is a degree of freedom that lets the rules bend around the past. Test 20 values of two settings and you have 400 variants; add three on/off filters and you have 3,200. Even if none of them has an edge, the best of 3,200 will look impressive by chance alone. This is data snooping: the more versions you try on the same data, the more the winner reflects luck. Statisticians have built formal corrections, such as White’s Reality Check (2000) and the probability of backtest overfitting described by Bailey and colleagues, but the practical rules are simple: keep parameters few, count every version you tried, and judge the winner against the best of the same number of random tries.

Then look at the neighbours. A robust setting sits on a plateau: nearby values give similar, decent results. An overfitted one is a spike: one cell shines while its neighbours are mediocre or losing, as on the right of the drawing below. Choose settings from the middle of a plateau, even if a spike scored higher, and treat a best value at the edge of the tested range as a sign that the search is unfinished.

Turn your phone sideways to read the image. Tap to close.

Two 6 × 6 heat maps of profit factor by fast moving-average length (10 to 35, across) and slow length (175 at the top to 50 at the bottom). Left, plateau: neighbours agree, with a check mark: a central block of 1.3 to 1.5 outlined by a dashed green frame, fading to 0.9–1.1 at the edges. Right, spike: one lucky cell, with a cross: a single dark green cell of 2.1 framed in red, surrounded by cells between 0.8 and 1.1.

Look-ahead bias: concrete examples

Look-ahead bias means the test uses information that was not available when the decision was made. It is easy to code by accident, and it can make almost any idea look brilliant. Two common forms are shown below. On the left, a rule decides at the 9:30 open to buy if the day closes above a level, using a close that will not exist until 16:00. On the right, each trade is tagged with whether the market was above its 200-day average, but the tag is computed on the exit day; trades that worked tend to end in strong markets, so the filter looks predictive when it only describes the outcome. Other forms to hunt for:

  • Same-bar decisions: using a bar’s close, high or low to trigger an entry at that bar’s open. Decide on the previous close and enter at the next open instead.
  • Exit-time information: filters, tags or categories computed at the exit or over the whole trade. Compute every filter with data up to the entry bar.
  • Higher-timeframe values before they close: reading today’s daily or weekly value on an intraday bar while it is still forming; see multi-timeframe analysis and repainting.
  • Revised data: fundamentals, economic figures or index membership as known today rather than on the date of the trade.
  • Full-sample statistics: normalizing with the average or volatility of the whole history, which includes the future.

Turn your phone sideways to read the image. Tap to close.

Two panels. Left, titled decided at the open with that day’s close: a green daily candle with dashed lines at the 16:00 close: not known yet and at the 9:30 open: the rule decides here, a buy triangle at the open and a red arrow from the close back down to the open with a red cross; underneath, with a check mark: fix: decide on yesterday’s close, enter at today’s open. Right, titled tagged with what was known only at the exit: a price line rising from entry to exit across a dashed 200-day average, with a red cross on above it on the exit day and a check mark on below it at entry; underneath: fix: filter only on what was known at entry.

Survivorship bias, costs and regime changes

Three more reasons a backtest flatters a strategy:

  • Survivorship bias: testing a stock strategy on today’s index members or today’s listed stocks leaves out the companies that went bankrupt, were taken over or dropped out of the index. The losers vanish from the data, so the results look better than anything you could have traded at the time. Use point-in-time membership and data that includes delisted stocks.
  • Costs and slippage: commission, spread and slippage are paid on every trade and are largest in R for short stops. A strategy with many small trades can turn from profitable to losing once realistic costs are entered; trading expectancy shows the arithmetic. Limit orders that the test assumes always fill are a hidden cost of the same kind.
  • Regime changes: volatility, trend strength, interest rates and market structure change. A system tuned on calm, trending years can fail in a choppy or volatile period. Check results year by year and in rising, falling and sideways markets, and be wary when most of the profit comes from one stretch.

Monte Carlo: reshuffle the trade order

A single backtest is one path through history. Reshuffling the order of its trades many times shows how lucky that path was. With a fixed risk in R, or a fixed fraction of equity, the total is the same in every order, because sums and products do not depend on order; what changes is the path, and with it the maximum drawdown and the longest losing streak.

In the hypothetical example below, 70 trades of +2R and −1R add up to +14R in every order. The backtest’s own order had a maximum drawdown of −6R, but across 1,000 reshuffles the median was −8R, the worst 5% of runs fell 13R or more and the worst case was −19R. Plan for the deeper numbers, not the backtest’s: if you could not sit through a 13R drawdown, the strategy is too big for you at that size. A related test resamples the trades with replacement, which also varies the total and gives a rough range for it.

Turn your phone sideways to read the image. Tap to close.

On the left, the same 70 hypothetical trades in different orders: a black line for the backtest order and four gray reshuffled paths, all ending at the same green point, labeled +14R in every order. On the right, a histogram of the maximum drawdown in 1,000 reshuffles, from −4R to −19R, with markers for backtest −6R, median −8R and, in red, worst 5% −13R.

Random entries: the baseline your strategy must beat

Zero is the wrong benchmark. In a market that rose over the test period, almost any long strategy makes money; in a choppy one, costs alone push random trading below zero. The fair baseline keeps everything about your strategy except the entry signal: same market and period, same side, same stop size, same exit rule, same costs and a similar number of trades, with entries at random times. Run it 1,000 times and see where your strategy falls.

If your strategy sits inside the band of random results, its entries add nothing that luck and market drift cannot explain. If it sits clearly above, the signal may carry information, in this sample. When the strategy was picked as the best of many variants, compare it with the best of the same number of random runs, not with a typical one, or the test is rigged in your favour.

Turn your phone sideways to read the image. Tap to close.

Equity curves over 0 to 200 trades, under the title 1,000 random-entry runs: same side, same stop, same exit rule. A gray hatched band labeled 90% of random runs widens with the number of trades around a dashed median random run that drifts slightly above the zero line. A green strategy line climbs above the band. A note reads in a rising market random longs drift up: beat the band, not zero.

TradingView Strategy Tester caveats

TradingView’s Strategy Tester simulates a Pine Script strategy with a broker emulator, and its defaults can make results look better or worse than real trading. Check the points below before trusting a report, and see how to backtest a trading strategy for the basic workflow. Features and plans change, so confirm the details in TradingView’s current documentation:

  • Fill timing: by default, an order created on a bar’s close fills at the next bar’s open. The process_orders_on_close setting fills it at that close instead, which is only realistic if you could really trade at the close.
  • Intrabar path: on historical bars the emulator only sees the open, high, low and close, so it assumes a path inside each bar. When a bar touches both the stop and the target, the assumed order decides the result. Bar Magnifier, available on Premium and higher plans at the time of writing, uses lower-timeframe data to resolve this, within a limit on the number of intrabar bars.
  • Repainting and look-ahead: request.security() with look-ahead enabled, higher-timeframe values read before their bar closes and recalculation on every tick can all make history differ from what would have happened live; the repainting guide shows how to check.
  • Costs: commission and slippage are zero unless you enter them on the Properties tab.
  • Limit fills: by default, a limit order can fill as soon as price touches it; the Verify price for limit orders setting requires price to trade through it by a number of ticks, which is more realistic at busy levels.
  • Chart type: Heikin Ashi, Renko and other non-standard charts use synthetic prices, so fills on them can be unrealistic; test on standard candles.
  • History: the test covers only the bars loaded on the chart, which may give too few trades for a reliable estimate. Step through the trades by hand with Bar Replay to confirm they match the rules.

A worked example: the best of 400 settings

Hypothetical: a trend-following rule on 10 years of daily data for one stock index fund, 2016 to 2025. It buys when a fast moving average crosses above a slow one and exits on the opposite cross, with a stop at 2 ATR; see the moving average crossover guide. 2016–2022 is in-sample; 2023–2025 is held back.

  1. Search: 20 fast lengths × 20 slow lengths = 400 variants, tested in-sample with commission and slippage.
  2. Winner: fast 13, slow 47, with a profit factor of 2.1 over 64 trades and a maximum drawdown of 7%.
  3. Neighbours: settings one step away (12–14 fast, 45–50 slow) score between 1.0 and 1.2: a spike, not a plateau.
  4. Plateau: a broad area around fast 20–30 and slow 100–150 scores 1.3 to 1.5 everywhere; its centre is fast 25, slow 125.
  5. Monte Carlo: reshuffling the winner’s 64 trades gives a median maximum drawdown of 10% and a worst 5% of 16% or more, far deeper than the 7% in the report.
  6. Random baseline: against single random-entry runs the winner looks special, beating 97% of them, but the best of 400 random runs reached a profit factor of 1.9, almost as high.
  7. Out-of-sample, run once: fast 13 / slow 47 makes a profit factor of 0.9 over 21 trades; fast 25 / slow 125 makes 1.2 over 9 trades, too few to prove anything.
  8. Decision: drop the spike, keep the plateau setting as a hypothesis, forward test it on paper or at a very small size and review it after enough new trades.

Checklist before you trust a backtest

Run through this list before any money depends on a backtest. Each “no” is a reason to keep testing, not to trade:

The same questions apply to any published statistics, including the simulated results table of the UpTickFlow Strategy: it describes the past and is not a promise.

  1. Were the rules written before the test, and did you count every variant you tried?
  2. Is there an out-of-sample period, or a walk-forward record, that the tuning never touched?
  3. Does the chosen setting sit on a plateau, with similar results for nearby values?
  4. Does every filter use only data available at the entry, with no same-bar or exit-time information?
  5. Does the data include delisted stocks and point-in-time index membership?
  6. Are commission, spread and slippage realistic, and do the results survive doubling them?
  7. Are there enough trades, and does the range of two standard errors around the expectancy clear zero?
  8. Does the strategy beat random entries with the same side, stop and exit, and the best of as many random runs as the variants you tried?
  9. Do Monte Carlo drawdowns fit your risk tolerance at the planned size?
  10. Do results hold across years and market regimes, and has a forward test on new data confirmed them?