Study guides / CFA L2: CFA Program Level II / Backtesting & simulation
Backtesting & simulation
CFA L2: CFA Program Level II · quick review
Testing an investment strategy on history and by simulation: the aims and steps of a backtest, the metrics and charts it reports, the biases that undermine it, historical scenario analysis, historical versus Monte Carlo simulation, simulation inputs and sensitivity analysis.
The rules
- Objective: a backtest simulates how a strategy would have performed on historical data, to test the investment idea and see its risk and return before real money is committed. It's a rehearsal, not a forecast.
- Steps: (1) design the strategy: the investment hypothesis, universe, signals or factors, weighting, rebalancing frequency and dates; (2) simulate it on historical data; (3) analyze the output. A rolling-window backtest fits the model on an in-sample window, tests it on the following out-of-sample period, then rolls forward, mimicking how the strategy would be used in real time.
- Metrics and visuals: return and risk measures (cumulative return, volatility, Sharpe and Sortino ratios, maximum drawdown, VaR and CVaR), the shape of the return distribution (skewness and kurtosis, especially fat left tails), the information coefficient (often the rank correlation between a signal and later returns), long–short returns from top minus bottom groups, turnover and correlation with other factors. Charts of cumulative wealth and drawdowns show whether results depend on one period.
- Survivorship bias: using only companies that still exist leaves out failures and inflates returns. Use data that include delisted firms.
- Look-ahead bias: using information that wasn't available at the time, such as restated figures or earnings before they were reported. Use point-in-time data and allow for reporting lags.
- Data snooping (overfitting): testing many variations and keeping the best finds patterns that are luck. Remedies: a tougher significance hurdle than usual, out-of-sample and cross-validation tests, and an economic rationale set out before testing.
- Other gaps: backtests often ignore transaction costs, market impact, liquidity and limits on short selling, and markets change (structural breaks), so live results usually fall short of the backtest.
- Historical scenario analysis checks how a strategy performed in different regimes, such as recessions versus expansions, or high versus low volatility or inflation, to see whether its results depend on one environment.
- Historical versus Monte Carlo simulation: historical simulation randomly samples, with replacement, from actual past returns; it keeps real fat tails and correlations but can't produce anything worse than history. Monte Carlo draws from a specified distribution with parameters estimated from history; it can produce new and more extreme outcomes, but results depend on the distribution chosen (model risk).
- Simulation inputs and decisions: the variables and their joint distribution (normal or fat-tailed, with correlations), the parameter values, the number of trials, the horizon and the decision rule being tested. Interpret the whole distribution of outcomes, such as percentiles and the chance of missing a target, not just the average.
- Sensitivity analysis: rerun the analysis with changed assumptions, such as a fat-tailed instead of a normal distribution, different parameters or different factor exposures, to see how robust the conclusions are.
Formulas
- Sharpe ratio: (Rp − Rf) ÷ σp
- Sortino ratio: (Rp − target return) ÷ downside deviation
- Maximum drawdown: Largest peak-to-trough fall in cumulative portfolio value
- Information coefficient: Correlation between factor scores and next-period returns (rank IC uses ranks)
Common traps
- In-sample results overstate what a strategy will earn; only out-of-sample results test it.
- Survivorship bias inflates returns; look-ahead bias uses data no one had at the time.
- Historical simulation can't produce a scenario worse than the worst in its data.
- Testing many variations and reporting the best is data snooping, however strong the statistics look.
How it's tested
Vignettes describe a backtest or simulation and ask which bias affects it, what a metric or chart shows, how to read a historical scenario analysis, or whether historical or Monte Carlo simulation fits the task.
Practise it
The full lesson, worked examples and practice questions are in Tickmark, where every answer choice is explained.
Open the full lesson