Regime Detection: Rolling Volatility, Markov Switching, and Forward-Test Bias

Abstract: Regime detection attempts to divide market history into distinct states, typically high and low volatility, to improve position sizing and hedging decisions. Two common mechanical approaches, rolling volatility thresholds and Markov switching models, identify regimes by fitting parameters to past data, but both suffer from a fundamental problem: the in-sample parameters that define regime boundaries often fail on unseen data. This paper compares the two methods, illustrates their mechanics with historical examples, and examines why regime filters typically degrade sharply in forward tests.

The core concept

A market regime is an unobserved state in which the statistical properties of returns, mean, variance, or correlation, differ materially from other periods. The assumption underlying regime detection is that recognizing the active regime helps traders adjust position size, hedge, or pause trading. Two dominant mechanical approaches attempt to infer regime membership from price data alone.

Rolling volatility uses a simple rule: compute volatility over a fixed lookback window (e.g., 20 or 60 trading days) and compare it to a predetermined threshold. If realized volatility exceeds the threshold, the market is in "high volatility regime"; otherwise, "low volatility regime." This approach requires no distributional assumptions and produces a transparent, easy-to-monitor signal.

Markov switching, introduced by Hamilton, models regime membership as a hidden Markov chain with two states and constant transition probabilities between them ^[1]. Each state has its own mean and variance; the filter computes the probability of being in each regime given observed returns. The model is estimated by maximum likelihood on historical data and updated as new returns arrive. Unlike rolling volatility, the Markov framework explicitly models regime persistence and belief updating.

How they work mechanically

Rolling volatility:

1. Choose a lookback window (e.g., 60 trading days).
2. On each day, compute the realized standard deviation of daily returns over the past 60 days.
3. Compare this to a fixed threshold (often set as a percentile of historical volatility, such as the 75th percentile of the training period).
4. If realized volatility exceeds the threshold, classify as "high regime"; otherwise, "low regime."
5. Use the regime classification to adjust trade size or hedge ratios.

The threshold is typically determined in-sample, by taking the median, mean, or a specified percentile of the rolling volatility series over the entire training period. This embedded look-ahead bias is the first clue that the filter may not survive forward testing.

Markov switching:

1. Specify a two-state model with parameters: mean returns μ₁ and μ₂, volatilities σ₁ and σ₂, and constant transition probabilities p₁₁ (probability of staying in regime 1) and p₂₂ (probability of staying in regime 2) ^[1].
2. Estimate all parameters by maximum likelihood on a training sample (e.g., 10 years of historical returns).
3. For each day after training, compute the filtered probability that the current return comes from regime 1 or regime 2, given all observed data and the estimated parameters.
4. Classify as the regime with higher probability and trade accordingly.

The Markov approach avoids an arbitrary threshold choice, but all parameters are frozen at training time. If the true volatility levels or transition probabilities change after training, the filter will misclassify regime membership ^[2].

Worked example

Consider daily returns of a broad U.S. Equity index from 2000 through 2022. Over this 23-year period, rolling 60-day annualized volatility ranged from approximately 8% to nearly 50%. A researcher trains a rolling volatility filter on January 2000 through December 2019 (20 years), computing the median rolling volatility of that period and setting it as the regime threshold.

Applying this threshold forward to 2020-2022, the filter correctly signals "high volatility" during the March 2020 COVID crash (when rolling volatility spiked above 40%) and correctly signals "low volatility" during the calm summer of 2021 (when rolling volatility fell below 15%). In-sample and on this particular forward period, the filter appears to work.

However, when the same procedure is repeated with training ending in 2000 (just after the dot-com crash) and testing from 2001-2005, results degrade. The median volatility computed from 2000 (a crisis year) is roughly 20%, much higher than typical volatility in the calm 2001-2004 period. When applied forward, the filter spends most of 2002-2004 signaling "low volatility" even though absolute volatility was elevated relative to 1999. The filter, trained in a tail event, fails to adapt to normal times ^[3].

A Markov switching model estimated on 2000-2010 might identify "normal" regime (μ ≈ 0.04% daily return, σ ≈ 1.2%) and "crisis" regime (μ ≈ 0.20%, σ ≈ 2.8%) with transition persistence around 0.95 in each state. When applied to 2011-2015, the filter occasionally assigns high crisis probability to normal-volatility periods and vice versa, because the 2000-2010 distribution does not precisely match 2011-2015 ^[2]. The parameters fitted during a decade that included two major crises do not generalize to a calmer decade.

Why regime filters fail forward

Three mechanisms explain performance degradation.

In-sample overfitting: A volatility threshold set at the 75th percentile of a training period is implicitly optimized to that specific period's extremes, skewness, and autocorrelation structure. If the test period has a different distribution of volatility, e.g., more frequent spikes or more clustering, the threshold will misclassify ^[3]. Even a threshold set at a theoretically neutral level, such as the historical median, depends on the sample median being stable, an assumption rarely met over decades.

Regime drift: The statistical properties of volatility regimes themselves are not constant. Equity volatility in the 1990s (lower, more stable) differed materially from volatility in the 2000s (higher, post-crisis crisis clustering). A Markov model estimated across both periods is a compromise that fits neither. Research on regime-switching models applied to equities, interest rates, and exchange rates has found that estimated transition probabilities and state volatilities change substantially across different historical windows ^[2].

Multiple-comparison bias: Practitioners often test many regime definitions (50, 100, or more volatility thresholds, different lookback windows, different regime models) and report results for the one with best in-sample performance ^[4]. Testing 100 candidate thresholds and choosing the highest-Sharpe-ratio performer on 10 years of data will almost certainly produce a threshold that is unreliable on new data. This is a form of selection bias that inflates reported performance substantially.

Unobserved drivers: Regimes are latent. Neither rolling volatility nor Markov switching directly observes the true regime-generating process; both infer it from returns alone. If true regimes depend on unobserved variables, sentiment, leverage, funding conditions, central bank actions, then both approaches will misidentify regime transitions ^[1]. A filter built only on returns misses essential information.

Limitations

Rolling volatility is transparent but assumes stationarity of the threshold, which is empirically false over long periods ^[3]. Small changes in threshold (a few percentage points) produce large changes in filter performance both in-sample and out-of-sample, indicating the filter is fragile and overly sensitive to training-period specifics.

Markov switching is theoretically appealing but imposes strong assumptions: exactly two regimes, constant transition probabilities, normally distributed shocks. Many assets exhibit more than two regimes (low volatility, normal, elevated, crisis, liquidity stress). Fitting a two-state model to data with four true regimes produces noisy classifications and misses important transitions ^[2]. Also, maximum likelihood estimates are sensitive to the training window; extending or shortening the training period often changes estimated parameters substantially, indicating parameter instability.

Both methods assume regime membership is forecastable from returns alone. In practice, regime drivers include monetary policy, credit conditions, and exogenous shocks, none of which are captured by a returns-only filter. A regime classification rule built on returns is necessarily incomplete.

Testing regime filters is subject to severe multiple-comparison bias. If a researcher tests 50 regime definitions on a training set and selects the one with highest Sharpe ratio, the reported performance is almost certainly inflated, sometimes by 50% or more. Independent teams applying the same filter to the same data often report very different results, signaling widespread parameter overfitting in the literature ^[4].

Summary

Regime detection via rolling volatility or Markov switching offers intuitive appeal: divide markets into high and low volatility states and adjust positioning accordingly. Both methods work in restricted settings, such as very short forward windows or assets with clearly defined volatility breaks. However, both suffer from a common pathology: the parameters that define regime boundaries are optimized to the training period and often fail to generalize. The rolling volatility threshold depends on a specific historical distribution of volatility; Markov parameters depend on the period's tail events and mean behavior. When tested on data far removed from the training period, or when market structure changes, both filters degrade sharply. Successful use of regime detection in live trading requires either very frequent reestimation (with its own risks of overfitting) or solid acceptance that regime classifications are noisy estimates with limited predictive power. Most published backtests of regime filters suffer from look-ahead bias, threshold optimization, and multiple testing, making reported performance unreliable for forward use.

Key definitions

Regime: An unobserved state of the market characterized by a distinct conditional distribution of returns, typically differing in variance, mean, or both.

Rolling volatility: The annualized standard deviation of returns computed over a fixed, moving window of recent trading days.

Markov switching: A statistical model in which an unobserved state follows a Markov chain, and observed returns are drawn from a distribution conditional on the current state.

Transition probability: The probability of remaining in the same regime (or switching to another) in the next period, estimated from historical data and held constant in the filter.

In-sample fit: The ability of a model to explain data it was trained on; high in-sample fit does not imply out-of-sample predictive power and often signals overfitting.

Forward test: Evaluation of a trading rule on data excluded from parameter estimation, the most reliable measure of real-world applicability.

Look-ahead bias: Using future information (such as a threshold computed from the entire training sample) to make a classification decision at a past point in time.

References

1. Hamilton, J. D. (1989). "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle." Econometrica, 57(2), 357-384. Https://doi.org/10.2307/1912559

2. Guidolin, M., and Timmermann, A. (2007). "Asset Allocation under Multivariate Regime Switching." Journal of Economic Dynamics and Control, 31(11), 3503-3544. Https://doi.org/10.1016/j.jedc.2006.11.006

3. Ang, A., and Bekaert, G. (2002). "Regime Switches in Interest Rates." Journal of Business and Economic Statistics, 20(2), 163-182. Https://doi.org/10.1198/073500102317351930

4. De Miguel, V., Garlappi, L., and Uppal, R. (2009). "Optimal versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy?" Review of Financial Studies, 22(5), 1915-1953. Https://doi.org/10.1093/rfs/hhn075

Educational research on historical data only. Not investment advice, not a signal, and never a performance promise. Past results do not predict future performance. Every reference is link-verified before publication and every paper is re-audited weekly against the library's editorial standard.

Last reviewed by the PropLedger research pipeline: 2026-10-25. Educational research on historical data, not financial advice.