Diebold-Mariano Test

Compare the forecast accuracy of two volatility forecasting methods against realized (squared return) Variance, per asset, via the Diebold-Mariano (1995) test.

Rather than comparing two different assets, this compares two different ways of forecasting the same asset’s next-day Variance – e.g. the simpler exponentially weighted (EWMA, see Risk.get_ewma_volatility) approach against a plain rolling-window Standard Deviation – to determine whether one is significantly more accurate than the other for a given asset.

The squared return is a noisy stand-in for the true Variance. Patton (2011) shows that the squared-error loss still ranks two volatility forecasts correctly with such a proxy, while the absolute-error loss can rank them the wrong way round, so loss="squared" is the safer choice. A negative statistic favors method_a.

The formula is as follows:

Realized Variance: rₜ²

\[\text{EWMA forecast ("ewma"):}\;\; \text{sigmaₜ} ^{2} = \lambda \cdot \text{sigmaₜ₋₁} ^{2} + (1 - \lambda) \cdot \text{rₜ₋₁} ^{2},\;\; \text{with}\; \lambda = \text{lambda}_{`} (0.94 \;\text{by default}),\;\; \text{so the forecast for t uses returns up to t - 1}\]

Rolling forecast (“rolling”): the sample variance (ddof=1) of the window_size (22 by default) returns up to t - 1

\[\text{dₜ} = L(\text{rₜ} ^{2} - f_{a},\; \text{ₜ}) - L(\text{rₜ} ^{2} - f_{b},\; \text{ₜ}),\;\; \text{with}\; L (e) = e ^{2} (\text{loss} = \text{"squared"}) \text{or} | e | (\text{loss} = \text{"absolute"})\] \[\text{DM} = \operatorname{mean}(d) / \sqrt{\text{γ₀} / n} \cdot \sqrt{(n - 1) / n},\;\; \text{with}\; \text{γ₀} = (1 / n) \cdot \operatorname{Σ}(\text{dₜ} - \operatorname{mean}(d)) ^{2} \text{and the Harvey},\;\; \text{Leybourne and Newbold (1997) correction for a one-step horizon}\] \[\text{P-Value} = 2 \cdot P(T > | \text{DM} |) \text{for}\; a \;\text{Student-t distribution with}\; n - 1 \;\text{degrees of freedom}\]

For more information about the method, see the following papers:

Diebold, F.X., & Mariano, R.S. (1995). “Comparing Predictive Accuracy.” Journal of Business & Economic Statistics, 13(3), 253-263.
Patton, A.J. (2011). “Volatility Forecast Comparison Using Imperfect Volatility Proxies.” Journal of Econometrics, 160(1), 246-256.

Also known as: DM test, forecast comparison test.

Diebold-Mariano Test in Python or via the MCP

Install the Finance Toolkit with:

pip install financetoolkit -U

Then call get_diebold_mariano_test as shown below.

from financetoolkit import Toolkit

toolkit = Toolkit(
    ["XOM", "CVX", "SHEL", "BP"],
    api_key="FINANCIAL_MODELING_PREP_KEY",
    start_date="2021-01-01",
    end_date="2025-12-31",
)

toolkit.econometrics.get_diebold_mariano_test(method_a="ewma", method_b="rolling")

Or, after setting up the Finance Toolkit MCP, ask:

Did an EWMA volatility forecast beat a rolling-window forecast for ExxonMobil, Chevron, Shell and BP from 2021 to 2025, according to the Diebold-Mariano test?

Which returns:

	XOM	CVX	SHEL	BP
Diebold-Mariano Statistic	-3.9115	-4.9425	-1.8262	-1.3358
P-Value	0.0001	0	0.0681	0.1818
Mean Loss Differential	0	0	0	0
Observations	1232	1232	1232	1232

Using Your Own Data

get_diebold_mariano_test runs on any data, not only prices: ratios, economic indicators, yield curves or your own dataset. Pass a DataFrame, a Series or a dictionary of them to Econometrics as data, with the dates in the rows or the columns, and set changes to how its changes are calculated: "percentage", "log", "difference" for rates and ratios, or None when the data are changes already. For example, for the monthly unemployment rates of the United States and Germany:

from financetoolkit import Economics, Econometrics

economics = Economics(start_date="2000-01-01")

unemployment = economics.get_unemployment_rate(
    countries=["United States", "Germany"], period="monthly"
)

econometrics = Econometrics(data=unemployment, changes="difference")

econometrics.get_diebold_mariano_test()

Parameters

get_diebold_mariano_test accepts the following parameters.

Parameter	Type	Description	Example
`period`	`str`	The data frequency for returns (daily, weekly, monthly, quarterly, or yearly). Defaults to “daily”, since `window_size` is expressed in return observations of this frequency.	`"weekly"`
`method_a`	`str`	The first volatility forecasting method, one of “ewma” or “rolling”. Defaults to “ewma”.	`"ewma"`
`method_b`	`str`	The second (competing) volatility forecasting method, one of “ewma” or “rolling”. Defaults to “rolling”.	`"rolling"`
`window_size`	`int`	The rolling window size used by the “rolling” method. Defaults to 22 (approximately one trading month).	`26`
`lambda_`	`float`	The decay factor used by the “ewma” method. Defaults to 0.94.	`0.97`
`loss`	`str`	The loss function to compare forecast errors with, one of “squared” or “absolute”. Defaults to “squared”.	`"absolute"`
`include_benchmark`	`bool`	Whether to include “Benchmark” among the assets tested. Defaults to False.	`True`
`rounding`	`int \| None`	The number of decimals to round the results to. Defaults to None.	`2`

Diebold-Mariano Test in Python or via the MCP

Using Your Own Data

Parameters

Related Forecast Evaluation