| Title: | Multivariate Version of the Diebold-Mariano Test |
|---|---|
| Description: | Allows to perform the multivariate version of the Diebold-Mariano test for equal predictive ability of multiple forecast comparison. Main reference: Mariano, R.S., Preve, D. (2012) <doi:10.1016/j.jeconom.2012.01.014>. |
| Authors: | Krzysztof Drachal [aut, cre] (Faculty of Economic Sciences, University of Warsaw, Poland) |
| Maintainer: | Krzysztof Drachal <[email protected]> |
| License: | GPL-3 |
| Version: | 1.1.5 |
| Built: | 2025-03-09 10:23:57 UTC |
| Source: | https://github.com/cran/multDM |
This function computes loss differential, i.e., differences between losses from -th and -th models.
d_t(e)d_t(e)
e |
|
matrix of loss differentials
Mariano R.S., Preve, D., 2012. Statistical tests for multiple forecast comparison. Journal of Econometrics 169, 123–130.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l)data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l)
This function computes Diebold-Mariano test for the equal predictive accuracy. The null hypothesis of this test is that two forecasts have the same accuracy. The alternative hypothesis can be specified as ”Both forecasts have different accuracy”, ”The first forecast is less accurate than the second forecast”, or ”The first forecast is more accurate than the second forecast”.
DM.test(f1,f2,y,loss.type="SE",h,c=FALSE,H1="same")DM.test(f1,f2,y,loss.type="SE",h,c=FALSE,H1="same")
f1 |
|
f2 |
|
y |
|
loss.type |
method to compute the loss function, |
h |
|
c |
|
H1 |
alternative hypothesis, |
class htest object, list of
statistic |
test statistic |
parameter |
|
alternative |
alternative hypothesis of the test |
p.value |
p-value |
method |
name of the test |
data.name |
names of the tested time-series |
Diebold, F.X., Mariano, R. 1995. Comparing predictive accuracy. Journal of Business and Economic Statistics 13, 253–265.
Harvey, D., Leybourne, S., Newbold, P., 1997. Testing the equality of prediction mean squared errors. International Journal of Forecasting 13, 281–291.
Hyndman, R.J., Koehler, A.B. 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688.
Taylor, S. J., 2005. Asset Price Dynamics, Volatility, and Prediction, Princeton University Press.
Triacca, U., 2018. Comparing Predictive Accuracy of Two Forecasts, https://www.lem.sssup.it/phd/documents/Lesson19.pdf.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts DM.test(f1=forecasts[,1],f2=forecasts[,2],y=ts,loss="SE",h=1,c=FALSE,H1="same")data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts DM.test(f1=forecasts[,1],f2=forecasts[,2],y=ts,loss="SE",h=1,c=FALSE,H1="same")
This function computes various loss functions for given realized values of time-series and a collection of forecasts.
loss(realized,evaluated,loss.type)loss(realized,evaluated,loss.type)
realized |
|
evaluated |
|
loss.type |
method to compute the loss function, |
matrix with columns corresponding to time index and rows to different models
Hyndman, R.J., Koehler, A.B. 2006. Another look at measures of forecast accuracy. International Journal of Forecasting 22, 679–688.
Taylor, S. J., 2005. Asset Price Dynamics, Volatility, and Prediction, Princeton University Press.
Triacca, U., 2018. Comparing Predictive Accuracy of Two Forecasts, https://www.lem.sssup.it/phd/documents/Lesson19.pdf.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE")data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE")
This function selects models with outstanding predictive ability basing on multivariate Diebold-Mariano test MDM.test.
MDM.selection(realized,evaluated,q,alpha,statistic="Sc",loss.type="SE")MDM.selection(realized,evaluated,q,alpha,statistic="Sc",loss.type="SE")
realized |
|
evaluated |
|
q |
|
alpha |
|
statistic |
|
loss.type |
method to compute the loss function, |
class MDM object, list of
outcomes |
|
p.value |
|
alpha |
|
eliminated |
|
Mariano R.S., Preve, D., 2012. Statistical tests for multiple forecast comparison. Journal of Econometrics 169, 123–130.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.selection(realized=ts,evaluated=forecasts,q=10,alpha=0.1,statistic="S",loss.type="AE")data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.selection(realized=ts,evaluated=forecasts,q=10,alpha=0.1,statistic="S",loss.type="AE")
This function computes multivariate Diebold-Mariano test for the equal predictive accuracy of two or more non-nested forecasting models. The null hypothesis of this test is that the evaluated forecasts have the same accuracy. The alternative hypothesis is that Equal predictive accuracy (EPA) does not hold.
MDM.test(realized,evaluated,q,statistic="Sc",loss.type="SE")MDM.test(realized,evaluated,q,statistic="Sc",loss.type="SE")
realized |
|
evaluated |
|
q |
|
statistic |
|
loss.type |
method to compute the loss function, |
class htest object, list of
statistic |
test statistic |
parameter |
|
alternative |
alternative hypothesis of the test |
p.value |
p-value |
method |
name of the test |
data.name |
names of the tested objects |
Mariano R.S., Preve, D., 2012. Statistical tests for multiple forecast comparison. Journal of Econometrics 169, 123–130.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.test(realized=ts,evaluated=forecasts,q=10,statistic="S",loss.type="AE")data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.test(realized=ts,evaluated=forecasts,q=10,statistic="S",loss.type="AE")
Sample artificial data.
data(MDMforecasts)data(MDMforecasts)
MDMforecasts is list object such that
MDMforecasts$ts is vector of time-series which is of interest to model
MDMforecasts$forecasts is matrix of 20 different forecasts of ts from 20 different forecasting models, each row represents different forecast and time is indexed by columns
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.test(realized=ts,evaluated=forecasts,q=10,statistic="S",loss.type="AE")data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts MDM.test(realized=ts,evaluated=forecasts,q=10,statistic="S",loss.type="AE")
Forecasts obtained from various methods applied to crude oil price.
data(oilforecasts)data(oilforecasts)
oilforecasts is matrix object such that its rows correspond to forecasts from various methods, i.e.,
REALIZED is the forecasted time-series,
DMA.DOW is the forecast from Dynamic Model Averaging with the dynamic Occam's window,
BMA.DOW is the forecast from Bayesian Model Averaging with the dynamic Occam's window,
DMA.1V is the forecast from Dynamic Model Averaging applied only to one-variable models,
BMA.1V is the forecast from Bayesian Model Averaging applied only to one-variable models,
DMS.1V is the forecast from Dynamic Model Selection applied only to one-variable models,
BMS.1V is the forecast from Bayesian Model Selection applied only to one-variable models,
TVP is the forecast from Time-Varying Parameters regression,
LASSO is the forecast from LASSO regression,
RIDGE is the forecast from RIDGE regression,
DYN.EL.NET is the forecast from the elastic net regression, with the elastic net mixing parameter changing with time index,
LARS is the forecast from the least-angle regression,
B.LASSO is the forecast from the Bayesian LASSO regression,
B.RIDGE is the forecast from the Bayesian RIDGE regression,
ARIMA is the forecast from the best ARIMA model according to AIC,
NAIVE is the naive forecast, i.e., the last observation is taken as a one-step ahead prediction,
MA is the moving average.
The data were taken from Juvenal and Petrella (2015). They cover the period between 1971 and 2009, and are in quarterly freqency. Time-series with missing observations were excluded from the original data set, resulting finally in 127 explanatory variables, instead of 150 in the original data set. In particular, the excluded time-series are the ones which start date is after 1971. The dependent time-series is the average world price of oil taken in logarithmic differences. The independent time-series represent various stationarity tranformations of macroeconomic and financial variables of the G7 countries and from the oil market, global economic activity and various commodity prices. The details of the original data set are given in the paper by Juvenal and Petrella (2015). The forecasting with various models, based on this data set, was done by the author of this package, just to provide some more concrete example set of forecasts. The independent variables were taken in the first lags. The forgetting parameters in DMA/DMS models were set to 0.99, resulting in the effective rolling window size of 100. Therefore, such a window was taken for the moving average. LASSO and RIDGE (also in the Bayesian versions), the elastic net, the least-angle regression and ARIMA models were estimated in rolling windows of the size of 100 observations. First 100 observations were excluded, and oilforecasts consists of the remaining last observations. The estimations were done with the following packages fDMA, forecast, glmnet, lars and monomvn.
Drachal, K. 2017. fDMA: Dynamic Model Averaging and Dynamic Model Selection for continuous outcomes. https://CRAN.R-project.org/package=fDMA
Friedman, J., Hastie, T., Tibshirani, R. 2010. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1–22.
Gramacy, R.B. 2017. monomvn: Estimation for Multivariate Normal and Student-t Data with Monotone Missingness. https://CRAN.R-project.org/package=monomvn
Hastie, T., Efron, B. 2013. lars: Least Angle Regression, Lasso and Forward Stagewise. https://CRAN.R-project.org/package=lars
Hyndman, R.J., Khandakar. Y. 2008. Automatic time series forecasting: the forecast package for R. Journal of Statistical Software 26, 1–22.
Juvenal, L., Petrella, I. 2015. Speculation in the oil market. Journal of Applied Econometrics 30, 612–649.
data(oilforecasts) ts <- oilforecasts[1,] forecasts <- oilforecasts[-1,] l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) q <- TB_MA(d=d,q.max=4) MDM.selection(realized=ts,evaluated=forecasts,q=q,alpha=0.1,statistic="Sc",loss.type="SE")data(oilforecasts) ts <- oilforecasts[1,] forecasts <- oilforecasts[-1,] l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) q <- TB_MA(d=d,q.max=4) MDM.selection(realized=ts,evaluated=forecasts,q=q,alpha=0.1,statistic="Sc",loss.type="SE")
MDM Object.The function prints selected outcomes obtained from MDM.selection.
## S3 method for class 'MDM' print(x, ...)## S3 method for class 'MDM' print(x, ...)
x |
an object of |
... |
not used |
The function prints models with outstanding predictive ability, their mean loss function, statistics corresponding to their loss differentials (they are the number of the models less one), and orders these statistics. It also prints the p-value of the test and the number of eliminated models. If no models with outstanding predictive ability were found, the funtions prints such an information.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts m <- MDM.selection(realized=ts,evaluated=forecasts,q=10,alpha=0.1,statistic="S",loss.type="AE") print(m)data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts m <- MDM.selection(realized=ts,evaluated=forecasts,q=10,alpha=0.1,statistic="S",loss.type="AE") print(m)
This function computes Tiao-Box test for autocorrelation, i.e, coefficient of p-th lag in VAR(p) model. Its null hypothesis is that p-th lag is not essential. The alternative hypothesis is that it is essential.
TB_AR_test(d,p)TB_AR_test(d,p)
d |
|
p |
|
class htest object, list of
statistic |
test statistic |
parameter |
|
alternative |
alternative hypothesis of the test |
p.value |
p-value |
method |
name of the test |
data.name |
name of the tested time-series |
Tiao, G.C., Box, G.E.P. 1981. Modeling multiple times series with applications. Journal of the American Statistical Association 76, 802–816.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) TB_AR_test(d=d,p=10)data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) TB_AR_test(d=d,p=10)
This function helps to find a lag in stationary VMA process with Tiao-Box procedure, i.e., the lag length beyond which we are willing to assume that the autocorrelation is essentially zero.
TB_MA(d,q.max)TB_MA(d,q.max)
d |
|
q.max |
|
The function searches for correlations smaller than or higher than , where is the lenght of the time-series.
numeric indicating the found lag length
Tiao, G.C., Box, G.E.P. 1981. Modeling multiple times series with applications. Journal of the American Statistical Association 76, 802–816.
data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) TB_MA(d=d,q.max=10)data(MDMforecasts) ts <- MDMforecasts$ts forecasts <- MDMforecasts$forecasts l <- loss(realized=ts,evaluated=forecasts,loss.type="SE") d <- d_t(l) TB_MA(d=d,q.max=10)