Out-of-Sample Validation

Out-of-sample validate an ARIMA or VAR forecast of every ticker in the Toolkit instance – fit only on a training portion of the history, forecast the held-out remainder, and score the forecast against what actually happened.

Also known as: hold-out validation, train/test split validation.

See forecast_evaluation_model.get_out_of_sample_validation for the general harness this wraps. Since that function takes a raw Python callable (not serializable for e.g. the MCP-facing tool layer), this controller method instead hardcodes the choice between the two Part-1 forecasting models via the model string:

• model="arima": time_series_model.get_arima_forecast is fit on each ticker’s own training-period series (p, d, q, include_constant control the model, same as get_arima_forecast).
• model="var": time_series_model.get_var_forecast is fit on the training-period series of each ticker together with other_tickers (lags controls the VAR order, defaulting to every other ticker in the Toolkit instance if not given); only that ticker’s own forecast column is scored against its holdout.

The formula is as follows:

• Training sample: the first int(train_fraction·n) observations (80% by default) of the column level (Adj Close by default); the remaining H observations are the holdout

• Ŷₜ: the H-step forecast of the model fitted on the training sample only, an ARIMA(p, d, q) (model=”arima”, ARIMA(1, 1, 1) by default) or a VAR(lags) on the ticker and other_tickers (model=”var”)

Out-of-Sample Validation in Python or via the MCP

Install the Finance Toolkit with:

pip install financetoolkit -U

Then call get_out_of_sample_validation as shown below.

from financetoolkit import Toolkit

toolkit = Toolkit(
    ["BTCUSD", "ETHUSD"],
    api_key="FINANCIAL_MODELING_PREP_KEY",
    start_date="2021-01-01",
    end_date="2025-12-31",
)

toolkit.econometrics.get_out_of_sample_validation()

Or, after setting up the Finance Toolkit MCP, ask:

Which returns:

Using Your Own Data

get_out_of_sample_validation runs on any data, not only prices: ratios, economic indicators, yield curves or your own dataset. Pass a DataFrame, a Series or a dictionary of them to Econometrics as data, with the dates in the rows or the columns, and set changes to how its changes are calculated: "percentage", "log", "difference" for rates and ratios, or None when the data are changes already. For example, for the monthly unemployment rates of the United States and Germany:

from financetoolkit import Economics, Econometrics

economics = Economics(start_date="2000-01-01")

unemployment = economics.get_unemployment_rate(
    countries=["United States", "Germany"], period="monthly"
)

econometrics = Econometrics(data=unemployment, changes="difference")

econometrics.get_out_of_sample_validation()

Parameters

get_out_of_sample_validation accepts the following parameters.

Related Forecast Evaluation