Directional Accuracy Score and Pesaran-Timmermann test in Python

The Pesaran-Timmermann (1992) test is a non-parametric test that determines whether a forecast predicts the change in direction of a time series correctly. That is, the test looks at the directional accuracy of the forecast, rather than the actual forecasted amount. As such, it is particularly suited to evaluate the accuracy of forecasts of economic values, such as returns on investments, where forecasted values can be positive as well as negative.

The Python function below calculates the Directional Accuracy Score, as well as Pesaran-Timmermann statistic and its p-value.

The null hypothesis of the PT test is that the signs in predicted values are independent of those of true values.

import numpy as np
import scipy.stats as stats


def pttest(y, yhat):
    """Given NumPy arrays with predictions and with true values, 
    return Directional Accuracy Score, Pesaran-Timmermann statistic
    and its p-value
    """
    size = y.shape[0]
    pyz = np.sum(np.sign(y) == np.sign(yhat))/size
    py = np.sum(y > 0)/size
    qy = py*(1 - py)/size
    pz = np.sum(yhat > 0)/size
    qz = pz*(1 - pz)/size
    p = py*pz + (1 - py)*(1 - pz)
    v = p*(1 - p)/size
    w = ((2*py - 1)**2) * qz + ((2*pz - 1)**2) * qy + 4*qy*qz
    pt = (pyz - p) / (np.sqrt(v - w))
    pval = 1 - stats.norm.cdf(pt, 0, 1)
    return pyz, pt, pval


if __name__ == "__main__":
    a = np.array([23, -2, 56, 51, 4, -45, -12, -24, -51, 78, -6, -7, -39, 31, 35])
    b = np.array([14, 3, 45, 23, -5, -56, 4, -11, -34, 29, 3, -11, -12, 24, 3])
    dac, pt, pval = pttest(a, b)
    print(f"Directional Accuracy: {dac}, PT stat: {pt}, p-value: {pval}")

This code is on github.

An R implementation of the PT test is available in the rugarch package.

An Excel example can be found here.

Reference

Pesaran, M. and Timmermann, A. (1992) ‘A simple nonparametric test of predictive performance’, Journal of Business and Economic Statistics, 10, 461-465. DOI 10.1080/07350015.1992.10509922.