Fractional Differentiation in Python: Keep Memory, Get Stationarity
Integer differencing makes a price series stationary, but it also discards most of the series’ memory. Fractional differencing lets you choose the smallest difference that still passes a stationarity test. This post covers the fixed-width-window (FFD) variant, a numpy-only implementation, and a min-d search. The numbers come from a seeded synthetic random walk, not live market data. We did not test this on live data, a brokerage, or capital.
The problem: integer differencing throws away memory
Returns (d=1) are stationary but memory-erased. Lopez de Prado’s Advances in Financial Machine Learning Ch. 5 (O’Reilly, Wiley) says integer-differentiated series “have a memory cut-off” and that applying complex techniques to memory-erased series “likely leads to false discoveries.” On our seeded series, d=1 gives an ADF tau of −47.773, but its correlation with the original log price falls to 0.022.
Fractional differencing: the theory
Fractional differencing, formalized for time series by Hosking (1981) (Biometrika, JSTOR), generalizes the difference operator to non-integer order (ARFIMA background). The weights are w_0 = 1 and w_k = -w_(k-1) * (d-k+1) / k. A small d keeps long-range dependence while pushing the series toward stationarity. Each lag contributes a decaying fraction of the differencing instead of the full subtraction that d=1 applies.
Fixed-width window FFD
An expanding-window fracdiff gives each point a different weight window, which induces a negative drift. The fixed-width window (FFD) from AFML Ch. 5 (sec. 5.5, p. 83) fixes this by truncating the weights once |w_k| falls below a threshold tau, so every output point uses the same window length. See the mlfinpy docs for a reference description. In our run, tau = 1e-3 gives a window of 73 weights at d = 0.2 and 34 at d = 0.6.
The min-d search: smallest d that passes ADF
Sweep d upward and take the smallest value where the ADF test rejects the unit root, then check that correlation with the original series stays high. A 2021 study in the International Journal of Advances in Engineering Sciences and Applied Mathematics applied this to four stock indexes. Its ADF crossings fall near d ≈ 0.1 for WIG20, about 0.4 for S&P 500, about 0.3 for DAX, and about 0.4 for Nikkei 225. The paper reports a selected range of 0.12 to 0.43. The optimal d is asset-specific, so do not hard-code one value.
Runnable implementation (numpy only)
This script uses only numpy. It implements the weights, the FFD filter, a plain Dickey-Fuller tau (solved with lstsq), and the sweep. The reference implementation is mlfinlab’s fracdiff.py; we cite it and did not run it.
import numpy as np
def frac_weights(d: float, tau: float = 1e-3) -> np.ndarray:
"""w_0 = 1; w_k = -w_{k-1} * (d - k + 1) / k, truncated when |w_k| < tau."""
w = [1.0]
k = 1
while True:
w_k = -w[-1] * (d - k + 1) / k
if abs(w_k) < tau:
break
w.append(w_k)
k += 1
return np.array(w[::-1]) # newest observation gets w_0
def frac_diff_ffd(series: np.ndarray, d: float, tau: float = 1e-3) -> np.ndarray:
"""Fixed-width window fractional differencing."""
w = frac_weights(d, tau)
width = len(w) - 1
out = np.full(len(series), np.nan)
for i in range(width, len(series)):
out[i] = np.dot(w, series[i - width : i + 1])
return out
def adf_tau(x: np.ndarray) -> float:
"""Plain Dickey-Fuller tau: regress dx on x (constant, no lag augmentation)."""
dx = np.diff(x)
x_lag = x[:-1]
X = np.column_stack([np.ones(len(x_lag)), x_lag])
beta, *_ = np.linalg.lstsq(X, dx, rcond=None)
resid = dx - X @ beta
s2 = resid @ resid / (len(dx) - 2)
se = np.sqrt(s2 * np.linalg.inv(X.T @ X)[1, 1])
return beta[1] / se
# Seeded synthetic series: random walk on log price
rng = np.random.default_rng(42)
T, sigma = 2500, 0.012
log_price = np.cumsum(rng.normal(0.0, sigma, T))
CRIT_5PCT = -2.86 # 5% critical value, plain DF tau
print(f"raw log price: ADF {adf_tau(log_price):.3f} corr 1.0000")
ret = np.diff(log_price)
print(f"d=1 (returns): ADF {adf_tau(ret):.3f} corr {np.corrcoef(ret, log_price[1:])[0,1]:.4f}")
for d in [0.2, 0.3, 0.4, 0.5, 0.6]:
ffd = frac_diff_ffd(log_price, d, tau=1e-3)
valid = ~np.isnan(ffd)
t = adf_tau(ffd[valid])
corr = np.corrcoef(ffd[valid], log_price[valid])[0, 1]
status = "PASS" if t < CRIT_5PCT else "fail"
print(f"d={d}: window {len(frac_weights(d, 1e-3))}, ADF {t:.3f} ({status}), corr {corr:.4f}")Running this script prints the table below.
Results: the d-vs-memory tradeoff
Measured in our run on the seeded series (seed 42, T = 2,500, daily sigma 0.012), with tau = 1e-3:
| Series | Window (weights) | ADF tau | Corr vs. price |
|---|---|---|---|
| Raw log price | — | −0.191 (non-stationary) | 1.000 |
| Integer difference d=1 | — | −47.773 (stationary) | 0.022 |
| FFD d=0.2 | 73 | −1.671 (fails 5%) | 0.9963 |
| FFD d=0.3 | 66 | −3.020 (passes 5%) | 0.9887 |
| FFD d=0.4 | 55 | −4.926 | 0.9736 |
| FFD d=0.5 | 44 | −7.884 | 0.9450 |
| FFD d=0.6 | 34 | −12.173 | 0.8931 |
The minimum passing d here is 0.3. It keeps correlation 0.9887 with the price, while d=1 keeps only 0.022. In this run, every extra unit of stationarity costs correlation.
Pitfalls and limits
- Threshold choice. At d = 0.2 the weights decay slowly, so a cutoff of 1e-5 needs a window of thousands of lags. At T = 2,500 the regression ran out of usable rows, and our run hit a singular matrix. That is why every reported result uses tau = 1e-3.
- Synthetic data. The series has no real market structure, so these numbers show the mechanics of the tradeoff, not a tradable edge.
- ADF variant. This is a plain Dickey-Fuller tau with a constant and no lag augmentation. Its 5% critical value (about −2.86) does not apply to augmented ADF tables; statsmodels’
adfulleradds lag terms.
FAQ
Is fractional differentiation better than using returns?
Not universally. Returns (d=1) are stationary but, per AFML Ch. 5, memory-erased. In our synthetic run, the minimum passing fractional d kept correlation 0.9887 with the price, versus 0.022 for returns. Whether that helps your model is an empirical question for each asset. The Springer study reports selected d values from 0.12 to 0.43 across four indexes.
How do I choose d for my own series?
Sweep d upward and take the smallest value where the ADF test rejects the unit root, then check correlation with the original series. Replace the synthetic generator in the script with your own series. Use the plain Dickey-Fuller critical value from the script, or switch to a full augmented test.
Why fixed-width FFD instead of the expanding window?
The expanding-window version gives each point a different weight window and induces a negative drift (AFML Ch. 5; see the mlfinpy docs). FFD truncates the weights at tau so every point uses the same window length.
← Back to all posts
