How the audit works.

Probability of Backtest Overfitting via Combinatorially Symmetric Cross-Validation (Bailey / Borwein / López de Prado). Splits the trials' performance history into two halves across every possible combination, and asks whether the in-sample winner also tends to lose out-of-sample. PBO = 0.5 is equivalent to a coin flip.

Combines statistical evidence (“e-values”, a betting-based alternative to p-values) across many purged train/test splits, in a way that stays valid even when the splits are correlated with each other.

Locked behind an “empirical validity certificate” (Landau/stable-distribution calibration). Only activates if a SizeCertificate object is supplied — the module refuses to take on trust a general robustness property that has already been empirically refuted in a known case.

FAIL if PBO ≥ 0.20, or the global e-value (corrected across CPCV splits) < 1/α, or e-BH selection retains no candidate, or the out-of-sample loss probability exceeds 50%.

Recomputes the Sharpe's p-value after correcting for the number of trials (Bonferroni / Holm / BHY — Benjamini-Hochberg-Yekutieli), then converts the corrected p-value back into a “deflated” Sharpe.

Computes the probability that the true Sharpe exceeds the maximum expected by pure chance after searching that many correlated trials, accounting for skew and kurtosis in the returns (non-normality).

From a corpus of published t-statistics, reconstructs how many unpublished trials must actually have been run (publication bias), rather than trusting the trial count the researcher declares.

FAIL if the haircut destroys significance, or DSR < 0.95, or the publication-bias ratio exceeds 2.0. WARN if the haircut passes but destroys more than 50% of the Sharpe, or if returns are markedly non-normal.

A sequential “e-process” test, valid at any stopping time (Shawn 2026 — unpublished, not peer-reviewed, single author, no code released; validated here only against a fixed McNemar baseline). Treats “does the candidate beat the incumbent” as a sequential betting game whose wealth process is a supermartingale under the null; Ville's inequality bounds the false-positive rate at any stopping time, even one chosen after looking at the data.

Lan-DeMets group sequential design, O'Brien-Fleming spending function by default. Computes how far the true false-positive rate is inflated by having looked at the results several times during the research.

E-value based test, normal-mixture and sign/median variants. Monitors an out-of-sample edge continuously with no fixed horizon, valid even if monitoring continues indefinitely — it loses in power what it gains in the right to keep looking.

FAIL if the PACE gate is not passed, if a naively significant result does not survive the sequential correction, or if neither monitoring route (mean / median) rejects H0.

Fonseca's availability calculus and type-and-effect system. Your pipeline is written as a symbolic “term” in a small calculus that tracks, for every computed value, the first instant at which all its underlying data was genuinely available. A pipeline that passes this typing is mathematically proven free of look-ahead (any error can only be over-cautious, never a false positive). It never imports or executes code — it reads the term as pure data.

Reconstructs the same kind of term from a traced execution path. This is no longer a proof — it only covers the path actually exercised.

Detects a different class of leak: not future data, but non-tradable data (suspended or illiquid periods) silently absorbed into a rolling-window computation before a row filter removes them.

Actually runs the pipeline under systematically perturbed input scenarios and checks that outputs obey properties that must hold in the absence of leakage — a resilience test, never a proof of absence (Fonseca, Theorem 3: this is the theoretical optimum for the undecidable fragment; no complete automatic checker is possible).

Looks for two precise code patterns via the syntax tree, without ever executing the file: a shuffled train/test split on time-ordered data, and a transformer (e.g. a scaler) .fit() before the split.

FAIL on any proven violation (line 1), any known leak found by tracing, any mask-contract failure, any violated metamorphic relation, or any high-severity AST finding (line 4).

Replays the same strategy under every combination of the declared choices, plots the distribution of resulting effects, and identifies which single decision explains the most variation.

Three statistics — median, significant share, Stouffer's Z — via a block bootstrap forcing H0. Tests whether the entire family of results, taken together, is incompatible with “no effect”, by resampling rather than an analytic formula (specifications are not independent).

Checks whether the combination chosen for publication is a typical result of the family, or its most flattering exception — the signature of selective reporting.

FAIL if none of the three joint tests rejects H0, or if the published specification sits above the 75th percentile of its own curve.

Bayesian Online Changepoint Detection (Adams & MacKay). Estimates, at each instant, the probability distribution over “how long the current regime has lasted”; a collapse in that most-likely run length signals an apparent break. Purely descriptive — no threshold applied here.

Familywise error rate calibrated by simulation (Martin et al.). Rather than an arbitrary threshold, Monte-Carlo simulates the null to measure empirically which threshold actually produces the intended false-alarm rate.

Optional, requires a multi-asset panel. Tracks the effective rank and spectral gap of a rolling cross-asset correlation structure; an effective rank near N means diversification is intact, near 1 means a single common factor absorbs everything — “diversification has evaporated”.

FAIL — not merely WARN — if the calibrated BOCPD statistic falls below the sFWER threshold. A detected regime change is escalated to FAIL precisely because it compromises the assumption every other module rests on.

Does each targeted module return bit-identical output when the same input is replayed with the same random seed? A prerequisite for everything else — without it, no relation violation can be distinguished from ordinary execution noise.

Checks whether necessary invariant properties of the targeted modules hold on the user's real inputs (e.g. scale invariance of a statistic). These relations were derived specifically for this project — no existing generic methodology provided relations for a statistical audit.

Known defects (“mutants”) are deliberately injected into the targeted modules, and the module checks whether the retained metamorphic relations still catch them. “A relation suite that passes while detecting nothing proves nothing.”

FAIL if a targeted module is not reproducible, if a metamorphic relation is violated, or (by default) if an injected defect escapes every retained relation.

Transient market-impact model of the Almgren-Chriss type, calibrated on real market data. From the share of market volume the strategy claims to have taken each period, computes the expected cost a real market would have charged, and flags a declared cost that is implausibly low (below the bid-ask spread alone, or below an “optimism ratio” band around the expected cost). The module explicitly refuses to guess market parameters — inventing them would produce a verdict worse than no verdict at all.

Comparison against an independent reference implementation — a minimal, auditable transcription of Yin et al.'s “Algorithm 1”. Independently recomputes the equity curve and Sharpe from raw weights, prices and cost rate, and compares against what was actually reported — catching invisible bugs such as a commission rate accidentally divided by 100. A Sharpe sign flip between the two (CSI) is the gravest possible finding.

FAIL if the declared cost is below the spread floor, if the optimism ratio exceeds 2.0, if the sign of the Sharpe flips (CSI), or if the reported return diverges from the reference by more than 1%.

At zero transaction cost, two correct engines must agree exactly. This test conditions the interpretation of everything else — if it fails or is absent, no divergence measured at real cost can be attributed to the cost model alone (it could come from a calendar, alignment, or valuation bug).

Four metrics, including CSI (Conclusion Sign Inversion) — the only threshold-free indicator: it blocks by construction if the sign of the Sharpe flips between two engines.

Compares, like a condition number, how much this particular backtest's complexity amplifies disagreement between engines relative to a minimal-complexity case — requires a second (minimal) case supplied separately.

FAIL if zero-cost agreement is broken, if the sign of the Sharpe flips (CSI), or if the Sharpe gap between engines exceeds 0.5.