How the audit works.

Probability of Backtest Overfitting via Combinatorially Symmetric Cross-Validation (Bailey / Borwein / López de Prado). Splits the trials' performance history into two halves across every possible combination, and asks whether the in-sample winner also tends to lose out-of-sample. PBO = 0.5 is equivalent to a coin flip.

Combines statistical evidence (“e-values”, a betting-based alternative to p-values) across many purged train/test splits, in a way that stays valid even when the splits are correlated with each other.

Locked behind an “empirical validity certificate” (Landau/stable-distribution calibration). Only activates if a SizeCertificate object is supplied — the module refuses to take on trust a general robustness property that has already been empirically refuted in a known case.

NOT ESTABLISHED (UNPROVEN) if PBO ≥ 0.20, or the global e-value (corrected across CPCV splits) < 1/α, or e-BH selection retains no candidate, or the out-of-sample loss probability exceeds 50%. A statistical conclusion — the evidence does not establish skill — not a demonstrated defect.

Recomputes the metric's p-value after correcting for the number of trials (Bonferroni / Holm / BHY — Benjamini-Hochberg-Yekutieli), then converts the corrected p-value back into a “deflated” score.

Computes the probability that the true value exceeds the maximum expected by pure chance after searching that many correlated trials, accounting for skew and kurtosis in the outputs (non-normality).

From a corpus of published t-statistics, reconstructs how many unpublished trials must actually have been run (publication bias), rather than trusting the trial count the researcher declares.

NOT ESTABLISHED (UNPROVEN) if the haircut destroys significance, or DSR < 0.95, or the publication-bias ratio exceeds 2.0. WARN if the haircut passes but destroys more than 50% of the metric, or if outputs are markedly non-normal. An impossible trial count (n_trials < 1) is refused, never computed. The verdict is conditional on the declared n_trials, which cannot be verified.

A sequential “e-process” test, valid at any stopping time (Shawn 2026 — unpublished, not peer-reviewed, single author, no code released; validated here only against a fixed McNemar baseline). Treats “does the candidate beat the incumbent” as a sequential betting game whose wealth process is a supermartingale under the null; Ville's inequality bounds the false-positive rate at any stopping time, even one chosen after looking at the data.

Lan-DeMets group sequential design, O'Brien-Fleming spending function by default. Computes how far the true false-positive rate is inflated by having looked at the results several times during the research.

E-value based test, normal-mixture and sign/median variants. Monitors an out-of-sample edge continuously with no fixed horizon, valid even if monitoring continues indefinitely — it loses in power what it gains in the right to keep looking.

NOT ESTABLISHED (UNPROVEN) if the PACE gate is not passed, if a naively significant result does not survive the sequential correction, or if neither monitoring route (mean / median) rejects H0. A PACE gate with fewer than 20 discordant pairs is underpowered: WARN, not a rejection.

Fonseca's availability calculus and type-and-effect system. Your pipeline is written as a symbolic “term” in a small calculus that tracks, for every computed value, the first instant at which all its underlying data was genuinely available. A pipeline that passes this typing is mathematically proven free of look-ahead (any error can only be over-cautious, never a false positive). It never imports or executes code — it reads the term as pure data.

Reconstructs the same kind of term from a traced execution path. This is no longer a proof — it only covers the path actually exercised.

Detects a different class of leak: not future data, but non-usable data (gapped or unavailable periods) silently absorbed into a rolling-window computation before a row filter removes them.

Actually runs the pipeline under systematically perturbed input scenarios and checks that outputs obey properties that must hold in the absence of leakage — a resilience test, never a proof of absence (Fonseca, Theorem 3: this is the theoretical optimum for the undecidable fragment; no complete automatic checker is possible).

Looks for four precise code patterns via the syntax tree, without ever executing the file: a shuffled train/test split on time-ordered data, a transformer (e.g. a scaler) .fit() before the split, a shuffling cross-validation splitter (KFold(shuffle=True), ShuffleSplit…), and a full-sample statistic (pandas or numpy mean / std / min / max…) applied to a variable before it is split. Not covered by this line: look-ahead through index shifts (shift(-k)), non point-in-time universes, undated fundamentals — lines 1 and 3 are the ones that address those.

FAIL — a demonstrated defect — on any proven violation (line 1), any known leak found by tracing, any mask-contract failure, any violated metamorphic relation, or any high-severity AST finding (line 4).

Replays the same model under every combination of the declared choices, plots the distribution of resulting effects, and identifies which single decision explains the most variation.

Three statistics — median, significant share, Stouffer's Z — via a block bootstrap forcing H0. Tests whether the entire family of results, taken together, is incompatible with “no effect”, by resampling rather than an analytic formula (specifications are not independent).

Checks whether the combination chosen for publication is a typical result of the family, or its most flattering exception — the signature of selective reporting.

NOT ESTABLISHED (UNPROVEN) if none of the three joint tests rejects H0, or if the published specification sits above the 75th percentile of its own curve. Grid alternatives may themselves be lists (e.g. sets of weekdays).

Bayesian Online Changepoint Detection (Adams & MacKay), on rank-based normal scores so that fat tails do not read as breaks. Estimates, at each instant, the probability distribution over “how long the current regime has lasted”. Purely descriptive.

Familywise error rate calibrated by simulation (Martin et al.). Reported for information only: its null assumes i.i.d. returns, so it fires on persistent volatility alone. Techniques D and E carry the verdict.

Optional, requires a multi-asset panel. Tracks the effective rank and spectral gap of a rolling cross-asset correlation structure; an effective rank near N means diversification is intact, near 1 means a single common factor absorbs everything — “diversification has evaporated”.

Did the mean return — hence the Sharpe — change? CUSUM on rank scores of returns divided by a causal EWMA volatility, normalised by a long-run variance, over the whole series and four sub-windows (Bonferroni-corrected), with Monte-Carlo thresholds. Measured on simulations: 79% detection of a Sharpe +2 → −0.5 switch, 93–98% of temporary episodes and offsetting breaks; a +1 → 0 switch is only caught about 30% of the time, an information limit rather than a defect.

Did the level of risk change beyond normal volatility clustering? κ2 test (Sansó, Aragó & Carrion) with a p-value from a parametric bootstrap under a GARCH(1,1) fitted to the series itself. Measured: 0–10% false alarms on persistent volatility, 80–92% detection of a ×1.25 to ×1.5 change.

NOT ESTABLISHED (UNPROVEN) if a performance break is detected (technique D): every single-number summary of the audit then averages two different performances. WARN if only a volatility break is detected (technique E): the risk taken changed, which is not a verdict on the strategy. The BOCPD alert (A, B) is descriptive and never sets the verdict.

Does each targeted module return bit-identical output when the same input is replayed with the same random seed? A prerequisite for everything else — without it, no relation violation can be distinguished from ordinary execution noise.

Checks whether necessary invariant properties of the targeted modules hold on the user's real inputs (e.g. scale invariance of a statistic). These relations were derived specifically for this project — no existing generic methodology provided relations for a statistical audit.

Known defects (“mutants”) are deliberately injected into the targeted modules, and the module checks whether the retained metamorphic relations still catch them. “A relation suite that passes while detecting nothing proves nothing.”

FAIL if a targeted module crashes, is not reproducible, or violates a metamorphic relation — a demonstrated defect of the tool, and the failure names the property. NOT ESTABLISHED (UNPROVEN) if an injected defect escapes every retained relation on your inputs: the tool cannot prove it would catch it there. Either way this module judges the tool, never your strategy: its failure only prevents the audit as a whole from reading PASS.

Compares each declared value against a reference baseline calibrated on real data, and flags a value that is implausibly small (below a hard floor, or below an “optimism ratio” band around the expected value). The module refuses to guess the baseline when it is not supplied — inventing one would produce a verdict worse than no verdict at all.

Comparison against an independent reference implementation — a minimal, auditable reimplementation. Independently recomputes the output series and headline metric from the raw inputs, and compares against what was actually reported — catching invisible bugs such as a scale factor accidentally divided by 100. A sign flip in the metric between the two (CSI) is the gravest possible finding.

FAIL — a demonstrated defect — if the declared value is below the floor, if the sign of the metric flips (CSI), or if the reported output diverges from the reference by more than 1%. NOT ESTABLISHED (UNPROVEN) if the optimism ratio exceeds 2.0: that ratio comes from a calibrated model, not an identity. A divergence fully explained by the rebalancing convention (the reference rebalances to the target weights on every date, your engine only when they change) is reported as a named WARN, not a FAIL — supply the rebalancing dates to confirm it.

At zero transaction cost, two correct engines must agree exactly. This test conditions the interpretation of everything else — if it fails or is absent, no divergence measured at real cost can be attributed to the cost model alone (it could come from a calendar, alignment, or valuation bug).

Four metrics, including CSI (Conclusion Sign Inversion) — the only threshold-free indicator: it blocks by construction if the sign of the metric flips between two engines.

Compares, like a condition number, how much this particular analysis's complexity amplifies disagreement between engines relative to a minimal-complexity case — requires a second (minimal) case supplied separately.

FAIL — a demonstrated defect — if zero-cost agreement is broken, if the sign of the metric flips (CSI), or if the metric gap between engines exceeds 0.5.