Reviews · ReferenceAvailable

The report

Every run prints four tables; the same run writes JSON to backtest-{owner}-{repo}.json (or --output). The printed tables are the contract to check. The JSON does not list missed PRs by number.

Sections

Capture curve — doug vs size-only vs random, at each flag rate, plus AUC
Cleared band — cleared count, missed defects, miss rate, density_lift per budget
Per-rule precision — every scoring rule: fired / hit / precision / lift. Dead rules stay in the table
Time-split holdout — hotspots learned on the older half only, scored on the newer half; requires ≥200 PRs and ≥8 defects with ≥4 in each half, otherwise it’s honestly omitted

Schema stability: early preview — field names may still move. The printed tables are the stable contract for now; treat the JSON as versioned-by-commit.

report.json (sketch)

{
  "repo": "getsentry/sentry",
  "prs": 5000,
  "auc": { "doug": 0.640, "size_only": 0.51 },
  "capture": { "0.20": 0.36 },
  "cleared_band": [ { "flag_rate": 0.3,
      "density_lift": 0.61, … } ],
  "rules": [ { "rule": "hotspot-path",
      "lift": 2.34, … } ],
  "holdout": { … }
}