Reviews · ConceptsAvailable
Defect labels
A backtest is only as honest as its labels. Coldworks labels a PR defect-inducing when later history reverts it — the revert is an engineer’s own on-the-record verdict that the change was bad.
Label sources (--labels)
- gitdefault
- Parse revert anchors out of git history directly. Dense, zero API quota, and reproducible from a clone alone.
- api
- Search GitHub for revert PRs via the API. Sparser but catches reverts that never say "revert" in the commit subject.
- both
- Union of the two. Labels are intersected with the harvested window — no outcome-dependent injection, which would bias the capture numbers.
Known limits, stated plainly: Reverts under-count defects (hotfixes without reverts are missed), and ~6% of git labels on sentry show timestamp anomalies we’re still triaging. The labels are good enough to rank on; we publish their weaknesses along with the wins.
# merged PR #4821 — "migrate retry-queue table"
merge 2026-03-02 a41f9c2
# nine days later:
commit 2026-03-11 e77d201
"Revert 'migrate retry-queue table' (#4821)"
→ #4821 labeled defect-inducing
# an engineer said so, in the record, at the time