Reviews · HonestyAvailable

What Coldworks gets wrong

Coldworks reviews every pull request in its own repository, and we write down what it got wrong. Just under a third of prospective findings are disproved by code it wasn’t shown — it reads a diff, not a repository, and it does not reliably distinguish what the diff proves from what it merely permits.

Two failure classes recur often enough to have names:

A conclusion drawn from an absence in the diff. “This name is used and no import was added” — when the import was already there, three files away.
Re-reporting a tradeoff the code already documents, as though it were news.

The first one produced a rule worth more than the bug it came from: a claim about an absence cannot be settled by looking at the same place the claim came from. “No import was added” is a fact about the diff; whether the import exists is a fact about the repo. Re-reading the diff confirms the finding every time and proves nothing — the check and the error are the same observation. In that particular case the linter had already answered it, green, before Coldworks ever spoke.

So every finding now gets a line at disposition time: was it real, disproved, or adjacent — wrong as stated, right about something nearby — and separately, did anything in the codebase actually change because of it.

Two axes, deliberately, because one column loses the cases that matter. A true finding that changed nothing is a re-report of something the code already says. A false finding that changed something found a real gap by the wrong route. The reviewer’s strongest mode is “this code does not justify itself” — and a single score would grade that as failure.

There is no rate here yet, and that is the point. Backfill was reconstructed from write-ups after the fact and is excluded from every rate by construction; the denominator is the prospective rows, scoped to one repository and one instrument, because a rate computed across two of either describes neither. The counts in the panel below are a dated snapshot, not a live counter — the log grows every time a finding is settled, and a number printed here cannot follow it. Read the current one off the log yourself; it ships in the repo and the command is in the panel. When there is a rate, it will not be called precision: whether a finding is true is a different quantity from whether it predicted a defect, and a finding can be true and worthless or false and load-bearing.

FINDINGS LOG · ONE LINE PER FINDING

{"pr": 28,
 "rule": "reader:missing-import",
 "verdict": "disproved",
 "changed": false,
 "settled_by": "api/doug/api.py:7 —
   already imported; ruff F821 was
   green before the finding"}

# as of 2026-08-27 — a snapshot, not a
#   counter. 205 rows; 193 prospective,
#   12 backfill excluded from every rate.
#   The reader on doug is 176 of those:
#   54 disproved, 85 real, 37 adjacent.
#   The plan lane writes here too, in its
#   own vocabulary, and is not in that
#   figure.

# today's number, from the log itself:
# python -m doug.findings_log rate \
#     --repo doug --rule-prefix reader: