Reviews · Getting startedAvailable

Quickstart

Backtest any public repository in one command. No account, no server — the CLI talks to GitHub, caches locally, and prints the full scoreboard.

Requirements

Python 3.14+ and uv
A GitHub token — picked up from the environment or the gh CLI. Without one you get the unauthenticated 60 req/hr limit, which is only enough for tiny runs.

What a run does

Labels defect-inducing PRs from revert anchors in git history (dense, zero API quota)
Harvests merged PRs into a resumable local cache (.backtest-cache/)
Replays the review scoring over the window and prints capture, cleared-band, per-rule, and holdout tables

Tip — dodge right-censoring: Young PRs haven’t had time to be reverted yet. Pass --before with a date at least a few weeks back so every PR in the window had a fair chance to fail.

Terminal

# install (from the repo, for now)
$ git clone https://github.com/coldworkshq/doug && cd doug/api
$ uv sync

# backtest a year of history
$ uv run doug-backtest getsentry/sentry \
    --limit 5000 --before 2026-06-15 --labels git

Output (abridged, real run)

5000 scored PRs · 67 in-window labeled defect-inducing

 flag rate     doug  size-only   random
      10%      14%         8%      10%
      20%     36%        19%      20%
      30%     57%        31%      30%

AUC — doug 0.640 · size-only 0.552 · random 0.500