Amen on getting good measurements — and broadly applicable to so many other disciplines in addition to engineering. Also appreciate being able to follow the back-and-forth on GitHub... making that process visible helps to promote transparency and accountability :)
To expand on the GitHub piece, I also love in the repo, the `AGENTS.md` file explicitly calls out the following:
"As part of disclosing agent involvement, fill in the ### Agent review section in the PR description. What to disclose, and how it's read:
- Disclose: what model/tool reviewed, whether the review ran in a fresh context and/or on a different model from the author, how many passes, and the findings — issues found, which were fixed, and which were dismissed with a one-line reason each.
- How it's read: maintainers weight the disclosed reviewer model and pass count when deciding how much independent review a PR still needs. Multiple passes, each in a fresh context, often catch issues a single pass misses — prefer that for non-trivial changes.
- Honesty: if no review pass was run, say so explicitly (disclosure is not a substitute for running one where required above). Never report a review that didn't happen — a fabricated or content-free review claim ("reviewed, looks good") is worse than disclosing none.
Amen on getting good measurements — and broadly applicable to so many other disciplines in addition to engineering. Also appreciate being able to follow the back-and-forth on GitHub... making that process visible helps to promote transparency and accountability :)
To expand on the GitHub piece, I also love in the repo, the `AGENTS.md` file explicitly calls out the following:
"As part of disclosing agent involvement, fill in the ### Agent review section in the PR description. What to disclose, and how it's read:
- Disclose: what model/tool reviewed, whether the review ran in a fresh context and/or on a different model from the author, how many passes, and the findings — issues found, which were fixed, and which were dismissed with a one-line reason each.
- How it's read: maintainers weight the disclosed reviewer model and pass count when deciding how much independent review a PR still needs. Multiple passes, each in a fresh context, often catch issues a single pass misses — prefer that for non-trivial changes.
- Honesty: if no review pass was run, say so explicitly (disclosure is not a substitute for running one where required above). Never report a review that didn't happen — a fabricated or content-free review claim ("reviewed, looks good") is worse than disclosing none.
"
https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/AGENTS.md#contribution-policy--required-checks-before-opening-a-pr