Contributor Review Experience¶
Canonical rules for how reviewers (agentic, e.g. Gitar, and human) communicate with PR authors. Goal: authors respond to a review with effort and stay, rather than disengage. This governs tone, stability, and ordering of feedback. What may block is defined in PR_AUDIT_PROTOCOL.md. Items marked (Project practice) have no cited study.
Sections 2.1, 2.5 and 2.6 are defaults for human reviewers; an agentic reviewer's own configuration (e.g. Gitar's) takes precedence on tone, praise and volume. The blocker rules in section 1 and PR_AUDIT_PROTOCOL.md apply to all reviewers.
1. Stable criteria (no moving goalposts)¶
- Blocker ledger. The first review of a PR posts the complete blocker list. Each blocker has an ID (
BLK-1...), rule class,path:line, evidence, and clearing condition. - Ledger is append-only for the same diff. A later review of unchanged lines shall not add a blocker. If the reviewer missed it, it is posted as
missed-earlierand is an advisory unless it is B1 (safety) or B2 (mechanical failure that was not observable before). - New blockers on new changes only. A push may add blockers only on lines that push changed, or on defects the push introduced.
- Clearing is binary. When the stated clearing condition is met, the blocker closes. The reviewer shall not raise the bar (e.g., ask for more evidence than the clearing condition named).
- Rule changes do not apply retroactively to open PRs. The review rules in force when the PR opened apply until merge, unless a safety rule (S-001..S-007) changed.
- Report progress as closed/open blockers (
BLK-1 cleared, BLK-2 open), never as a percentage. (Project practice.)
2. Comment form¶
- Lead with what is correct when a change is substantially sound; state it specifically rather than as a bare "LGTM". (Project practice.)
- Address the code, not the author. No "you", no sarcasm, no rhetorical questions. Write "this call can return null at
X:42", not "you forgot". - Explain why. Every requested change carries its rationale or a link to the governing rule. Explanations are frequently missing from reviews and are an identified gap. [C4]
- Make it actionable. Prefer a concrete suggestion (GitHub suggestion block, or exact command to run) over a description of a problem. Suggestions make feedback actionable for submitters. [C5]
- Separate severity labels:
blocker,advisory,question,nit. For human reviewers, hold nits until blockers are cleared. (Project practice.) - Bound the volume. As a human default, keep advisories to about 5 per review, highest value first, and summarize the rest as a count. Reviews on many-file changes yield a lower share of useful comments. [C2]
- One consolidated review per push, not a stream of comments. Bots keep noise low; noise and unclear bot output are reported by maintainers and contributors as discouraging. [C3]
- Explain tool output. A failing check names what it means and how to reproduce locally.
3. Newcomers and first PRs¶
- Poor or discouraging feedback on a contribution is associated with contributors not returning; social barriers to a first contribution are documented. [C3][C6]
- For first-time contributors, state expectations once (link
CONTRIBUTING.md), classify the single most important change first, and offer to take over mechanical fixes (formatting, locale parity) where policy allows. - Evidence requirements (gate 10) are stated with the matching artifact and an example, not only as a demand.
4. Pushback self-check (reviewer, before posting)¶
Pushback is a blocked change experienced as unnecessary interpersonal conflict, and it is associated with frustration and developers abandoning projects. It is rare but costly when it happens. [C1] Before posting a blocker, verify:
- It is in B1-B7 with evidence and a clearing condition.
- It was not already visible in an earlier push (section 1.2).
- The comment follows section 2.2 and 2.3.
- The author can clear it in one push without guessing.
5. Response expectations¶
- State expected turnaround for the author-side and reviewer-side once per PR; do not escalate silence into a blocker. (Project practice.)
- A disagreement on a non-safety blocker is resolved by the maintainer citing the rule; the reviewer does not repeat the demand.
6. Measuring (optional, per release)¶
Track per PR: blockers opened vs. cleared, missed-earlier count, push rounds to CLEAR, and author return rate for first-time contributors. Egelman et al. found logged review metrics can flag likely pushback with high recall but low precision, so use them to prompt reviewer self-review, never to judge individuals. [C1]
Evidence & References¶
Verified against search results in the authoring session; full texts were not read. Observational studies show association, not causation.
- [C1] Egelman, Murphy-Hill, Kammer, Hodges, Green, Jaspan, Lin. "Pushback in Code Review: Characterizing and Detecting Negative Interpersonal Interactions" (also titled "Predicting Developers' Negative Feelings about Code Review"). ICSE 2020, pp. 174-185. Survey of 1,317 developers at one company plus review logs; pushback is relatively rare, has negative repercussions, can lead to frustration, stress, and abandoning projects; metrics predict it with high recall, low precision.
- [C2] Bosu, Greiler, Bird. "Characteristics of Useful Code Reviews: An Empirical Study at Microsoft." MSR 2015. ~1.5M comments; usefulness associated with change characteristics such as the number of files.
- [C3] Wessel et al. "Quality gatekeepers: investigating the effects of code review bots on pull request activities." Empirical Software Engineering, 2022. Maintainers reported communication noise and newcomer dropout as unexpected bot effects; the paper cites Steinmacher et al. ICSE 2018 ("Almost there") on poor PR feedback discouraging further contributions.
- [C4] Widyasari et al. "Explaining Explanations: An Empirical Study of Explanations in Code Reviews." ACM TOSEM, 2025. Explanations are important but unevenly supplied (as summarized in the Awesome Code Reviews overview; the paper itself was not opened).
- [C5] "How Do Developers Use Code Suggestions in Pull Request Reviews?" arXiv:2502.04835, 2025. Suggestions make feedback more actionable; reviewers use them more for newcomers. Preprint, 46 projects.
- [C6] Steinmacher et al. "Social barriers faced by newcomers placing their first contribution in open source software projects." CSCW 2015, pp. 1379-1392. Cited via secondary sources; not opened.
Not supported by evidence found: that a fixed ledger or append-only blocker rule reduces dropout (section 1 is project practice derived from the pushback and unnecessary-delay findings), and the specific cap of 5 advisories.