GovernanceNovember 19, 20266 min read

AI Doc PRs Pass CI and Still Lose Reviewers' Trust. A New Study Shows Why.

A survey of 239 reviewers found that willingness to accept a CI-passing docs update fell from 85.4% to 66.9% when AI wrote it. The authors call the missing piece reviewability debt. Here is what that looks like in a docs PR, and what to do about it.

Here is a question from a recent survey. A pull request updates some documentation. CI passed. Would you accept it without asking the author for more explanation?

When a human wrote the change, 85.4% of reviewers said they were likely or very likely to. When AI wrote it, that fell to 66.9%.

That number comes from Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests, a paper posted to arXiv in October 2026 by Md Shamimur Rahman, Khairul Alam, Banani Roy and Chanchal K. Roy at the University of Saskatchewan. They surveyed 239 software practitioners in 31 countries. Most had reviewed at least ten AI-authored PRs, and about half had reviewed more than forty.

The paper is about code review in general, not docs. But the docs result stood out to us.

Reviewers don't reject AI PRs outright

The paper doesn't find that reviewers turn AI PRs away for being AI. In the authors' words, "AI authorship was not a categorical rejection signal."

What reviewers wanted was an AI-authored PR that "arrived as an accountable contribution." In their terms that means bounded scope, a rationale grounded in the project, validation beyond CI, a contributor who responds to feedback, and someone identifiable who owns the change after it merges.

When those things were missing, reviewers slowed down. The most common reason to put a PR aside was "a vague or generic AI-generated description" (64.9% of respondents). Very large PRs with no clear decomposition came next, at 51.0%.

If you have reviewed an agent's docs PR titled "Update documentation to reflect recent changes" that touches 30 pages, you already know this feeling.

Why docs took one of the biggest hits

The survey asked about eight kinds of change. Refactors that claim no behavior change had by far the largest drop, from 52.3% to 19.7%. Documentation updates came second, down 18.4 points, about the same as test additions (18.0) and more than low-risk cleanup (12.6).

The paper doesn't analyze docs separately, so what follows is our reading, not theirs.

Refactors drop the most because CI can't prove a refactor preserved behavior. The authors make that point directly: CI "can show that checks passed, but not whether a generated refactor preserved behavior."

Docs have the same problem, worse. CI on a docs repo checks that the build works, links resolve, and maybe that Vale is happy with your style. None of that tells you whether the page is true. Whether the flag is really called --force, whether the default changed last release, whether the feature is still in beta: CI checks none of it. The only validation that counts is a person who knows the product reading the page.

So a docs PR from an agent arrives with green checks that mean very little, a description that may be fluent without showing anyone understood the change, and often no obvious owner. The authors call this reviewability debt.

Reviewability debt

Their definition: "the cost transferred to reviewers when a PR appears complete but leaves intent, scope, validation, or ownership unresolved."

The reviewer pays it, through "defensive checking, repeated clarification, queue delay, or reduced mentoring." If you have approved an AI docs PR without really checking it, you didn't avoid the debt. You passed it on to your readers.

Two other findings fit docs teams well.

First, reviewers weigh behavior in the current PR far more than reputation. A professional response to initial feedback was rated highly or extremely influential by 82.4%, and testing evidence by 81.2%. An active GitHub profile: 23.4%.

Second, the authors say "Disclosure alone is too weak as an AIPR governance mechanism." An "AI-generated" label tells you who wrote the change, not whether anyone checked it.

What they recommend: a stewardship trace

Instead of disclosure, the paper argues for a stewardship trace: "a visible chain from generated output to human judgment that shows what was generated, what was manually revised, what validation was run, what assumptions remain open, and who will maintain the change."

For a docs PR, that translates fairly directly:

  1. What was generated. Which pages the agent wrote or rewrote, and from what (a code diff, an issue, a release).
  2. What a human changed. The edits a reviewer made on top of the agent's text.
  3. What was checked. Not "CI passed." Which claims someone verified against the product, and by whom.
  4. What is still open. "Not sure if this applies to the self-hosted version" is useful. Silence isn't.
  5. Who owns it. The person who answers when a customer says the page is wrong.

The authors also argue against one rule for all AI changes. Routine changes can get light checks, while "refactors, migrations, security, performance, accessibility, and user-facing changes require stronger evidence and ownership." Most documentation is user-facing by definition. A typo fix and a rewritten install guide shouldn't go through the same gate.

What you can do this week

You can start on these without new tooling.

  • Give agent docs PRs a template. Ask for the source of the change, the pages touched, anything the agent was unsure about, and the human owner. A PR that can't fill it in is the PR the survey says reviewers will defer anyway.
  • Cap the scope. One feature or one release per PR. Thirty pages in one diff is how reviewers end up skimming.
  • Route by risk. Changelog wording can go to anyone. Install steps, security pages, pricing and anything legal should go to someone who knows that area, by name.
  • Write down who approved it. A green check from a bot isn't a sign-off. Neither is "LGTM" from someone who never opened the rendered page.

Where DraftView fits

We build DraftView, so take this part with that in mind. It handles the review half of the trace, not the agent half. Your agent still has to say where its change came from.

  • The right person actually reads it. DraftView shows the PR as the rendered page, so a PM, support lead or legal reviewer can check the content without reading a Markdown diff. Their edits go back to the PR as GitHub suggested changes.
  • AI PRs are labelled. Doc PRs from connected repos land in a review queue, labelled as AI-authored when the author, branch, PR body or commit trailers point to an AI tool.
  • Revisions are checked against feedback. When the agent pushes a fix, DraftView looks at whether it touched the parts reviewers commented on, and shows what changed since your review.
  • The approval is a record. Sign-offs are append-only and tied to the commit they approved, with the approver's role and the CI state at the time. If a later push changes the docs, the sign-off is voided and the PR goes back to review, so an approval doesn't carry over to text nobody read. On the Governance plan you can export the record for an auditor.

If you want to see it, try it on any public PR or read how to prove human review of AI-generated documentation.

Caveats worth knowing

The authors are careful about limits, so we should be too. This is a survey: self-reported answers, from a self-selected sample. "AI-authored" meant what respondents perceived as AI-authored, not verified provenance. And every respondent rated the human-authored version before the AI version, so the authors "avoid interpreting the reported differences as effects caused solely by AI authorship."

The docs number is one row in one table. It is a strong hint, not a measured docs workflow.

Even with those limits, the main point is useful. Reviewers aren't refusing AI-written docs. They are objecting to being the only person who checked anything. The paper's answer is to show the human judgment before review starts, not after.

The full paper is on arXiv under CC BY 4.0, and Section 5 is worth reading if you set review policy for your team.