Review the intent, not the diff

on

We build Reqlan, a tool that supports the workflow below, so read this with that bias in mind. Most of this post is about the workflow, and it works with or without our tool.

Here's a review that'll feel familiar if you lead a team using Claude Code or Cursor agents. A pull request lands. It touches 23 files. About half the lines are tests. CI is green. The description, which the agent also wrote, is confident and tidy. You've got twenty minutes before your next meeting.

Where do you start?

Most of us (us included, more often than we'd like) start at the top of the diff and scroll. That habit came from a world where a human wrote the change, the diff was a few hundred lines, and the author could explain any line if you asked. Each of those assumptions is getting weaker.

Why the diff stopped being a good starting point

Diff review works when the diff is the scarce thing. You read it because a person spent a day producing it and every line reflects a decision.

With agents, the diff is cheap. The agent can produce several hundred plausible lines in minutes, rewrite them when you push back, and add tests that pass against its own code. The scarce thing now is your attention, and the thing you most need to check isn't any particular line. It's whether the change does what was meant.

There's a second problem, and the testing debate this month made it hard to ignore. When an agent writes the code and the tests, the tests mostly tell you the agent agrees with itself. Kun Chen's DeepSWE experiment (thread mirror) found that banning the agent from writing tests didn't measurably hurt success. His explanation was that "both the implementation and the tests were simply the agent's interpretation of our intent." We went through that result in the previous post in detail. The short takeaway for review: a green test suite on an agent PR is weaker evidence than it looks.

So reading the diff top to bottom has two failure modes. It burns attention on lines that don't matter, and it treats the agent's tests as independent evidence when they aren't.

What "review the intent" means in practice

The alternative isn't "don't read the code." It's changing the order. We think of it as a join: intent on one side, claims on the other, and the diff is just whatever's left over.

  1. Start with the intent. What was this change supposed to achieve, in words? Not the PR description, which the agent may have written to match whatever it did, but the requirement as someone agreed it.
  2. Find the claims. Which tests claim to prove that intent? Which code says it implements it?
  3. Read those first. Read the proving tests closely enough to answer: if the intent were violated, would this test fail?
  4. Then skim the rest. Everything not linked to an intent is either scaffolding, refactoring or scope creep. Each deserves a different level of attention, and you can only tell them apart once you know what the intent was.

If you've ever reviewed a change by first asking the author "what's this meant to do?", you already do this. The difference with agents is that "ask the author" isn't reliable. The agent will give you an answer, but it's generated after the fact.

The intent has to live somewhere

This is where it gets practical. You can't review against intent if the intent only exists in a Slack thread, a ticket that's drifted, or the prompt you gave the agent two hours ago.

Spec-driven development, in its various forms, is an attempt to fix that. Tools like Spec Kit, OpenSpec and Kiro all put the intent into a written artefact before the agent starts. That's a big step up from prompting and hoping. We cover how each of them links requirements to tests in the next post.

The thing a reviewer needs on top of a written spec is the connection. A spec document saying "partial refunds must never exceed the amount paid" helps. A spec where that sentence is a named thing, and you can see which tests claim to prove it and which code claims to implement it, helps a lot more. Without the links, you're still doing the join in your head.

And the links drift. That's spec drift: the spec says one thing, the code does another, and nothing tells you when they diverged. With agents producing changes quickly, drift builds up faster than humans notice it.

How reqlan supports this

Here's what our tool does, limited to what we can confirm in the repository.

  • Requirements are named ideas in .rq files, written in plain language.
  • In the spec, an idea can declare proven by links to the tests that prove it.
  • In code and tests, rq: comments pin a location back to an idea, and an idea can also point at code with implemented in.
  • Backlinks are computed, so from an idea you can see everything that points at it without maintaining it by hand.
  • reqlan check statically analyses the graph and flags references that don't resolve and linked files that don't exist.
  • The VS Code/Cursor extension and language server show this while you edit, and an MCP server lets agents search and retrieve ideas from the same graph and run the check.

What that gives a reviewer is a starting point that isn't the diff. For a change that touches refund_partial_order, you can open the idea, read the sentence a human wrote, see which tests claim to prove it, and see which code is pinned to it. That's your reading list. The rest of the diff is context.

It also suggests some cheap signals worth checking before you read anything. To be clear about the seam here: only the first is something check does for you. The rest are questions you ask using the links, by eye:

  • Did reqlan check pass? If the agent renamed a test file and left a proven by dangling, or pinned code to an idea that doesn't exist, you'll know straight away.
  • Did the change touch code linked to an idea, without touching that idea's proving tests? That's worth a look. Maybe it's fine. Maybe the behaviour changed and the tests didn't.
  • Did the change add new code with no pin? Not always a problem, but often where scope creep hides.
  • Did the agent edit the idea itself? If the words of the intent changed in an agent PR, that's the first thing to read, and probably the most important review question in the whole change.

That last one is our favourite. If you keep the named ideas as human-approved text and treat agent edits to them as a red flag, you've put a clear line between "what we want" and "what the agent did." It's the same principle that DeepSWE v1.1 applied to its verifier tests (v1.1 post): keep the thing that judges the work separate from the thing being judged.

A worked example (illustrative)

To make it concrete, here's how we'd walk an imaginary PR. This is a made-up scenario, not a real customer or repo.

The ticket: "Allow partial refunds." The agent's PR changes the refund service, adds a migration, updates two API handlers, and adds 14 tests.

Step 1, intent. Open the named idea refund_partial_order. It says a customer can refund some items without cancelling the rest, and the refund total can never exceed what was paid. Two sentences. That's what you're checking.

Step 2, claims. The idea declares proven by one test file. The refund service and one handler carry rq: refund_partial_order pins. The other handler and the migration carry no pins.

Step 3, read the proof. Open the linked test file. Is there a case where the requested refund exceeds the amount paid, and does it expect a rejection? If not, the most important half of the intent isn't proven, however many tests went green. That's your main review comment, and you found it in about five minutes.

Step 4, the rest. The unpinned handler turns out to also change how full refunds are logged. Out of scope for this ticket. Worth a separate question. The other 11 or so new tests point at nothing. You skim them as scaffolding and move on.

None of this needs a clever tool. You could do it with a written spec and discipline. A graph and a checker make it faster, and they make the "which tests claim to prove what" part visible instead of something you reconstruct each time.

Where this approach runs out

We want to be upfront about where intent-first review stops helping, and where our tool specifically stops.

A declared link isn't a proof. proven by tells you which test claims to prove an idea. If that test asserts nothing useful (the classic assert 1 == 1), the link resolves and check passes. Reqlan won't catch that. You still need to read the proving test, and mutation testing is a good complement for "would this test fail if the code were wrong?" Reqlan doesn't do mutation testing itself.

Deleted test cases slip through. check flags a linked test file that's gone missing. It doesn't notice if an agent deletes one test case inside a file that still exists, even if proven by named that case. If your agents edit proving tests, watch for removed cases in the diff, or protect those files another way.

Renames are manual. Links are by name, and there's no rename support. If an idea is renamed, the references need updating, and check will tell you which ones broke.

Garbage in, garbage out. If the named idea is vague, or the agent wrote it and nobody read it, intent-first review just moves the rubber stamp up one level. The workflow only pays off if a person actually owns the words.

It doesn't replace reading the code. Security issues, performance problems and plain bugs in unlinked code still need someone to look. Intent-first review changes what you read first, not whether you read.

Getting started without changing everything

You don't need to adopt a whole methodology to try this. A small version:

  1. Pick the next agent PR on an area that matters, like billing, auth or anything with money or permissions.
  2. Before opening the diff, write down in one or two sentences what the change must do and must never do.
  3. Find the test that proves the "must never" part. If there isn't one, that's your first review comment.
  4. Only then read the diff.

If that helps, the next step is keeping those sentences somewhere durable and linking them to tests and code. That could be Reqlan, or the traceability features in the spec tool you already use. We compare those options, as honestly as a biased team can, in the next post.

If you want to try ours, it's at reqlan.com, with the source on GitHub.

Sources

Comments

Post a Comment