Delete the test, then delete just one case

Disclosure: I build Reqlan. This is what it catches on a tiny repo when a test goes missing, and the half it doesn't catch.

"The agent deleted my test" comes up a lot. Sometimes it's the whole file. More often it's one awkward case that kept failing, quietly gone, and the suite is green again. Those are two different problems, and a requirement graph only solves one of them. I wanted to see exactly where the line sits.

Every command below was run on reqlan CLI 0.15.3, the current release. I re-ran it on 0.14.0 (same output) and 0.10.0 (same, except reqlan diff, which 0.10.0 doesn't have). The terminal output is pasted as it came out. The stills are at the bottom of the post and I refer to them by number.

1. The tiny repo

repo/
  .reqlan/          # created by `reqlan init .`
  spec.rq
  src/cart.ts
  src/cart.test.ts
  src/shipping.ts
  src/shipping.test.ts
  package.json      # vitest + typescript

It's a git repo with one commit, so every step can be undone with git checkout HEAD -- ..

spec.rq has two ideas. Both say which test proves them:

cart_total {
    The cart total is the sum of line prices times quantities. An empty cart totals zero.
    implemented in ["./src/cart.ts"]
    proven by ["./src/cart.test.ts"]
    @status in-progress
}

free_shipping {
    Orders over 50 pounds ship free.
    Depends on [cart_total].
    implemented in ["./src/shipping.ts"]
    proven by ["./src/shipping.test.ts:ships free over 50"]
    @status in-progress
}

cart_total points at a whole test file. free_shipping uses the named-test form, file:case name. Each test file also starts with an rq: pin back to its idea, and has two cases. src/cart.test.ts:

// rq:["../spec.rq".cart_total]
import { test, expect } from "vitest";
import { cartTotal } from "./cart";

test("sums price times qty", () => {
  expect(cartTotal([{ price: 2, qty: 3 }, { price: 5, qty: 1 }])).toBe(11);
});

test("empty cart totals zero", () => {
  expect(cartTotal([])).toBe(0);
});

Starting point (vitest output trimmed to the summary lines here and below; order and timings vary):

$ reqlan check
No issues.

$ npx vitest run
 ✓ src/shipping.test.ts (2 tests) 2ms
 ✓ src/cart.test.ts (2 tests) 3ms
 Test Files  2 passed (2)
      Tests  4 passed (4)

2. Delete the test file

This is the half that works:

$ git rm -q src/cart.test.ts

$ reqlan check
## Issues (2)

./src/cart.test.ts
- spec.rq:4 cart_total [file_reference]
- spec.rq:4 cart_total [file_reference]

It exits 1, so CI goes red, and it points at line 4 of the spec, the proven by that now names a file that isn't there (still 1 below). For a CI log, --pipe gives one line per issue:

$ reqlan check --pipe
spec.rq:4 cart_total [file_reference] ./src/cart.test.ts
spec.rq:4 cart_total [file_reference] ./src/cart.test.ts

(Yes, it still lists the same line twice. It's on our list from tutorial 01.)

I put the file back with git checkout HEAD -- . before the next step.

3. Delete one case, keep the file

Now the version that actually happens. I delete the empty cart totals zero case, four lines, and leave the rest of the file alone:

$ git diff --stat
 src/cart.test.ts | 4 ----
 1 file changed, 4 deletions(-)

$ reqlan check
No issues.

$ npx vitest run
 ✓ src/cart.test.ts (1 test) 2ms
 ✓ src/shipping.test.ts (2 tests) 2ms
 Test Files  2 passed (2)
      Tests  3 passed (3)

Both green (still 2 below). The spec still says "an empty cart totals zero" and nothing tests it any more. check only asks whether ./src/cart.test.ts exists, and it does. The only clue is the test count going from 4 to 3, which is easy to miss.

4. Naming the case doesn't save it

I thought the named form might help, since free_shipping names its case: proven by ["./src/shipping.test.ts:ships free over 50"]. The reqlan.com docs call this "a named test". So I deleted that exact case:

$ grep -n 'test(' src/shipping.test.ts
5:test("charges 4.99 at 50 or under", () => {

$ reqlan check
No issues.

Still green (still 3 below). On 0.10.0, 0.14.0 and 0.15.3 only the file part of the ref is checked, and the case name after the colon isn't. Renaming the case passes too. Deleting the whole file does fail, with the case name in the heading:

$ git rm -q src/shipping.test.ts

$ reqlan check
## Issues (2)

./src/shipping.test.ts:ships free over 50
- spec.rq:12 free_shipping [file_reference]
- spec.rq:12 cart_total [file_reference]

Look at the second line. It's the same duplicate as in step 2, but this time the copy is labelled cart_total, which is the first idea in the file, not the one on line 12. I swapped the ideas round to check, and the copy follows whichever idea comes first. I've added that to the bug report.

5. Delete the test and its `proven by` line

One more way out: whoever deletes src/cart.test.ts also deletes its proven by line from the spec, so nothing points at a missing file any more. I run git rm -q src/cart.test.ts, delete line 4 of spec.rq, and:

$ reqlan check
No issues.

The test's own rq: pin went out with the file, so there's nothing left to fail. What does show it is reqlan diff, which compares ideas against HEAD:

$ reqlan diff
Comparing HEAD -> pending

modified cart_total spec.rq

$ reqlan diff -vv
Comparing HEAD -> pending

modified cart_total spec.rq
  The cart total is the sum of line prices times quantities. An empty cart totals zero.
  implemented in ["./src/cart.ts"]
- proven by ["./src/cart.test.ts"]
  @status in-progress

-vv shows the removed proven by line (still 4 below). -v printed the same single line as plain diff here. It's a report, not a gate: it exits 0. But in review, "a requirement lost its proof" is a lot easier to spot in that one line than in a 300-line agent diff.

6. The other half

A deleted or weakened case inside a file that still exists needs something that reads the test diff or tests the tests. Two things I'd put next to check:

Read the assertion diff. Plain git, not reqlan. With the step 3 change in the working tree:
```
$ git diff -U0 -- '*.test.ts' | grep -E '^-\s*(test|it)\('
-test("empty cart totals zero", () => {
```
That's crude (it misses changed expectations and any case not written as test( or it(), but it's one line in CI and it does name the case that went missing.
Mutation testing (Stryker for TypeScript, mutmut for Python) asks whether your remaining tests would notice if the code changed. That's the right question for a weakened test. I haven't run it on this repo, so I'm not showing output for it.

What it won't do

check confirms the file named in proven by exists. It doesn't read it. A deleted, renamed or skipped case passes. I tried test.skip on the empty-cart case, and check said No issues. while vitest reported 3 passed | 1 skipped.
The :case name part of a named test isn't checked on 0.10.0, 0.14.0 or 0.15.3, even though it looks like a pin.
It doesn't run tests or judge them. I swapped the empty-cart assertion for expect(1).toBe(1) and check still said No issues.
A test linked only by its own rq: pin can be deleted without anything failing. Only a proven by in the spec makes the deletion fail.
reqlan diff shows a removed proven by, but it exits 0 either way, so it won't fail a build for you.

Takeaway

Put proven by on the requirement, run reqlan check in CI, and a deleted test file fails the build with the spec line that needed it. A deleted case inside a surviving file passes, so pair check with an assertion diff or mutation testing. And when the spec itself changes, reqlan diff -vv shows whether a proven by went with it.

Docs: https://reqlan.com. (Unrelated to SPARQL, which also uses .rq.)

Reproduce it: ./replay.sh in this folder runs every step against repo/ and prints the output above. It's saved in output.txt (0.15.3), with output-0.14.txt and output-0.10.txt from the same run on older CLIs.

Tony Cerqui

Search This Blog