Requirement-to-test traceability compared: Spec Kit, OpenSpec, Kiro and Reqlan

on

We build Reqlan, which is one of the tools in this comparison, so read this with that bias in mind. We've tried to describe each tool from its own docs, credit others where they got there first, and be clear where ours falls short. If we've got something wrong about your tool, tell us and we'll fix it.

Spec-driven development has gone from niche to normal over the last year, mostly because of coding agents. If an agent is going to write the code, you'd better write down what you want first. Most of the tools agree on that part.

Where they differ is what happens after. Once you have a requirement and some tests, how do you know which test proves which requirement? And how do you find out when that link breaks? That's requirements traceability, and with agents writing many of the tests, it matters more than it used to (the first post covers why).

This post compares how several tools handle the requirement-to-test link. It isn't a ranking, more us trying to decompose the problem into a few dimensions and see where each tool lands. These tools do different jobs, and several of them work together.

First, a note on names

There are a few projects with names close to ours, and it's worth clearing up. Reqlan isn't related to reql or reqlang, and it's not a fork or rebrand of SpecGraph. If you searched for one of those and found reqlan, you're in a different place. We're not claiming any connection, and it's not a clone of Spec Kit, OpenSpec or Kiro either. It sits at a different layer, which we'll explain below.

What we're comparing

For each tool, we asked the same four questions:

  1. Where does the link live? In the spec, in the test, or inferred?
  2. How is it found? Declared syntax, token search, or a model guessing?
  3. When is it checked? While editing, in CI, or on demand?
  4. What does it cover? Requirement to test only, or code too?

Spec Kit

Spec Kit is GitHub's toolkit for spec-driven development. It gives you a structured flow from specification to plan to tasks, and it works with a range of coding agents.

On traceability, we couldn't find a built-in requirement-to-test matrix in core Spec Kit. If it's there and we missed it, we'll happily correct this.

The community has filled the gap. The spec-kit-trace extension reads REQ-XXX identifiers in spec.md and the test suite, surfaces untested requirements, and flags orphan tests whose requirement IDs have disappeared from the spec.

Strengths: simple, works with any test framework, easy to adopt, and it covers both directions. Where it stops: the link is a token in the test, found by search. A test that mentions REQ-012 counts as linked whether or not it actually tests that requirement.

OpenSpec

OpenSpec structures requirements as SHALL statements with GIVEN/WHEN/THEN scenarios, which read a lot like test cases. It has a clean change-proposal workflow that suits brownfield work.

Tests aren't a first-class part of the model yet. Users have asked for that in Discussion #1760, and in the meantime people work around it with schema.yaml or AGENTS.md rules telling the agent how to connect scenarios to tests.

Strengths: scenarios are written in a test-shaped way, so the step from scenario to test is short, and the proposal flow is good at keeping change intent explicit. Where it stops: there's no checked link from a scenario to the test that implements it, so you rely on convention and the agent following instructions. The community discussion suggests this is on people's minds.

Kiro

Kiro takes a different and genuinely interesting approach. It writes requirements in EARS notation, extracts properties from them, and generates property-based tests. Its docs describe a "traceable link" between requirements and those tests. Property tests are optional by default.

Strengths: property-based tests check a rule across many generated inputs, which goes some way towards answering "does this test actually check the requirement?" That's a question most traceability tools, ours included, don't answer. Where it stops: the link is to tests Kiro generates. Tests you already have, or write by hand, are a separate matter. And if the same agent extracts the property and writes the code, you still need a person reading the property.

Reqlan

Here's ours, limited to what we can confirm in the repository.

Requirements are named ideas in .rq files, in plain language. In the spec, an idea declares proven by links to the tests that prove it (a file, or a named test case in a file) and can declare implemented in links to code. In code and tests, rq: comments pin a location back to the idea. Backlinks are computed. reqlan check statically analyses the whole graph and flags references that don't resolve and linked files that are missing. A VS Code/Cursor extension and language server show this while you edit, there's a CLI for CI, and an MCP server lets agents search and retrieve ideas from the graph, run the same check, and compare the graph between two git revisions.

Strengths: links are declared, not grepped or inferred. They're checked as a graph, including while you type. It covers idea to test and idea to code in one place. Where it stops (and we want to be specific):

  • It won't catch a test that asserts nothing. If proven by points at a test that's effectively assert 1 == 1, the link is valid and check passes. Mutation testing complements Reqlan here. Reqlan doesn't include mutation testing.
  • It flags a missing test file, not a deleted test case. If an agent removes one test case from a file that still exists, check won't notice, even when proven by named that case.
  • No rename support. Links are by name. Rename an idea and you update references yourself. check shows you what broke.
  • It isn't a workflow. It doesn't do proposals, plans or task breakdowns. Our intent is for it to sit alongside tools like Spec Kit or OpenSpec rather than replace them, but there's no integration with either.

Others worth knowing

This comparison would be unfair without a few tools that do some of this well, in some cases better.

  • Tessl. Spec requirements link to tests with [@test] ../tests/..., and a check-spec-links.sh script checks that the linked files exist. That's the same shape as reqlan's proven by. We didn't invent the idea of declaring the test from the spec.
  • BMAD's Test Architect (TEA). A trace workflow builds a matrix and a PASS/CONCERNS/FAIL gate with coverage thresholds by priority (P0 at 100%, P1 at 90% or more, overall at 80% or more). It resolves the test oracle in order: formal requirements, then contract/spec artefacts, then external pointers, and as a brownfield fallback, "synthetic" journeys inferred from source. The gate is a thoughtful touch for teams that want a release decision, not just a report.
  • StrictDoc. It models REQ, TEST_CASE and TEST_RESULT as typed nodes and can ingest JUnit and Robot XML results (commit). It's aimed at safety and ASPICE work, and it already does rigorous typed requirement-to-test-to-result traceability. If you need that level of rigour, especially with test results linked in, look at StrictDoc first.

Side by side

Tool Where the link lives How it's found When it's checked Covers
Spec Kit + spec-kit-trace REQ-XXX token in tests Token search On demand / CI Req ↔ test
OpenSpec Convention (scenario → test) Agent instructions Not checked Req → scenario
Kiro Generated property tests Generated from EARS When tests run Req → property test
Tessl [@test] in spec Declared Script Req → test file
BMAD TEA Requirements, pointers, or inferred Oracle order + inferred fallback Trace run, gate Req ↔ test, with gate
StrictDoc Typed nodes Declared Build / import Req → test → result
Reqlan proven by / implemented in in spec, rq: comments in code and tests Declared Editor (LSP) + CLI Idea → test, idea → code

The summaries are ours, written from each project's docs, so check the source links for the full picture.

The thing none of them solve

Every tool here, ours included, mostly answers "is there a link?" Few answer "is the linked test any good?" Kiro's property tests and mutation testing come closest. A traceability matrix full of green ticks pointing at tautological tests is still the problem Kun Chen described in his DeepSWE thread (mirror): the agent's tests can be "simply the agent's interpretation of our intent."

What traceability does buy you is knowing where to look. If you can see which test claims to prove which requirement, you can read that test. If a link breaks (a name or file disappears), you hear about it. It won't tell you when the meaning drifts and the names stay put. That's useful, but it's not proof of correctness, and anyone selling it as such (us included) should be called on it.

How to choose

Some honest suggestions, held loosely:

  • Already using Spec Kit and want a quick win? Try spec-kit-trace. It's light and works today.
  • Using OpenSpec? Follow Discussion #1760, and consider something that checks scenario-to-test links alongside it.
  • Want tests that check rules, not examples? Kiro's property-based approach is worth a look, and you can add property tests to any stack.
  • Regulated or safety work? StrictDoc.
  • Want links checked in the editor, across tests and code, in one graph, alongside whichever workflow tool you use? That's what we built Reqlan for. Have a look at reqlan.com or the GitHub repo, and keep the limits above in mind.

Whichever you pick, add mutation testing or property testing for the "does this test actually check anything" question. No traceability tool answers that one for you.

Sources

Comments

Post a Comment