Testing what an agent wrote

Every argument for letting an agent write the implementation ends up leaning on the same thing: the suite. Reading the diff helps, and reviewing agent-written Rails is still worth the hour, but a person who can review four hundred lines carefully in an hour is not a check that keeps pace with a process emitting four hundred lines a minute. The suite is the only check that does.

Which makes the suite the thing to be suspicious of. A test written by the same process that wrote the code, in the same pass, from the same reading of what the code should do, inherits that reading's mistakes and then certifies them. Green stops meaning correct and starts meaning consistent.

Why this is suddenly everybody's problem

DHH opened Rails World 2026 in Austin on 23 September. His own post about the keynote says "It's pencils down, people. Writing code by hand is no longer an economically viable skill for most programmers at most companies." The @rails account's summary of the session describes 37signals as having gone pencils down on handwritten code.

The half that the summaries drop: the product announced alongside it, HEY 2.0, is native apps on every platform over a Rust backend, which he said is efficient enough to serve HEY's peak traffic on a Raspberry Pi. The creator of Rails is moving his flagship product's server off Ruby and saying so from the Rails World stage. That is not a small footnote and pretending otherwise would be embarrassing. What DHH actually said takes the keynote apart properly, including the parts that are bad news.

For this page the consequence is narrower, and it survives either reading of the keynote. If an agent writes the implementation, the test is the last artefact standing between the implementation and production. A test the same agent wrote in the same pass is not that artefact. It is a second copy of the same guess.

A test that passes either way

A request spec from a Rails 8.1 codebase, running Ruby 4.0.5, looks like this:

it "renders the current step" do
  get onboarding_path
  expect(response).to have_http_status(:ok)
end

The onboarding flow has three steps and a gate that redirects an un-onboarded user into it. The assertion is true when the controller serves the step the user is actually on. The assertion is equally true when it serves the first step every time, or the last one, or a step belonging to a different user. Replace the lookup of the current step with Onboarding::Flow.first and this example stays green forever.

Two examples below it in the same file do not have that property:

it "breaks the last step out of the turbo frame so completion loads the dashboard full-page" do
  get onboarding_path(Onboarding::Flow.steps.last.to_param)
  expect(response.body).to include('data-turbo-frame="_top"')
end

it "keeps intermediate steps inside the frame" do
  get onboarding_path(Onboarding::Flow.first.to_param)
  expect(response.body).not_to include('data-turbo-frame="_top"')
end

That pair names one behaviour and pins both sides of it. There is exactly one implementation that passes both, and a change to the frame-breaking rule fails one of them within a second.

Model specs have their own version of the weak shape, and it is so idiomatic that it reads as correct:

it "allows a user to be referred only once" do
  existing = create(:referral)
  duplicate = build(:referral, referred: existing.referred)
  expect(duplicate).to be_invalid
end

be_invalid is true when the uniqueness validation fires. It is also true when the factory drifts and the record is missing a required association, when a referrer presence validation fires instead, and when somebody adds an unrelated validation next year. The example describes uniqueness in its name and asserts something much broader in its body, and the name is what everybody reads.

Why the default is the weak one

An agent writes the test after the code and from the code. That ordering is the whole problem: an assertion derived from an implementation cannot disagree with the implementation. Ask for a spec covering a method and you get, in effect, a restatement of the method's branches in RSpec syntax, which will pass for as long as the branches are the branches, correct or not.

There is a second force pushing the same direction. The agent is optimising for a run that comes back green, because that is the signal it is graded on and the signal it iterates against. The cheapest green assertion is always the widest one. be_invalid is cheaper than naming the error attribute. have_http_status(:ok) is cheaper than asserting what came back. Nothing in the loop charges the agent for width.

The position this page takes: the defect is in the ordering, not in the model. A human handed the implementation and asked to write tests for it produces the same failure, which is why test-first was invented by people who had no agents to blame.

What would change my mind is a workflow where the agent never sees the implementation. Give it the acceptance criteria, withhold the diff, and the assertions do get sharper. I have not managed to keep that separation alive across a long session, because the next turn opens the file anyway and the criteria are back in the same context window as the code. If you can hold the line, hold it. Conventions are the context is the cheaper version of the same idea: write down what a correct test looks like here, so the agent's default is already narrow.

The tool that asks the question directly

Mutation testing answers the one question coverage will not: if this code were wrong, would anything fail? A mutation tester edits your source, reruns the tests that cover the edit, and records whether they noticed. An edit that leaves the suite green is an alive mutation, and it is a piece of behaviour nothing is holding in place.

In Ruby that tool is mutant. Its README leads with the question this page is about: "Mutation testing for Ruby. AI writes your code. AI writes your tests. But who tests the tests?" RubyGems shows 0.17.0 released on 17 September 2026, and the README claims support for Ruby 3.2 through 4.0 and Rails 7.2 through 8.1, which is a narrow window that happens to include a current Rails 8.1 app.

The README's quick start is one line:

mutant run --use rspec --usage opensource --require ./lib/person 'Person#adult?'

For anything bigger, the settings move into .mutant.yml, config/mutant.yml or mutant.yml:

---
includes:
  - lib
requires:
  - my_app
environment_variables:
  RAILS_ENV: test
integration:
  name: rspec
  arguments:
    - --fail-fast
    - --seed
    - '0'
    - spec
jobs: 8
mutation:
  timeout: 1.0

Two things about that quick start are worth reading twice. --usage opensource is a licence declaration, not a verbosity flag: mutant is free on public repositories and a paid subscription otherwise, listed on the README at 30 dollars a month or 250 dollars a year per developer. And the README's advice on an alive mutation has two branches, not one: either add the missing test, or delete the original code, because a mutation nothing notices is sometimes a line that was doing nothing.

The prefix match that decides which examples run

mutant does not run your whole suite for every mutation, and the rule it uses instead is where a normal Rails suite falls over. The rspec integration selects examples by the longest matching prefix of example group descriptions. For a subject Foo::Bar#baz it looks for groups described Foo::Bar#baz, then Foo::Bar, then Foo, taking the most specific match that exists.

RSpec.describe Subscription therefore covers every subject on that class, and a nested describe "#live?" narrows to one method. That part works exactly as advertised, and it is why model specs are the easy place to start.

Now look at what a Rails suite calls its request specs:

RSpec.describe "Onboarding", type: :request do
RSpec.describe "Stripe webhooks", type: :request do

Those descriptions are strings, and the strings name features rather than constants. No subject in the application has a name that prefix-matches either of them. So the controller, the form objects and the flow that the onboarding request spec exercises thoroughly are, as far as mutant is concerned, untested, and every mutation inside them survives. The first run on a Rails app produces a wall of alive mutations in code that is genuinely well covered, and the honest reading of that report is that the tool could not find the tests, not that the tests are bad.

The way out is not renaming request specs after controller classes. mutant's rspec documentation provides mutant_expression metadata to map a group onto the subjects it really covers, and mutant: false to exclude groups that are slow or flaky. Both are annotations you add by hand, one group at a time. Across 141 spec files that is the real adoption cost of mutation testing in Rails, and nobody writes it on the tin.

What a mutation pass costs

Numbers first, so the shape is concrete. The suite I have been quoting is 689 examples across 141 spec files, against roughly 7,200 lines of application and lib Ruby. A mutation pass reruns a subset of those examples once per mutation, and mutant generates many mutations per method: negating a condition, swapping a boolean, deleting an argument, replacing a literal, removing a method call entirely.

I have not run a full pass over that suite, so I am not going to quote you a wall-clock figure I did not measure. The arithmetic is the honest answer: total time is mutations times the runtime of the selected examples, divided by jobs. That multiplier is large enough that mutant's own documentation concedes the point. It recommends the full pass on CI "as long as feasible" and switching to incremental mode "when the CI cycle time gets too annoying", with full passes kept as a nightly job.

Incremental mode is the --since flag:

mutant --since master 'ProjectNamespace*'

The flag filters subjects down to those with a line changed since a git reference, using git diff semantics, which is what makes a mutation run fit inside a pull request. The documented caveat matters for agent-written changes specifically: mutant detects direct source changes only. A subject whose behaviour changed because a constant it reads was edited elsewhere is not selected, and a large agent-authored refactor touching a shared value is exactly the change that slips through.

There is a cost that is not measured in minutes, and I think it is the larger one. Hand a surviving mutation back to an agent and ask it to fix the test, and you reliably get an example that kills that mutation and nothing adjacent to it. The mutation report becomes the spec, the suite fills with assertions shaped like defects rather than like behaviour, and the coverage number that improves is mutation score. Surviving mutations are a prompt for a person to ask what the behaviour was supposed to be. Feeding them straight to the thing that wrote the weak test is a loop with nobody in it.

Coverage is the wrong number to raise

Anybody looking for a rails test coverage agent workflow is usually looking for a percentage to put in CI, and line coverage is the worst number available for this particular problem. Line coverage answers whether a line executed. A test that calls a method and asserts nothing at all executes every line in it. Both vacuous examples earlier in this page contribute full line coverage for the code they fail to check, which is the whole point: the number cannot see the assertion.

SimpleCov's enable_coverage :branch raises the bar to branches rather than lines, and it is a real improvement, because an untaken else is now visible. It still does not look at a single expectation.

The application these examples come from ships no coverage gem at all. Its test group is rspec-rails, factory_bot_rails, faker, shoulda-matchers, plus prosopite and pg_query for N+1 detection, and there is no SimpleCov in the Gemfile. That is a defensible position and I hold it: adding SimpleCov would not have flagged one of the weak assertions above, and a percentage in CI creates pressure to write the kind of test that raises it, which is precisely the wide, assertionless kind. What would change my mind is branch coverage used the way mutant's selection config uses a coverage report, as an input for choosing which tests to run rather than as a target to hit.

Writing the assertion that survives

The shape to copy is in this example, which pins an idempotency guard on Stripe webhook events:

it "does not yield again for the same event id" do
  described_class.process_once(id: "evt_1", type: "payment_intent.succeeded") { nil }
  yielded = false
  expect do
    described_class.process_once(id: "evt_1", type: "payment_intent.succeeded") { yielded = true }
  end.not_to change(described_class, :count)
  expect(yielded).to be(false)
end

Two assertions, on two different observable consequences of one rule, and they fail for different reasons. Break the guard so a duplicate event inserts a row and the count assertion fails. Weaken it so the row is deduplicated but the block runs anyway, which is the version that double-charges a customer, and only the second assertion notices. One assertion here would have left half the rule unheld.

Four habits produce assertions in that shape, and they are worth writing down somewhere the agent reads:

1. Assert the value, not its truthiness. expect(plans.map(&:key)).to eq(%i[starter pro business]) fails when the order changes; expect(plans).to be_present never fails at all.
2. Assert the negative beside the positive, in the same file. A rule with only its happy path pinned is a rule with one side nailed down and the other swinging.
3. Name the attribute in a validation failure. expect(duplicate.errors[:referred_id]).to be_present says which rule fired; be_invalid says only that something did.
4. Pin the boundary, not a point in the middle of the range. A test for a limit of 3 that passes 1 survives every mutation of the limit except deletion.

An AGENTS.md for Rails is where those four sentences belong, because an agent that reads them before writing the spec is cheaper than a mutation run that finds the same four problems afterwards. Token-efficient Rails is the same argument about the implementation: the suite and the conventions are both ways of telling the agent what correct means without spending a human on it.

What this page does not cover

Minitest and Test::Unit. mutant integrates with both and the selection rules differ from the rspec prefix match, and I have run neither, so describing them here would be describing documentation rather than experience.

Whether an agent should run mutant itself as part of its own loop. The tool is a CLI with an exit status, so wiring it up is trivial and that is not the question. The question is whether an agent optimising against a mutation score writes better tests or just narrower ones, and the paragraph above about feeding surviving mutations back is my current guess rather than a result.

System tests and anything with a browser in it. Every technique here assumes an example that runs in under a second, because mutation testing multiplies whatever that number is. When Rails is still the answer covers the stack choice that makes fast tests possible in the first place, which is upstream of all of this.