Rails testing tools, priced by what each one can see

A Rails application has more testing machinery in it than anybody uses, and the question people bring to it is which Ruby on Rails testing framework to install, which is the decision that matters least. What matters is what each tool can see, because a defect is only catchable by something that boots enough of the application to reach it, and most arguments about Rails testing tools are conducted without that number in front of anybody.

This page is the layer above three others. RSpec vs Minitest in Rails settles which DSL, Rails system tests prices the browser, and FactoryBot covers where test data comes from. None of them answers what to reach for when you have a bug to catch.

Everything below was run on this machine: Ruby 4.0.5 on arm64-darwin25 with 12 cores, PostgreSQL 17.7 on port 15432, two real Rails 8.1.3.1 applications, and a throwaway app generated today, which came out at Rails 8.1.4. Gem versions are rspec-rails 8.0.4, brakeman 8.0.6, bundler-audit 0.9.3, capybara 3.40.0, selenium-webdriver 4.49.0, prosopite 2.2.0, minitest 6.0.6.

Eight base classes, and the choice between them is a boot decision

Rails 8.1 defines eight test case classes, and one bin/rails runner loaded all of them here without a gem beyond the framework:

%w[ActiveSupport::TestCase ActiveJob::TestCase ActionMailer::TestCase ActionView::TestCase
   ActionDispatch::IntegrationTest ActionDispatch::SystemTestCase ActionCable::TestCase
   ActionMailbox::TestCase].each { |k| puts "#{k} #{Object.const_get(k) ? "OK" : ""}" }

ActiveSupport::TestCase OK
ActiveJob::TestCase OK
ActionMailer::TestCase OK
ActionView::TestCase OK
ActionDispatch::IntegrationTest OK
ActionDispatch::SystemTestCase OK
ActionCable::TestCase OK
ActionMailbox::TestCase OK

Picking one of those is not a stylistic choice, it is a decision about how much of the application starts up before your assertion runs. ActiveSupport::TestCase gives you a database connection and 60 assert_* methods and nothing else. ActionDispatch::IntegrationTest adds routing, the full middleware stack, the controller and the template. ActionDispatch::SystemTestCase adds Puma, chromedriver and a browser process.

If you write RSpec, you are still using those classes. rspec-rails lib/rspec/rails/example/request_example_group.rb:13 reads include ActionDispatch::IntegrationTest::Behavior. A request spec and a Rails integration test are the same object with two vocabularies in front of it, which is why "integration testing" means something narrower in Rails than it does in the industry generally: in Rails it means one process, one HTTP call, no browser.

The ladder, priced on a suite that exists

This site's own RSpec suite is 200 *_spec.rb files and 15,374 lines of them, and it runs single process on the machine described above: 1665 examples in 28.2 s. Split by directory:

Directory	Examples	Wall	Per example
spec/lib	101	0.278 s	2.7 ms
spec/models	408	2.51 s	6.2 ms
spec/repository	22	0.221 s	10.0 ms
spec/services	178	3.86 s	21.7 ms
spec/requests	800	18.86 s	23.6 ms

Request specs are 48 percent of the examples and 67 percent of the clock. That ratio is the whole economics of a Rails suite, and it is the number to look at before anybody proposes a faster CI runner. The boilerplate product repository runs the same shape: 851 examples across 142 spec files in 19.51 s today, up from the 846 recorded two days ago.

The floor underneath every one of those numbers is the boot. bundle exec ruby -e 'require "./config/environment"' takes 0.81 s here. An rspec invocation with a filter that matches nothing takes 1.07 s of wall clock and runs zero examples. So spec/lib, all 101 examples of it, is free: the entire directory costs less than a third of a second against a 1.07 s fixed price you have already paid. The conclusion people draw from that is usually wrong. Moving logic into lib/ to get faster tests saves 3.5 ms per example and nothing else, because you still paid the second.

The rung above the table is the browser, at roughly 130 ms per test against 4.6 ms for the same assertions as integration tests. That measurement and the reasons for it are in Rails system tests and are not repeated here.

"Unit testing" names a thing Rails does not have

Rails ships no unit test class. ActiveSupport::TestCase is the base of all seven of the others, and what a Rails team calls a unit test is a model test, which opens a PostgreSQL connection, wraps itself in a transaction, and rolls back. The 6.2 ms per model example above is mostly that.

The genuinely unit-shaped tests in this repository are the 101 in spec/lib, which test plain Ruby objects and touch no connection, at 2.7 ms. The difference is a factor of 2.3 and it is invisible next to the 1.07 s boot. Which is the useful version of the advice: separating pure logic from Active Record is worth doing for reasons of design, and the test speed argument for it does not survive being measured.

What only a request spec can see

N+1 queries are the clearest case of a defect that exists at exactly one layer. This site detects them with prosopite, wired in config/environments/test.rb:34:

require "prosopite/middleware/rack"
config.middleware.use(Prosopite::Middleware::Rack)
config.after_initialize { Prosopite.raise = true }

Rack middleware. It scans one HTTP request at a time, which means it sees a controller and a template and does not see your factory setup. A model spec cannot trigger it, a service spec cannot trigger it, and a system test triggers it in a different process. Driven by hand in a scratch example against the real models here, it produces:

RAISED Prosopite::NPlusOneQueriesError
N+1 queries detected (0.6ms):
  SELECT "leads".* FROM "leads" WHERE "leads"."id" = $1 LIMIT $2
  SELECT "leads".* FROM "leads" WHERE "leads"."id" = $1 LIMIT $2
  SELECT "leads".* FROM "leads" WHERE "leads"."id" = $1 LIMIT $2

Three identical selects and a raise. The comment above that middleware line in the repository says what the placement buys: "The middleware scans each HTTP request (not the factory setup), so an N+1 in a controller/view raises and fails the request spec." That is the argument for having more than one kind of test, and it is not the argument usually given: it is not about confidence levels or pyramids, it is that some instruments are only installed at one altitude.

One detail if you go looking: Prosopite.raise is a writer with no matching reader, because Prosopite.raise resolves to Kernel#raise. Reading the flag back needs Prosopite.instance_variable_get(:@raise), which returned true.

bin/ci, which is the framework's own answer

Rails 8.1 shipped a CI runner, so automated testing in a Rails application now has a default shape and a file to argue about. The railties changelog entry reads "Introduce bin/ci for running your tests, style checks, and security audits locally or in the cloud", credited to Jeremy Daer and DHH, and it generates two files. bin/ci is five lines. config/ci.rb is the list. Generated today by rails new:

CI.run do
  step "Setup", "bin/setup --skip-server"
  step "Style: Ruby", "bin/rubocop"
  step "Security: Gem audit", "bin/bundler-audit"
  step "Security: Brakeman code analysis", "bin/brakeman --quiet --no-pager --exit-on-warn --exit-on-error"
  step "Tests: Rails", "bin/rails test"
  step "Tests: Seeds", "env RAILS_ENV=test bin/rails db:seed:replant"
end

The Tests: Seeds step is the one worth stealing if you take nothing else from the file. db:seed:replant truncates and reseeds the test database, which turns db/seeds.rb into something the build breaks on. Seeds rot silently otherwise: nothing else in a Rails test run ever executes them.

The engine is ActiveSupport::ContinuousIntegration, 145 lines in activesupport 8.1.3.1. It sets ENV["CI"] = "true", times each step, prints a green tick or a red cross with the elapsed time, and aggregates. On the empty generated app the whole run is 7.85 s. On this site, whose config/ci.rb has been edited to add an importmap audit, bin/rails zeitwerk:check and bundle exec rspec in place of bin/rails test, it is 48.10 s:

✅ Setup passed in 5.85s
✅ Style: Ruby passed in 1.84s                      (558 files inspected, no offenses detected)
❌ Security: Gem audit failed in 0.39s
✅ Security: Importmap vulnerability audit passed in 1.06s
✅ Security: Brakeman code analysis passed in 3.62s
✅ Autoloading: Zeitwerk check passed in 1.16s
✅ Tests: Prepare database passed in 1.44s
❌ Tests: RSpec failed in 32.75s
❌ Continuous Integration failed in 48.10s

bin/ci keeps going after a step fails

ActiveSupport::ContinuousIntegration#step has a two-line body and the second line is report(title) { results << system(*command) }. system returns false on a non-zero exit, the false goes into an array, and the array is checked once at the end by success?. No set -e, no early return.

Proved rather than read: adding step "After the failure", "echo I still ran" immediately below a failing test step gave

❌ Tests: Rails failed in 1.87s
I still ran
✅ After the failure passed in 0.00s
✅ Tests: Seeds passed in 0.56s
❌ Continuous Integration failed in 7.02s

and echo $? printed 1.

The cost is real and you should decide about it rather than discover it. A failed Setup step means every later step runs against an application whose dependencies did not install, so one broken Gemfile produces seven red steps and you read all of them to find the first. What you buy is the other case, which is more common: a run where RuboCop and Brakeman and the suite each have one thing wrong, and you see all three in one pass instead of three pushes.

The security steps are two tools, and neither one is a penetration test

Brakeman reads your source. Run here with --summary it reports the scope precisely:

Rails Version: 8.1.3.1
Brakeman Version: 8.0.6
Duration: 1.880541 seconds
Controllers: 69
Models: 76
Templates: 215
Security Warnings: 0
Ignored Warnings: 1

79 named checks, listed in full in that output, covering SQL, CrossSiteScripting, Redirect, MassAssignment, UnsafeReflection, SessionSettings, EOLRails and seventy-two others. Static analysis over a parse tree, in under two seconds, with no database and no running server.

bundler-audit reads your lockfile. It diffs the gem versions in Gemfile.lock against ruby-advisory-db, which on this machine reported "1247 advisories, last updated 2026-09-23 19:47:20 -0400, commit fb34fedf8a96f99e54bcfb9306519996a70baa25". The whole check is 0.39 s.

Neither is a penetration test, and a Rails application ships nothing that is. A penetration test is somebody with an intent sending traffic at a running instance: session fixation attempts, forced browsing at object ids, header injection, timing the login form. The closest thing in the box is ActionDispatch::IntegrationTest, which sends real requests through the real middleware stack and is where you write the adversarial cases yourself. The next section is what that means in practice.

The hole Brakeman scores clean

An ArticlesController in the throwaway app, a Rails 8.1 scaffold trimmed to two actions with one line of ownership added to index:

class ArticlesController < ApplicationController
  before_action :set_article, only: %i[ show update ]

  def index
    @articles = Article.where(user_id: current_user_id)
  end

  def update
    if @article.update(article_params)
      redirect_to @article
    else
      render :edit, status: :unprocessable_content
    end
  end

  private
    def current_user_id
      session[:user_id]
    end

    def set_article
      @article = Article.find(params[:id])
    end
end

index scopes. set_article does not. Any signed-in session can PATCH any article in the table. Brakeman 8.0.6 over that application: Controllers: 2, Models: 2, Templates: 8, Security Warnings: 0, then No warnings found.

Seven lines of integration test find it:

test "a signed in user cannot update somebody else's article" do
  get "/login/1"
  theirs = articles(:theirs)
  patch article_url(theirs), params: { article: { title: "taken over" } }
  assert_response :not_found
  assert_equal "Theirs", theirs.reload.title
end

status: 302, title now: "taken over"
Expected response to be a <404: missing>, but was a <302: Found>

Add a genuine injection to the same file and Brakeman does its job immediately:

def search
  @articles = Article.where("title = '#{params[:q]}'")
end

Confidence: High
Category: SQL Injection
Check: SQL
Message: Possible SQL injection
Code: Article.where("title = '#{params[:q]}'")
File: app/controllers/articles_controller.rb
Line: 9

The line between those two results is worth stating plainly, because it decides what you have to write by hand. Brakeman recognises dangerous shapes. Missing authorisation has no shape: a scoped current_user.articles.find and an unscoped Article.find are the same expression with a different receiver, and no parse tree tells you which one this application intended. Every authorisation rule in your app is yours to test, the test is an integration test, and the assertion that matters is the negative one.

The dead end: a green test that proved nothing

The first version of the ownership test above passed. assert_response :not_found was green against a controller with no authorisation in it at all, and stopping there would have shipped a test that asserts a 404 arrives and never notices when a 302 replaces it.

log/test.log had the reason:

AbstractController::ActionNotFound (The destroy action could not be found for the :set_article
callback on ArticlesController, but it is listed in the controller's :only option.

The scaffold had been trimmed down to show and update with destroy left behind in the only: list. raise_on_missing_callback_actions has been a default since Rails 7.1 and it raises on the first request that touches the controller. In the test environment config.action_dispatch.show_exceptions = :rescuable turns that raise into a rendered 404, which is exactly what the assertion was looking for.

Two things follow. A status-code assertion is the weakest assertion in an integration test, because every misconfiguration in the stack arrives dressed as a status code. And a test asserting a failure should be run once against code that ought to pass it: green on the broken version is not a passing test, it is a test that has not been read.

Two ignore files, one gem audit, two answers

On this repository, today:

$ bundle exec bundle-audit check
No vulnerabilities found

$ bin/bundler-audit
Name: ruby_llm
Version: 1.16.0
CVE: CVE-2026-67991
GHSA: GHSA-42r3-x6vx-x49x
Criticality: High
Title: Polynomial-Time Regular Expression Denial of Service (ReDoS) vulnerability
Solution: update to '>= 2.0.0.rc1'

Vulnerabilities found!

Same bundler-audit 0.9.3, same Gemfile.lock, same advisory database. The difference is one flag. bin/bundler-audit is the Rails 8.1 template verbatim:

ARGV.concat %w[ --config config/bundler-audit.yml ] if ARGV.empty? || ARGV.include?("check")

and bundler-audit's own default, in cli.rb, is method_option :config, type: :string, aliases: '-c', default: '.bundler-audit.yml'. Two different filenames. This repository's accepted-risk list lives in .bundler-audit.yml, where the CVE-2026-67991 entry carries thirty lines explaining why the only published fix is a release candidate of a major version that renames two tables. config/bundler-audit.yml is the generated stub and its entire ignore list is CVE-THAT-DOES-NOT-APPLY.

So the audit that runs in bin/ci cannot see any exception this project has ever recorded, and bin/ci on this site fails on an advisory the team decided about in writing and committed. The bug is this repository's, not the framework's, and it is the failure mode to check for in any application older than Rails 8.1 that has since adopted bin/ci: your exceptions are in one file and your gate reads the other. Both filenames are plausible and neither tool warns that the other file exists.

Coverage is in the standard library

No gem is needed to measure line coverage in a Rails application. Ruby's Coverage module, required early enough through RUBYOPT so that it is running before config/environment loads anything, produces the numbers directly. A 15-line prelude over this suite reported:

app/ files instrumented: 220
relevant lines: 4023, executed: 3939, 97.9%
  helpers/application_helper.rb: 7 of 43 lines never ran
  controllers/sessions/omniauth_controller.rb: 5 of 29 lines never ran
  services/analytics/report.rb: 5 of 129 lines never ran

The cost is 6.6 s on a 28.2 s run, 34.86 s against 28.2 s wall. SimpleCov is a reporting layer on top of this, and it is worth its dependency for the HTML output and the per-group thresholds, not for the measurement.

What 97.9 percent is worth is a separate question, and the answer is less than it looks: line coverage records that a line executed, not that any assertion would notice if it changed. Testing what an agent wrote takes that apart with mutant against this same code.

The test type nobody puts in the list

spec/repository/ holds 22 examples that run in 0.221 s and assert nothing about the application. They read the repository as text: that every image in a content file resolves through the asset pipeline, that every image has alt text long enough to describe what it shows, that every published article has a social card committed, and that no em-dash or en-dash has reappeared anywhere in a tracked file.

Nothing in Rails suggests writing these and nothing stops you. They are cheap, because they need no database and no requests, and they catch the class of mistake that no amount of application testing reaches: a rule the team agreed on in a document and then stopped following.

What is announced and not here yet

herb:check compiles every HTML+ERB template through Herb and exits non-zero on the ones it rejects, announced in This Week in Rails on 2026-09-18 under rails/rails pull request 58770. It is not in railties 8.1.3.1: lib/rails/tasks/ contains eight rake files and none of them is herb, and bin/rails -T | grep -i herb returns nothing here. A template that raises at render time is currently caught by a view spec, a request spec or a user, in that order of preference.

The call, and what would change it

Reach for the cheapest layer that can see the defect, and know which layer that is before you write anything. Business rules and validations go in model specs at 6.2 ms. Anything about who is allowed to do what goes in an integration test, because authorisation has no shape a scanner recognises and the assertion has to be negative. N+1 queries and middleware behaviour go in request specs, because that is where the instruments are installed. The browser is for Stimulus, Turbo and things that are invisible to the server, at 130 ms and a flake budget.

Run bin/ci rather than rspec locally, and edit config/ci.rb until the local gate and the remote gate are the same list. Every project eventually grows a "but CI also runs X" problem, and the Rails 8.1 answer to it is a 20-line Ruby file that a person can read.

What would change this: a Rails release that makes system tests cheap enough to displace request specs. Most of the 130 ms is process startup and CDP round trips rather than anything Rails controls, so this is not close.

What this post does not cover

Which DSL to write the tests in, which is RSpec vs Minitest. Where the test data comes from, which is FactoryBot and Rails fixtures vs factories. What a browser test costs and how it goes flaky, which is Rails system tests. Contract testing against third-party APIs, which needs WebMock or VCR and is a gem decision, not a Rails one. Load testing: ApacheBench and k6 are outside the framework entirely and the numbers depend on your hosting, not your code. Parallel test runners, which are measured in the RSpec versus Minitest piece. And actual penetration testing, which is a person, a scope document and an engagement, not a rake task.