Rails credentials, one file per environment

A deploy dies on this line:

Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit` (ArgumentError)

Nothing is wrong with secret_key_base. It is sitting in the encrypted file, where it has been since the app was generated. What is wrong is that the container has no key to decrypt that file with, so Rails read an empty credential hash, found no secret_key_base in it, and reported the consequence instead of the cause. Everything below was run against Rails 8.1.4 on Ruby 4.0.5, with every quoted line copied out of the installed gems.

The file, the key, and nothing else

Two artifacts carry the whole feature. config/credentials.yml.enc is YAML encrypted with aes-128-gcm (activesupport-8.1.4/lib/active_support/encrypted_file.rb:29) and base64'd, with the initialization vector and the auth tag appended after -- separators. config/master.key is 32 hex characters on one line, no trailing newline that matters, mode 0600, git-ignored by the generated .gitignore.

Thirty-two hex characters is sixteen bytes, which is the key length aes-128-gcm wants. EncryptedFile.generate_key is SecureRandom.hex(MessageEncryptor.key_len(CIPHER)) and returns exactly that. bin/rails secret returns 128 characters, and people paste it into RAILS_MASTER_KEY constantly, because both are "the secret". Here is what that costs:

$ RAILS_MASTER_KEY=$(bin/rails secret) bin/rails runner 'p Rails.application.credentials.secret_key_base'
message_encryptor.rb:307:in 'OpenSSL::Cipher#key=': key must be 16 bytes (ArgumentError)

        cipher.key = @secret

Not a Rails error. A raw OpenSSL one, from inside the cipher, because check_key_length guards encrypt and not decrypt (encrypted_file.rb:103 against :108). A key of the wrong length is caught when you write the file and not when you read it. The command that hands you a correct key without opening an editor is bin/rails runner 'puts ActiveSupport::EncryptedConfiguration.generate_key'.

Where Rails looks, and the two lookups that drift apart

credentials_defaults is nine lines and it decides everything (railties-8.1.4/lib/rails/application/configuration.rb:643):

def credentials_defaults
  content_path = root.join("config/credentials/#{Rails.env}.yml.enc")
  content_path = root.join("config/credentials.yml.enc") if !content_path.exist?

  key_path = root.join("config/credentials/#{Rails.env}.key")
  key_path = root.join("config/master.key") if !key_path.exist?

  { content_path: content_path, key_path: key_path }
end

Read it twice, because the two fallbacks are independent. The content path falls back on its own existence check, the key path falls back on its own, and nothing anywhere asserts that the file and the key belong together. Deleting config/credentials/production.key while config/credentials/production.yml.enc stays in place does not produce a missing-key error. It produces a wrong-key error, because config/master.key exists and Rails will cheerfully try it:

$ RAILS_ENV=production bin/rails runner 'puts 1'
codec.rb:57: AEAD authentication tag verification failed (ActiveSupport::MessageEncryptor::InvalidMessage)
    from encrypted_file.rb:109:in 'ActiveSupport::EncryptedFile#decrypt'

The environment variable name is not per-environment either. env_key: "RAILS_MASTER_KEY" is a hardcoded default at lib/rails/application.rb:512; there is no RAILS_PRODUCTION_KEY. Production reads config/credentials/production.yml.enc and expects RAILS_MASTER_KEY to hold the contents of production.key. Kamal's generated .kamal/secrets does RAILS_MASTER_KEY=$(cat config/master.key) out of the box, which is right for a single shared file and wrong the day you add a per-environment one, and nothing warns you. Deploying Rails with Kamal 2 covers the rest of that file.

A per-environment file replaces the shared one

The most expensive misconception about credentials per environment is that the two files layer, so you can keep shared values in config/credentials.yml.enc and override a few in config/credentials/production.yml.enc. They do not layer. The first branch of credentials_defaults picks one path and the shared file is then never opened.

Proven on a generated app whose shared file still had the commented-out aws block and a real secret_key_base:

$ EDITOR=... bin/rails credentials:edit --environment production
Adding config/credentials/production.key to store the encryption key: d03e5acac0fe6f7343ef360efb30e7ac

$ RAILS_ENV=production bin/rails runner 'p Rails.application.credentials.dig(:stripe, :secret_key); p Rails.application.credentials.aws'
"sk_live_PROD"
nil

Development, meanwhile, still reads config/credentials.yml.enc and returns nil for the Stripe key. Two files, two disjoint worlds, one command to create the second one.

The one kindness Rails does here is that credentials:edit --environment production generates a fresh secret_key_base into the new file, because skip_secret_key_base is true only for development and test (credentials_command.rb:111). That saves the naive case. It does not save the case where you open the editor, select all, and paste your keys over the generated content: the new file loses its secret_key_base and production stops booting, with the error at the top of this page. Append to that buffer, never replace it.

credentials:show resolves differently from credentials:edit

Both commands take --environment. Only one of them honours it as an instruction rather than a hint. In credentials_command.rb, edit forces the paths (:22):

if environment_specified?
  @content_path = "config/credentials/#{environment}.yml.enc" unless config.overridden?(:content_path)
  @key_path = "config/credentials/#{environment}.key" unless config.overridden?(:key_path)
end

show has no such block (:35). It boots the app with RAILS_ENV set and then reads whatever credentials_defaults resolved, which means it goes through the fallback. On an app with no per-environment production file, bin/rails credentials:show --environment production prints the contents of the shared config/credentials.yml.enc with no indication that it did so. Run before and after creating the production file, the same command answers from two different files.

That is not a bug and it is the right behaviour for show, since the fallback is what production will actually read. It is still the single fastest way to convince yourself you have configured production when you have configured everybody. When the answer matters, print the path: bin/rails runner 'puts Rails.application.config.credentials.content_path' with RAILS_ENV set.

The missing key in production

Now the failure from the opening, with its mechanism. config.require_master_key defaults to false (configuration.rb:77) and is passed straight through as raise_if_missing_key: config.require_master_key at application.rb:517. With it false, handle_missing_key returns nil instead of raising (encrypted_file.rb:126). Then EncryptedFile#read hits this:

def read
  if !key.nil? && content_path.exist?
    decrypt content_path.binread.strip
  else
    raise MissingContentError, content_path
  end
end

A nil key takes the same branch as a missing file. EncryptedConfiguration#read rescues MissingContentError and returns "" (encrypted_configuration.rb:62), with the comment "Allow a config to be started without a file present". So the credential hash is empty, every lookup returns nil, and the first thing that notices is secret_key_base=, which raises the ArgumentError about secret_key_base at configuration.rb:543.

If you supply SECRET_KEY_BASE as an environment variable, as a lot of Heroku apps do, even that tripwire goes. The app boots, Rails.application.credentials.stripe is nil, and the failure surfaces hours later as an unauthenticated Stripe call.

One line fixes the diagnosis for good:

# config/environments/production.rb
config.require_master_key = true

$ RAILS_ENV=production bin/rails runner 'puts 1'
Missing encryption key to decrypt file with. Ask your team for your master key and write it to
/absolute/path/to/app/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

That is the true sentence, and it costs a refusal to boot without a key, which is what you wanted. Turn it on in production and leave it off in development, where a fresh clone with no key should still start.

One legitimate keyless boot survives that switch, and the generated Dockerfile relies on it: RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile. That variable is checked before anything else in secret_key_base (configuration.rb:525) and routes to generate_local_secret, which writes a throwaway hex string to tmp/local_secret.txt. Asset compilation needs a booted app and no real secret, so the image builds without ever seeing your key.

A credential or an environment variable

Both are readable at boot, so the choice is about where the value lives and what it costs to change. A credential lives in git, versioned with the code that reads it, reviewable in a pull request, impossible to lose when a platform dashboard is wiped, and identical on every machine that has the key. An environment variable lives with the platform, changes without a commit, and can differ per container without anybody noticing.

The rule that has held up: exactly one secret is an environment variable, RAILS_MASTER_KEY, and every other secret is a credential. Everything non-secret and deploy-shaped stays an environment variable, because DATABASE_URL, RAILS_LOG_LEVEL and a feature flag are not secrets and should not require a deploy to change. The precedence rules agree with that split. secret_key_base reads ENV["SECRET_KEY_BASE"] || Rails.application.credentials.secret_key_base (configuration.rb:528), and the key itself is read_env_key || read_key_file (encrypted_file.rb:53), environment first in both.

Environment-first on the key has a sharp edge. A RAILS_MASTER_KEY left over in your shell from last week shadows a perfectly correct config/master.key, in development, where you are not looking for it:

$ RAILS_MASTER_KEY=$(ruby -rsecurerandom -e 'print SecureRandom.hex(16)') bin/rails runner 'puts Rails.application.credentials.secret_key_base'
AEAD authentication tag verification failed (ActiveSupport::MessageEncryptor::InvalidMessage)

The bill for credentials is rotation. Changing an SMTP password means editing config/credentials/production.yml.enc, committing it and deploying, which you cannot do from a phone at midnight and cannot delegate to somebody without repository write access. That cost is named again in Action Mailer in production, which takes the same position for the same reason. What would change it: a team large enough that a rotation blocking on CI is a real outage, or an audit requirement to log who read which secret when. At that point a secrets manager wins and the master key becomes the thing it fetches.

Making an encrypted file reviewable

An encrypted blob in a diff is a few hundred characters of base64 that changed from end to end because one character changed. Rails ships a fix that almost nobody turns on:

$ bin/rails credentials:diff --enroll
Enrolled project in credentials file diffing!

That appends two lines to .gitattributes pointing both credential paths at a rails_credentials diff driver, and the next credentials:edit registers the driver in git config. After which git diff on the encrypted file shows the plaintext change:

--- a/config/credentials/production.yml.enc
+++ b/config/credentials/production.yml.enc
@@ -11,3 +11,4 @@ secret_key_base: e4f1ebf9e72568ade2de1737...

 stripe:
   secret_key: sk_live_PROD
+  publishable_key: pk_live_ZZZ

Here is the part that did not work. On this machine the first attempt gave fatal: unable to read files to diff, and the enrollment had been reported as successful. The cause is at diffing.rb:42, where diffing_driver_configured? runs git config --get diff.rails_credentials.textconv with no --local, so it searches global config too. A stale ~/.gitconfig entry from an older Rails convention, rails encrypted:show, answered that check, Rails skipped configuring the correct driver, and git then invoked a command that cannot work without a path argument. git config --show-origin --get diff.rails_credentials.textconv names the offending file in one line, and git config --local diff.rails_credentials.textconv 'bin/rails credentials:diff' ends it.

One warning that is not Rails' fault: the plaintext now lands in your terminal scrollback, your pager, and whatever your editor does with diffs. Enroll on a laptop, not on a shared screen.

Reading one from application code

Rails.application.credentials is an ActiveSupport::EncryptedConfiguration, which behaves like OrderedOptions. Four readers, three behaviours:

credentials.nope                  # => nil
credentials.dig(:nope, :deeper)   # => nil
credentials.nope!                 # => KeyError: :nope is blank
credentials.fetch(:nope)          # => KeyError: key not found: :nope

The bang form and fetch are the ones worth reaching for at boot for anything the app genuinely cannot run without, because a nil that travels is a nil you debug in a controller.

Memoization is the other thing to know. Rails.application.credentials is @credentials ||= (application.rb:494) and EncryptedConfiguration#config is @config ||=. Writing a new value to the encrypted file from inside a running process does not change what that process sees:

["before", nil]
["on disk now", "sk_test_WRITTEN_JUST_NOW"]
["same process, Rails.application.credentials", nil]

A fresh bin/rails runner on the same file printed "sk_test_WRITTEN_JUST_NOW". Credentials are a boot-time read, every time.

Scattering Rails.application.credentials.dig(:stripe, :secret_key) across an app is the usual next mistake, because the key names become untyped strings in thirty places and a rename is a grep. One module per app, with a method per secret, makes the whole credential surface one file you can read top to bottom.

What the boilerplate's wizard writes

LaunchKit ships with no keys and no *.yml.enc at all, which is deliberate: a buyer should not inherit somebody else's secrets or receive a file they cannot decrypt. bin/setup runs bin/rails credentials:bootstrap (lib/tasks/credentials.rake), which reads config/credentials.yml.example as a structure reference, recursively blanks every leaf, and writes a config/credentials/<env>.key plus <env>.yml.enc for every file found under config/environments, each with its own generated secret_key_base. It is idempotent: an environment that already has both files is skipped, so re-running bin/setup never overwrites what you filled in.

Blank leaves are the point. /admin/setup reads that environment's encrypted file through Setup::Credentials, lists every blank key as a form field, and writes back with ActiveSupport::EncryptedConfiguration#write. There is an environment switcher, so you fill production's credentials from your laptop, where you hold the key, and then commit. The wizard is gated to local environments outright, which is the correct call: a deployed admin console that can rewrite secrets at runtime is a much worse idea than a slightly annoying deploy loop.

The detail that shows somebody hit this in anger is the Stripe test button. It refuses to report success when the submitted key differs from the one the running server booted with: "Saved keys don't match the running server. Restart the server, then test the Stripe connection again." That guard exists because of the memoization above. Without it the wizard would test a key the application is not using and print a green check.

Not covered here

Active Record Encryption is a different feature with different keys (active_record_encryption in credentials, three of them) and none of the resolution rules above apply to it. Arbitrary encrypted files through bin/rails encrypted:edit config/whatever.yml.enc use the same EncryptedFile class and are worth knowing about, but they are not credentials. Rotating a master key, which means decrypting with the old one and re-encrypting with the new one while both are live, deserves its own page. And every hosted secrets manager is out of scope, including the interesting question of whether RAILS_MASTER_KEY should be fetched from one at container start.