Two ways an LLM call fails, and why one fallback does not cover both

Two things end an LLM call early: the provider is unhappy, or you are out of money. Both end with "use the other model", which is why they get written as one branch and then behave wrong. The budget is knowable before the call. The outage is only knowable after it.

The fallback that happens before the call

Model selection runs first, with no network involved. Ai::ModelPicker asks one question: is there a backup configured, and has the primary model's monthly quota been used up?

def call
  if @config.backup? && Ai::ModelQuota.new(@config).exceeded?
    backup
  else
    [ @config.model, @config.provider ]
  end
end

backup? requires both a backup model and a backup provider to be set, so a half-filled configuration behaves as no backup rather than as a provider mismatch discovered at request time. When there is no backup and the quota is gone, the picker still returns the primary model: the job will call it, the provider will bill for it, and the quota turns out to be a dashboard number rather than a limit. That is a deliberate choice and it is worth knowing before you rely on the field.

Ai::ModelQuota#used counts chats, not tokens:

def used
  Chat.this_month.joins(:model).where(ruby_llm_models: { model_id: @config.model }).count
end

A completion is the unit, which keeps the number legible in an admin screen and makes it wrong for anyone whose prompts vary wildly in size. Counting tokens would be more accurate and less comprehensible, and this is the kind of trade a quota field should state out loud.

Zero means unlimited

A Rails AI quota is an integer column, and empty means "no limit", which in practice arrives as zero. So unlimited? is quota.zero?, and exceeded? is !unlimited? && used >= quota.

The consequence is the one nobody expects: setting the quota to zero does not stop completions, it removes the limit entirely. Somebody trying to pause AI spending by typing 0 into the admin gets the opposite of what they meant. The right lever for that is the feature flag, which makes the whole ai module answer 404, and the quota field is for budgeting rather than for switching off.

remaining returns Float::INFINITY in the unlimited case rather than a large integer, which keeps the comparison honest at the cost of a value the progress bar has to special-case. percent_used returns 0 when unlimited, because a bar filling toward a limit that does not exist is a lie drawn in CSS.

The fallback that happens inside the rescue

The second fallback cannot be decided in advance, because the information arrives as an exception. The job names exactly four classes:

RETRYABLE = [ RubyLLM::RateLimitError, RubyLLM::ServiceUnavailableError, RubyLLM::OverloadedError, RubyLLM::ServerError ].freeze

Rate limited, unavailable, overloaded, server error. All four mean the request was reasonable and the provider could not serve it, which is exactly when a second provider helps.

What is absent matters more. RubyLLM::UnauthorizedError and a malformed request are not in the list, and raise unless config.backup? re-raises anything that is not retryable. An invalid API key that quietly fell back to a backup model would look like a working feature with a strange accent, and the misconfiguration would surface weeks later as a bill from a provider nobody meant to use.

For the same reason the backup is worth putting on a different provider. A backup model hosted by the primary's provider shares its outage, its rate limit and its status page, so the failover fires and fails on the same error.

Regenerating without a duplicate message

The rescue path has one detail that decides whether any of it works. chat.ask(prompt) writes the user message and then calls the provider. Calling ask again after a failure writes the user message a second time, and the conversation permanently contains it twice.

rescue *RETRYABLE => e
  raise unless config.backup?

  Rails.logger.warn("AI primary (#{model}) failed: #{e.class} - failing over to #{config.backup_model}")
  reset_response(chat, target)
  chat.with_model(config.backup_model, provider: config.backup_provider).complete { |chunk| stream(chat, chunk, target) }
end

complete continues from the messages already on the chat, so the user's prompt is used and not re-persisted. with_model swaps model and provider on the same chat object, which is why one Chat record can carry an answer produced by a model different from the one it was created with.

reset_response is the other half. The failed attempt may already have streamed a partial sentence into the target, and the backup model appends from the beginning. Without the replace that clears it, the reader gets half of one answer welded to all of another, with no indication that anything went wrong. This is the failure that survives a test suite easily, since a test asserting the final content passes on the second answer alone.

Where the numbers behind the quota live

Ai::ModelQuota counts chats because counting tokens means reading a different table entirely, one that moved between major versions of the gem. Where ruby_llm 2.0 keeps token counts covers that table, and the per-user allowance that is enforced in the controller before any of this model selection happens.

The streaming side of the failover, including what reset_response broadcasts and why a partial answer is visible at all, is in streaming an LLM answer with Turbo Streams.

What this page does not cover

Retrying the same model. ActiveJob retries are not configured for this job, so a rate limit with no backup configured raises and the completion is lost rather than attempted again a second later. Exponential backoff against the primary would be the cheaper first move for a rate limit, and it is not here.

Nor does it cover a third model. The fallback is one level deep: if the backup also fails, the exception propagates and the user sees a failed job rather than a third attempt. A chain would need the retryable set applied to each link and a rule about how far to walk it.