Failing over to a second provider without asking twice

Most advice about Rails LLM error handling starts with "wrap the call in a retry". With ruby_llm that advice is already wrong, because a ruby_llm retry is configured in the gem and has already happened by the time you see an exception.

What the gem does before you get a chance

ruby_llm builds its Faraday connection with the retry middleware configured from its own settings:

RubyLLM.config.max_retries              # => 3
RubyLLM.config.retry_interval           # => 0.1
RubyLLM.config.retry_backoff_factor     # => 2
RubyLLM.config.retry_max_interval       # => 30
RubyLLM.config.retry_interval_randomness # => 0.5

Three retries after the first attempt, starting at a tenth of a second, doubling, with jitter so a fleet of workers does not synchronise its retries into a second spike.

The list of what it retries on is worth reading, because it is the same list most people write into their own rescue:

Errno::ETIMEDOUT, Timeout::Error, Faraday::TimeoutError, Faraday::ConnectionFailed,
Faraday::RetriableResponse, RubyLLM::RateLimitError, RubyLLM::ServerError,
RubyLLM::ServiceUnavailableError, RubyLLM::OverloadedError

Pointing a chat at a provider that answers 429 and counting the requests that actually arrive gives four, not one. So a rate limit that clears within a second is handled before your code hears about it, and anything that does reach your rescue has already failed four times over roughly a second.

That changes what a rescue is for. An LLM failover is not "try again", which the gem has already finished doing. It is "try somewhere else".

The one thing the gem will not retry

Look at the condition the middleware is given:

retry_if: lambda { |env, _exception|
  env[:method] == :post && idempotent?(env) && !stream_delivered?(env)
}

stream_delivered? is the interesting clause. Once a streaming response has started handing chunks to your block, the gem stops retrying that request, because replaying it would restart an answer whose beginning is already on the reader's screen.

The consequence lands in your lap rather than the gem's. A stream that dies after four chunks leaves those four chunks wherever you put them, and whatever you do next has to deal with them. If you failover, the second model begins its own answer from the first word, and appending it to what is already there produces one paragraph made of two answers.

Which errors mean what

RubyLLM::Error is the base class for provider failures and wraps the HTTP response, so e.response carries the status and body. The subclasses map onto status codes, and running each one against a provider that returns it confirms the mapping:

The provider answers	You get
429	RubyLLM::RateLimitError
503	RubyLLM::ServiceUnavailableError
401	RubyLLM::UnauthorizedError

The first two are worth failing over on. The third is not, and it is the one people catch by accident. A wrong API key that quietly runs on your backup model looks like a working feature with a slightly different accent, and the misconfiguration surfaces weeks later as a bill from a provider nobody meant to use.

There is a second class of error that deliberately escapes rescue RubyLLM::Error entirely. ConfigurationError and ModelNotFoundError descend from StandardError directly, because they are your mistakes rather than the provider's, and a rescue written for an outage should not swallow a typo in a model id.

Regenerating without asking twice

One detail decides whether failover works at all, and it is not the rescue. chat.ask(prompt) writes the user message and then calls the provider. Call ask again after a failure and the question is written a second time, permanently, in a conversation the model will read back later.

RETRYABLE = [ RubyLLM::RateLimitError, RubyLLM::ServiceUnavailableError,
              RubyLLM::OverloadedError, RubyLLM::ServerError ].freeze

chat.with_model(primary, provider: primary_provider)
begin
  chat.ask(question)
rescue *RETRYABLE
  chat.with_model(backup, provider: backup_provider).complete
end

complete continues from the messages already on the chat, so the question is used and not re-persisted. Counted after a real failover, the chat holds one user message. Replacing complete with a second ask gives two.

with_model swaps model and provider on the same chat object, which is why one Chat record can carry an answer produced by a model different from the one it was created with, and why the ruby_llm_model_id column on a chat is the model that answered rather than the model that was chosen.

For the same reason the backup belongs on a different provider. A second model hosted by the primary's provider shares its outage, its rate limit and its status page, so the failover fires and fails on the same error.

What the cost table looks like afterwards

A failover is invisible to the reader and very visible in ruby_llm_usages. One question, answered after a primary that stayed down, left five rows: four with status failed and one succeeded.

Four of those five rows are input tokens the provider processed for text nobody ever saw, and they are still billed. A cost report that filters on status = 'succeeded' is therefore not a cost report, it is a report of what worked, and the difference grows exactly when a provider is having a bad day. What acts_as_chat puts behind one line covers the table those rows land in.

What this page does not cover

A chain of more than one backup. The pattern above is one level deep: if the backup also fails, the exception propagates. A chain needs the retryable set applied at each link and a rule about how far to walk it, and every extra link multiplies the worst-case latency by the gem's own four attempts.

Nor does it cover giving up gracefully. What a user should see when both providers are down is a product decision rather than a gem one, and the honest options are a queued retry and an error, not a fabricated answer.