The AI layer in Rails

An LLM call is an HTTP request that takes eight seconds and may fail halfway. Everything awkward about AI in Rails comes from those two facts: eight seconds is far too long to hold a request open, and "may fail" covers several different events that deserve different answers.

The shape this takes in a Rails app

The controller does almost nothing. Ai::CompletionsController#create checks the user's monthly allowance, creates a Chat, enqueues AiCompletionJob, and redirects. The page it redirects to subscribes to that chat with turbo_stream_from @chat and waits.

Everything slow happens in the job, which is the only part that talks to a provider. It picks a model, calls chat.ask(prompt) with a block, and broadcasts each chunk as it arrives. When the stream ends it replaces the whole target with the finished answer, so a reader who arrives late or reloads the page gets the complete text rather than an empty div.

That split is what makes the rest tractable. The request finishes in milliseconds and the job owns every hard case: the provider being slow, the provider being down, the budget being gone.

def create
  if Ai::UsageQuota.new(Current.user).exceeded?
    redirect_to(new_ai_completion_path, alert: t("ai.flash.quota_exceeded")) and return
  end

  chat = Current.user.chats.create!(model: Ai::Config.current.model)
  AiCompletionJob.perform_later(chat, prompt_param)
  redirect_to ai_completion_path(chat)
end

Streaming without a WebSocket layer

The token-by-token effect people associate with a chat UI is, in Rails 8, a broadcast_append_to per chunk and nothing else. Streaming an LLM answer with Turbo Streams works through the whole path: the job's streaming block, why the final message is broadcast a second time as a replace, the Stimulus controller that hides the spinner on the first token, and what happens when a browser subscribes after the stream already finished.

Solid Cable carries the broadcasts, which means Postgres carries them. No Redis, no second process beyond the job runner you already have for ActiveJob. In development that is the difference between one command and three.

Two failures, two fallbacks

A provider outage and an exhausted budget both end with "use the other model", which makes them look like one problem. They are not, and the code has two separate answers. The two fallbacks covers the quota check that runs before the call, the four exception classes that trigger the error fallback inside the rescue, and the detail that makes the second one work at all: regenerating with chat.complete rather than asking again, so the user's message is not written twice.

It also covers what is deliberately not caught. An invalid API key and a malformed request are not transient, and failing over to a second provider on those hides a configuration error behind a slightly worse answer.

Knowing what it cost

A feature that spends money per request needs a number afterwards, and this is where a gem upgrade can silently take one away. LLM token usage is recorded per provider attempt rather than per answer, so one visible reply can account for two rows. Where ruby_llm 2.0 keeps token counts explains the move from token columns on messages to a ruby_llm_usages row per provider attempt, why the dashboard that summed the old columns kept returning zero without ever raising, and how the join back to chats has to be written by hand because the association is polymorphic.

The same page covers the two quota objects, which are easy to conflate: one limits a single user's completions per month, the other watches total usage of the primary model and is what trips the quota fallback.

What this hub does not cover

Retrieval, embeddings and vector search. Nothing here builds an index over your own documents, and the NON_CHAT filter in Ai::Config deliberately excludes embedding models from the pickers, so the registry this app shows you is chat models only.

Nor does it cover agents and tool calling. ruby_llm supports both and the schema has the ruby_llm_tool_calls table for them, but no feature in this codebase uses one, and writing a page about code nobody here has run is how a documentation site starts inventing.