The AI layer in Rails

An LLM call is an HTTP request that takes eight seconds and may fail halfway. AI in Rails, as this boilerplate ships it, is three answers to that one fact: where the call runs, what limits it, and what tells you afterwards what it cost.

The gem underneath is ruby_llm, and what the gem itself does is a separate subject with its own pages. Rails and AI covers the schema it generates, its retry policy and what each provider needs before it answers. Everything below is the code in this repository.

The shape this takes in a Rails app

The controller does almost nothing. Ai::CompletionsController#create checks the user's monthly allowance, creates a Chat, enqueues AiCompletionJob, and redirects. The page it redirects to subscribes to that chat and waits.

def create
  if Ai::UsageQuota.new(Current.user).exceeded?
    redirect_to(new_ai_completion_path, alert: t("ai.flash.quota_exceeded")) and return
  end

  chat = Current.user.chats.create!(model: Ai::Config.current.model)
  AiCompletionJob.perform_later(chat, prompt_param)
  redirect_to ai_completion_path(chat)
end

Everything slow happens in the job, which is the only part that talks to a provider. The request finishes in milliseconds and the job owns every hard case: the provider being slow, the provider being down, the budget being gone. The input is capped at 8000 characters on the way in, which is the cheapest thing in the whole module and the one that stops a single prompt being unbounded.

Streaming without a WebSocket layer

The token-by-token effect people associate with a chat UI is, in Rails 8, one broadcast_append_to per chunk and nothing else. Streaming an LLM answer with Turbo Streams works through the whole path: the job's streaming block, why the finished answer is broadcast a second time as a replace, the Stimulus controller that hides the spinner on the first token, and what happens when a browser subscribes after the stream already ended.

Solid Cable carries the broadcasts, which means Postgres carries them. No Redis, and no second process beyond the job runner the application already runs.

Two quotas, and the zero that means the opposite

Two quota objects sit in this module, and conflating them is the easy mistake. Ai::UsageQuota is per user and counts chats, not tokens:

FREE_MONTHLY_LIMIT = 20
SUBSCRIBED_MONTHLY_LIMIT = 500

def used = @user.chats.this_month.count

One completion is one chat, so the allowance a user sees is a count of things they did rather than a number of tokens they cannot estimate. It is checked in the controller before the job is enqueued, which is early enough that an over-quota user costs nothing at all.

Ai::ModelQuota is the other one: global, admin-set, and watching the primary model across the whole application. Its field is an integer, and empty arrives as zero, so unlimited? is quota.zero?.

The consequence catches everybody once. Setting the quota to zero does not stop completions, it removes the limit entirely, so somebody trying to pause AI spending by typing 0 gets the opposite of what they meant. The lever for that is the feature flag, which makes the whole ai module answer

1. The quota field is for budgeting.

What the dashboard had to be rewritten to read

Admin::AiController reports completions, tokens and an estimated cost per month. Tokens are the part that broke: on a ruby_llm 2.0 install they are not on messages any more, and Message.column_names.grep(/token/) returns an empty array. Summing a column that does not exist returns zero and raises nothing, so the dashboard read zero on every screen while every other figure on the same page stayed correct.

It reads the gem's own table instead, and the join is written by hand because the association it needs is polymorphic:

CHATS_JOIN = <<~SQL.squish.freeze
  INNER JOIN chats ON chats.id = ruby_llm_usages.chat_id
                  AND ruby_llm_usages.chat_type = 'Chat'
SQL

The cost figure beside it is tokens multiplied by a flat estimated rate, and it says so. The gem records a real per-row cost that varies by model, and summing that instead is the accurate version this module does not do yet.

What this page does not cover

The gem. Its schema, its error classes, its retry policy and its provider requirements are in Rails and AI, and repeating them here would be two pages answering one question.

Retrieval, embeddings and vector search. Nothing here indexes your own documents, and the NON_CHAT filter in Ai::Config deliberately keeps embedding models out of the pickers, so the thirteen providers the admin offers are chat providers out of the seventeen the gem registers.

Agents and tool calling. The schema has a table for them and no feature in this codebase uses one, so writing that page would mean describing code nobody here has run.