Rails and AI

Putting an LLM in a Rails app is mostly not about prompts. A Rails LLM feature is an HTTP call that takes eight seconds, fails in four distinguishable ways, costs money per token, and returns its answer in fragments. ruby_llm handles the provider differences. Everything else is still yours.

What follows is what we learned wiring it into a production Rails 8 application, including the parts that were wrong the first time.

What ruby_llm actually is

A single chat interface over seventeen providers, plus an ActiveRecord layer that persists the conversation. Most Rails AI work that is not prompt writing is one of the two. The interface is small enough to fit in a paragraph: RubyLLM.chat gives you a chat, ask sends a message and returns the reply, passing a block streams it in chunks, and with_model switches which model answers.

The ActiveRecord layer is where the surprises are. One line on a model, acts_as_chat, and your application has three new associations and six tables. What acts_as_chat puts behind one line walks the schema: which tables you will query, which ones sit empty until you use a feature you may never use, and the load-order rule that decides whether your app boots in production.

Version numbers matter more than usual here. Token accounting lived on messages before 2.0 and lives in ruby_llm_usages after it, and the old columns do not raise when you sum them.

The failure modes are the feature

An LLM provider is a dependency you cannot test into reliability. It rate limits, it overloads, it returns 503, and it does all three more often than a database does.

ruby_llm maps those onto an exception hierarchy worth reading before you write a rescue, and it also retries four of those classes three times before you ever see one. That reorders the problem: by the time your rescue runs, "try again" is finished, and what is left to decide is whether to try somewhere else.

UnauthorizedError is the class to keep out of that decision. Catching it alongside the transient ones turns a wrong API key into a feature that silently runs on your backup model and a bill nobody expected. Failing over without asking twice covers the split, the retries that already happened, and the detail that decides whether failover works at all: regenerating with complete instead of ask.

One interface, seventeen providers, and the seam

The pitch for a gem like this is that providers become interchangeable. In the chat call they genuinely are: the same ask reaches Anthropic, OpenAI, Gemini, Bedrock, Vertex AI or a local Ollama, and the reply comes back in the same shape.

The seam is one layer down. Seventeen providers and the credentials that refuse to be uniform reads configuration_requirements off each provider class rather than guessing: twelve want one API key, Azure and Ollama want an endpoint, Bedrock wants a region, and Vertex AI wants a project and a location. The two big clouds require the least of anybody, because both already know who you are and only need to be told where. A settings screen with one password field per provider cannot express five of the seventeen.

What these pages do not cover

Not prompt engineering. Nothing here is about what to write in the prompt, which is a skill these pages have no claim to teach and which changes faster than any page can track.

Not what any one application does with the gem. These pages stay at the level of ruby_llm itself, because that is the level the question is asked at. The AI layer in this Rails boilerplate is the worked example on the other side of that line: the same gem with a job, two quotas and an admin dashboard around it, which is a different question and gets a different page.

Not an agents-and-tools reference either. ruby_llm supports tool calling and ships a ruby_llm_tool_calls table for it, and no feature we run uses one, so writing that page would mean describing code we have not executed. What is here has been run in production, and it stops there.