Streaming an LLM answer with Turbo Streams, and the second broadcast that saves it

A completion takes several seconds and the reader stares at nothing for all of them. The fix is to put each chunk on the screen as it arrives, and in Rails 8 that costs one line inside the streaming block rather than a WebSocket layer.

The subscription is one line in the view

The page that shows a completion subscribes to the chat record itself:

<%= turbo_stream_from @chat %>

No channel class, no ApplicationCable::Channel subclass, no JavaScript. turbo_stream_from takes any object and derives a signed stream name from it, and the job broadcasts to that same object. Two developers can read the view and the job and see the connection without a third file explaining the naming convention.

What the page also needs is a stable target to append into. The response partial renders a div whose id is passed in, defaulting to ai_response, because a page showing several answers at once needs one id per answer or two streams append into the same element and interleave.

<div id="<%= target %>" class="prose max-w-none text-gray-800">
  <%= Ai::WidgetOutput.render(content, local_assigns[:template_key]) if content.present? %>
</div>

One broadcast per chunk

ruby_llm streams by yielding chunks to the block passed to ask. The whole streaming path is that block:

def stream(chat, chunk, target)
  text = chunk.content.to_s
  return if text.blank?

  Turbo::StreamsChannel.broadcast_append_to(chat, target: target, html: ERB::Util.html_escape(text))
end

Two details in four lines. Blank chunks are dropped, because providers emit chunks carrying only metadata and each one would otherwise cost a broadcast that appends nothing. And the text is escaped explicitly, which is the part worth stopping on: this is model output going straight into the DOM, on a path where no view helper escapes anything for you. A model that emits a <script> tag, or quotes one back from the user's own prompt, is otherwise appending markup to your page.

Solid Cable carries those broadcasts, so the transport is a Postgres table rather than Redis. For a Rails 8 app that already runs Solid Queue for the job, the streaming feature adds no new infrastructure at all.

The second broadcast, and why a reload needs it

Appending is an event. It changes the DOM of every browser currently subscribed, and it writes nothing anywhere. Reload the page halfway through and the answer is gone, because the appends happened to a document that no longer exists and the database has the message only when the model has finished with it.

So the job broadcasts once more when the stream ends, replacing the entire target with the assistant message read back from the database:

def finalize(chat, target, template_key = nil)
  Turbo::StreamsChannel.broadcast_replace_to(
    chat, target: target, partial: "ai/completions/response",
    locals: { content: chat.messages.where(role: "assistant").last&.content, target: target, template_key: template_key }
  )
end

Now the page has two ways to be correct. A browser watching live sees chunks and then one replace. A browser arriving afterwards renders the same partial from the same message on first load, since show reads the last assistant message into @answer. A reload mid-answer loses the partial text and gets the rest when the stream finishes, which is the failure mode nobody notices because it looks like a slightly slow answer rather than a bug.

Hiding the spinner without polling

The remaining piece is cosmetic and is worth the twenty lines. The generating hint has to disappear when the first token lands, and the page has to follow the text as it grows. Neither event is something the server can tell the page about, since the server's whole contribution is appends into a div.

A Stimulus controller watches the target instead:

connect() {
  this.observer = new MutationObserver(() => this.update())
  this.observer.observe(this.responseTarget, { childList: true, subtree: true, characterData: true })
}

MutationObserver fires on any DOM change inside the target, whatever caused it, so it needs to know nothing about Turbo. The update handler hides the spinner once the target has text and calls scrollIntoView on it. The observer is disconnected in disconnect(), which matters more than it looks: Turbo caches pages and restores them, and an observer left attached to a detached element is a leak that accumulates over a browsing session rather than crashing anything.

What decides whether the stream is even possible

Which model runs the completion is chosen before the job opens a connection, and the choice can change again halfway through. A provider that rate limits after four chunks leaves those four chunks on the screen, and the backup model then starts its own answer from the first word into the same DOM target. Appending the second on top of the first produces one paragraph made of two answers, which reads as a model losing its train of thought rather than as an error.

The job handles that with a broadcast_replace_to that empties the target before the retry begins, which is the only reason a mid-stream failover is invisible to the reader. The two fallbacks covers which four exception classes reach that path, and the separate quota check that picks a model before any of this happens.

What this page does not cover

Cancelling a stream in flight. The chats table carries a cancelled boolean and no code in this flow reads it, so a user who closes the tab pays for the rest of the completion anyway. Stopping a provider call already in progress means a check inside the streaming block and a way to signal it from a controller, and neither exists here.

Nor does it cover backpressure. Every chunk is a broadcast, a provider emitting many small chunks produces many small Postgres writes through Solid Cable, and nothing batches them. At the traffic this is written for that is fine, and it is the first thing that would need measuring if it were not.