Finding a Rails memory leak
A Puma worker that starts at 95 MB and is at 400 MB by Thursday is the most common performance report a Rails team files, and the first question asked about it is almost always the wrong one. The question is not "where is the leak". It is "is this a leak at all", because a Rails process that grows and then stops is normal and a Rails process that grows in a straight line is a bug, and for the first few thousand requests the two are indistinguishable on any graph a hosting dashboard will draw for you.
Everything below ran on one machine: Ruby 4.0.5 on arm64-darwin25, Apple M2 Max, Rails 8.1.4,
puma 8.0.2, rack 3.2.7, memory_profiler 1.1.0. The application is a rails new --minimal app served
by a single-process Puma with one thread, RAILS_ENV=production and eager loading on, driven by
/usr/sbin/ab. One thread on purpose: the numbers are per process and a thread pool only multiplies
them. Load figures were taken with ApacheBench at the concurrency stated beside each one.
The probe, and the two numbers it reads
Every number in this post comes out of one method. It reads RSS from ps, because that is the
figure the platform kills you on, and it reads the garbage collector's own counters, because those
are the figures that say why.
module Probe
def self.rss_kb
`ps -o rss= -p #{Process.pid}`.to_i
end
def self.report
s = GC.stat
format(
"pid=%d rss_mb=%.1f heap_live_slots=%d heap_pages=%d old_objects=%d " \
"malloc_increase_bytes=%d major_gc=%d minor_gc=%d\n",
Process.pid, rss_kb / 1024.0, s[:heap_live_slots], s[:heap_allocated_pages],
s[:old_objects], s[:malloc_increase_bytes], s[:major_gc_count], s[:minor_gc_count]
)
end
end
Hung off a route at /stats, Probe.report answers in about a millisecond. The shell script that
drives every table below is the other half:
#!/bin/bash
# drive.sh <path> <chunks> <requests-per-chunk>
P=$1; CHUNKS=$2; N=$3
printf "requests probe\n"
printf "%8d %s\n" 0 "$(curl -s http://127.0.0.1:3781/stats)"
for i in $(seq 1 $CHUNKS); do
/usr/sbin/ab -n $N -c 4 -q "http://127.0.0.1:3781/$P" > /dev/null 2>&1
printf "%8d %s\n" $((i*N)) "$(curl -s http://127.0.0.1:3781/stats)"
done
Two actions, one of which leaks
Here are the two actions at the top of the controller. The line numbers matter later, so they are the
real ones and the closing end is mine: the file has more methods below it.
class ProbesController < ApplicationController
# The leak: a constant that is appended to and never truncated.
RECENT_SEARCHES = []
def leak
RECENT_SEARCHES << {
query: params.fetch(:q, "shoes") * 40,
path: request.fullpath,
ip: request.remote_ip,
at: Time.now
}
render plain: "leaked #{RECENT_SEARCHES.size}\n"
end
# No leak. Allocates exactly as much per request, retains none of it.
def churn
rows = 20_000.times.map { |i| { query: "shoes #{i}" * 40, at: Time.now } }
render plain: "churned #{rows.size}\n"
end
end
churn is the innocent one. It allocates 20000 Hashes and 20000 Strings per request, returns a
count, and holds on to nothing. ./drive.sh churn 8 500, which is eight rounds of ab -n 500 -c 4:
requests probe
0 pid=85253 rss_mb=95.1 heap_live_slots=313204 heap_pages=304 old_objects=211827 malloc_increase_bytes=5664790 major_gc=5 minor_gc=16
500 pid=85253 rss_mb=138.7 heap_live_slots=406853 heap_pages=924 old_objects=321626 malloc_increase_bytes=60412 major_gc=6 minor_gc=517
1000 pid=85253 rss_mb=142.2 heap_live_slots=460871 heap_pages=924 old_objects=366904 malloc_increase_bytes=87300 major_gc=6 minor_gc=1017
1500 pid=85253 rss_mb=144.8 heap_live_slots=296463 heap_pages=926 old_objects=216764 malloc_increase_bytes=51092 major_gc=7 minor_gc=1406
2000 pid=85253 rss_mb=144.8 heap_live_slots=296476 heap_pages=926 old_objects=216779 malloc_increase_bytes=51092 major_gc=7 minor_gc=1656
2500 pid=85253 rss_mb=144.8 heap_live_slots=296489 heap_pages=926 old_objects=216794 malloc_increase_bytes=51092 major_gc=7 minor_gc=1906
3000 pid=85253 rss_mb=144.8 heap_live_slots=296506 heap_pages=926 old_objects=216809 malloc_increase_bytes=60412 major_gc=7 minor_gc=2156
3500 pid=85253 rss_mb=144.8 heap_live_slots=296521 heap_pages=926 old_objects=216824 malloc_increase_bytes=51092 major_gc=7 minor_gc=2406
4000 pid=85253 rss_mb=144.8 heap_live_slots=296544 heap_pages=926 old_objects=216839 malloc_increase_bytes=51092 major_gc=7 minor_gc=2656
A 46 percent jump in resident memory in the first 500 requests, and then nothing. Flat at 144.8 MB
for 2500 more requests, heap_pages pinned at 926, old_objects back to 216839 after the major
collection at request 1500 and then moving by 15 per sample, which is the probe endpoint itself.
That plateau is the whole story of most "leaks": the worker sized its heap for its actual workload,
which nobody had ever made it do before, and stopped. Report that graph after ten minutes of traffic
and it is a leak. Report it after an hour and it is a process.
leak is the guilty one, and it is deliberately as small as real leaks usually are: one Hash, four
values, about 600 bytes. ./drive.sh leak 10 2000:
requests probe
0 pid=89407 rss_mb=94.6 heap_live_slots=307380 heap_pages=304 old_objects=211833 malloc_increase_bytes=5646326 major_gc=5 minor_gc=16
2000 pid=89407 rss_mb=97.8 heap_live_slots=286489 heap_pages=314 old_objects=224755 malloc_increase_bytes=7512 major_gc=6 minor_gc=28
4000 pid=89407 rss_mb=98.0 heap_live_slots=290233 heap_pages=317 old_objects=235328 malloc_increase_bytes=7512 major_gc=6 minor_gc=44
6000 pid=89407 rss_mb=101.4 heap_live_slots=318085 heap_pages=365 old_objects=244488 malloc_increase_bytes=7512 major_gc=7 minor_gc=62
8000 pid=89407 rss_mb=105.6 heap_live_slots=350816 heap_pages=415 old_objects=254251 malloc_increase_bytes=7512 major_gc=8 minor_gc=71
10000 pid=89407 rss_mb=105.8 heap_live_slots=355037 heap_pages=419 old_objects=264481 malloc_increase_bytes=7512 major_gc=8 minor_gc=81
12000 pid=89407 rss_mb=108.3 heap_live_slots=370237 heap_pages=452 old_objects=273889 malloc_increase_bytes=7512 major_gc=9 minor_gc=89
14000 pid=89407 rss_mb=111.0 heap_live_slots=394912 heap_pages=488 old_objects=283959 malloc_increase_bytes=7512 major_gc=10 minor_gc=98
16000 pid=89407 rss_mb=111.2 heap_live_slots=380030 heap_pages=491 old_objects=294887 malloc_increase_bytes=7512 major_gc=10 minor_gc=108
18000 pid=89407 rss_mb=114.3 heap_live_slots=435922 heap_pages=535 old_objects=303021 malloc_increase_bytes=7512 major_gc=11 minor_gc=114
20000 pid=89407 rss_mb=115.7 heap_live_slots=424316 heap_pages=557 old_objects=313292 malloc_increase_bytes=7512 major_gc=12 minor_gc=122
Read the two RSS columns side by side and the innocent action looks far worse: 49.7 MB gained
against 21.1 MB. Read the old_objects column and the verdict reverses. churn came back to
leakwent 211833, 224755, 235328, 244488, 254251, 264481, 273889, 283959, 294887, 303021, 313292, which is 101459 objects over 20000 requests, or 5.07 objects retained per request, and the slope does not bend once in ten samples.
That is the measurement. Not RSS, which is dominated by the heap the process needed anyway, but
old_objects, the count of objects that have survived three garbage collections and been promoted
to the old generation. A Ruby memory leak is by definition a set of objects that keeps surviving, so a
leak is by definition a rising old_objects, and a rising old_objects over a long enough window
has no other explanation available.
heap_live_slots is the counter people reach for first and it is the wrong one. Look at its column
in the leak run: 307380, then 286489, then 290233, up to 435922, back to 380030. It samples whatever
the last garbage collection happened to leave behind, so it jitters by 100000 slots between
consecutive readings of the same leaking process. Two readings of heap_live_slots tell you nothing.
Two readings of old_objects far enough apart tell you everything.
At 1.08 KB of RSS per request, the arithmetic on that slope says this worker adds 400 MB somewhere around 380000 requests. Arithmetic on a slope is not a measurement and the real number will be worse on the day the slope steepens, which is what an uncapped collection does when the site gets popular.
Neither counter catches every leak
old_objects missed the next leak entirely. Same application, an action that pushes one 10 KB String
onto a thread local:
def thread_leak
trail = (Thread.current[:trail] ||= [])
trail << ("x" * 10_000)
render plain: "thread #{Thread.current.object_id} holds #{trail.size}\n"
end
Five thousand requests, one Puma thread, and the probe at the end:
pid=74961 rss_mb=95.3 heap_live_slots=310120 heap_pages=305 old_objects=211828 ...
1 threads holding a trail
puma srv tp 001 holds 5000
pid=74961 rss_mb=150.5 heap_live_slots=290834 heap_pages=307 old_objects=221540 ...
RSS went up 55.2 MB, which is 5000 times 10 KB and change. old_objects went up 9712, which is
about two objects a request and indistinguishable from the probe's own noise. heap_pages moved by
two. A 10 KB String's bytes live in a malloc'd buffer outside the object slot, so a leak made of a
few large Strings, a few StringIO buffers or a few big Hashes is invisible in the object counters
and obvious in RSS, and the leak made of many small objects is the reverse. Watch both columns or
you will confidently clear a process that is leaking 55 MB per 5000 requests.
Three heap dumps name the line
Knowing a process leaks is a day's work away from knowing which line leaks, unless you take heap dumps, in which case it is about four minutes. The technique is three dumps and a set difference: anything born between the first and the second and still alive at the third is retained by something, because two dumps apart it has already survived everything the collector was going to do to it.
The dump endpoint:
def dump
path = Rails.root.join("tmp", "heap-#{params.fetch(:tag)}.json")
GC.start(full_mark: true, immediate_sweep: true)
File.open(path, "w") { |f| ObjectSpace.dump_all(output: f) }
render plain: "#{path} #{File.size(path)}\n"
end
For the file and line keys to appear in that output at all, the process must have been recording
allocations since before the objects existed, which means an initializer:
if ENV["TRACE_ALLOCATIONS"] == "1"
require "objspace"
ObjectSpace.trace_object_allocations_start
Rails.logger.warn "allocation tracing on"
end
Then: dump, 500 requests, dump, 500 requests, dump. The three files came out at 75185472, 76617755 and 77773447 bytes, one JSON object per line, about 219000 objects in the first. The analyzer is 35 lines of plain Ruby with no gem:
# analyze.rb a.json b.json c.json
# Objects born between dump a and dump b, still alive in dump c.
require "json"
def generations(path)
gens = {}
File.foreach(path) do |line|
o = JSON.parse(line)
gens[o["address"]] = o
end
gens
end
a, b, c = ARGV.map { |p| generations(p) }
max_gen_a = a.values.filter_map { |o| o["generation"] }.max
max_gen_b = b.values.filter_map { |o| o["generation"] }.max
puts "dump a: #{a.size} objects, newest generation #{max_gen_a}"
puts "dump b: #{b.size} objects, newest generation #{max_gen_b}"
puts "dump c: #{c.size} objects"
born = b.select { |addr, o| o["generation"] && o["generation"] > max_gen_a }
survived = born.select { |addr, _| c.key?(addr) }
puts "born between a and b: #{born.size}"
puts "still alive in c: #{survived.size}"
puts
def short(f)
f.to_s.sub("#{Dir.pwd}/", "./").sub(%r{\A.*/gems/}, "")
end
counts = survived.values.group_by { |o| "#{short(o["file"])}:#{o["line"]}" }
counts.sort_by { |_, v| -v.size }.first(8).each do |site, objs|
bytes = objs.sum { |o| o["memsize"].to_i }
puts format("%6d objects %9d bytes %s", objs.size, bytes, site)
end
ruby analyze.rb tmp/heap-a.json tmp/heap-b.json tmp/heap-c.json, 2.6 seconds:
dump a: 218780 objects, newest generation 22
dump b: 221493 objects, newest generation 25
dump c: 223889 objects
born between a and b: 1871
still alive in c: 1635
326 objects 13040 bytes actionpack-8.1.4/lib/action_dispatch/middleware/remote_ip.rb:182
326 objects 13040 bytes rack-3.2.7/lib/rack/request.rb:604
326 objects 26080 bytes <internal:timev>:271
326 objects 52160 bytes ./app/controllers/probes_controller.rb:6
326 objects 104320 bytes ./app/controllers/probes_controller.rb:7
1 objects 40 bytes puma-8.0.2/lib/puma/thread_pool.rb:380
1 objects 80 bytes actionpack-8.1.4/lib/action_dispatch/request/session.rb:21
1 objects 80 bytes /Users/mehdifarsi/.rvm/rubies/ruby-4.0.5/lib/ruby/4.0.0/random/formatter.rb:174
Line 6 is RECENT_SEARCHES << { and line 7 is query: params.fetch(:q, "shoes") * 40. The leak is
found, and the interesting part of that output is the three lines above it. remote_ip.rb:182 is
the String that request.remote_ip returns, rack/request.rb:604 is request.fullpath, and
<internal:timev>:271 is Time.now. Three of the top five sites are inside gems, all three are
being retained, and none of them is the bug. They are the leak's contents. A leak drags in whatever
its payload references, so the reading skill this output needs is to find the deepest line that is
in your own application and stop there, and to distrust any conclusion of the form "the leak is in
Rack".
Two other things that output says out loud. All five leaking sites report exactly 326 objects, not
500, because the filter is generation > max_gen_a and the first dump landed part-way through
generation 22; the requests whose objects were born in that same generation are excluded by
construction. And the counts are equal across sites because each request retains exactly one object
from each, which is what a leak with a fixed payload looks like and is a useful thing to recognise:
uneven counts mean the payload varies, which usually means the retained thing is a collection.
What allocation tracing costs
ObjectSpace.trace_object_allocations_start is not free and the bill is large enough to plan
around. Two boots each way, the probe read immediately after Puma announced it was listening:
TRACE_ALLOCATIONS=0 pid=80909 rss_mb=94.5 heap_live_slots=310073 heap_pages=305 old_objects=211826 ...
TRACE_ALLOCATIONS=1 pid=81114 rss_mb=132.5 heap_live_slots=296763 heap_pages=305 old_objects=211875 ...
TRACE_ALLOCATIONS=0 pid=81274 rss_mb=95.5 heap_live_slots=310088 heap_pages=305 old_objects=211828 ...
TRACE_ALLOCATIONS=1 pid=81419 rss_mb=126.5 heap_live_slots=302433 heap_pages=305 old_objects=211874 ...
Between 31 and 38 MB before the process serves anything, because the initializer runs before eager
loading and every constant Rails defines gets a source record attached. Throughput, ab -n 3000 -c 1
on the cheap endpoint and ab -n 2000 -c 1 on the one that allocates 20000 Hashes per request:
TRACE_ALLOCATIONS=0 /leak Requests per second: 2274.67 [#/sec] (mean)
TRACE_ALLOCATIONS=1 /leak Requests per second: 1247.66 [#/sec] (mean)
TRACE_ALLOCATIONS=0 Requests per second: 121.69 [#/sec] (mean)
TRACE_ALLOCATIONS=1 Requests per second: 14.82 [#/sec] (mean)
A 45 percent cut on an endpoint that barely allocates, and 8.2 times slower on one that allocates
hard. The cost scales with allocation rate, so the worse your endpoint, the worse the instrument.
That rules out leaving it on, and it does not rule out what it is actually for: turning it on for one
worker, for four minutes, behind a flag, on the box where the leak reproduces. ObjectSpace.dump_all
also writes 75 MB for a 95 MB process, so the disk it writes to needs to exist and the endpoint needs
to be closed to the public, since a heap dump contains every String in the process.
memory_profiler reported 600 bytes and I believed it
The first tool I reached for was memory_profiler, wrapped around the leaking line inside the action, and it is the reason this section exists.
Total allocated: 640.00 B (5 objects)
Total retained: 600.00 B (4 objects)
Six hundred bytes. I read that as "no leak here" and went looking somewhere else, which cost me the next twenty minutes. The report is correct and the inference is garbage: memory_profiler measures one block, and a leak is a small number multiplied by your traffic. Six hundred bytes a request across three million requests is 1.7 GB. The tool cannot tell you that, because the thing that makes 600 bytes a bug is a fact about the request rate, which is outside the block.
There is a second trap in the same gem's API, and it produced a 500 in production mode before it
produced anything else: pretty_print(to_file: io) wants a path, so handing it a StringIO raises
TypeError: no implicit conversion of StringIO into String. The first positional argument is the IO.
The subscriber leak whose memory was not the subscribers
An ActiveSupport::Notifications.subscribe called per request is a real leak and a common one,
because the call reads like registration and behaves like an append. The action subscribes once,
instruments once, and reports the listener count:
def subscribe
ActiveSupport::Notifications.subscribe("probe.tick") { |*| }
ActiveSupport::Notifications.instrument("probe.tick") { 1 }
n = ActiveSupport::Notifications.notifier.listeners_for("probe.tick").size
render plain: "listeners=#{n} " + Probe.report
end
Six rounds of ab -n 500 -c 1, reading time per request and RSS after each:
listeners ms/req rss_mb
501 5.457 106.7
1002 6.061 136.8
1503 7.890 159.4
2004 8.085 193.8
2505 9.023 237.2
3006 11.304 274.4
RSS from 106.7 MB to 274.4 MB across that table, against a boot figure of 94 to 95 MB measured over
and over above, and time per request doubled. I wrote down "each subscriber costs about 67 KB" and
that sentence was wrong, which I only found out because 67 KB for a Proc is absurd enough to check. The check, in bin/rails runner, subscribes 3000 times and never fires the event:
3000 subscribers only, no instrument:
rss 92.6 -> 92.8 MB (+0.2), live_slots 208521 -> 220539 (+12018)
1000 instruments with 3000 listeners: 0.370 s
after the instruments: rss 92.9 MB, live_slots 220591
Three thousand subscribers cost 0.2 MB and four object slots each. Essentially none of the 167.7 MB
in that table was the leaked objects. The leak's cost is instrument having to call 3000 blocks and build 3000
events every time anything is instrumented, and all that garbage is transient, so what the memory
graph shows is the plateau from the first section, sized by the fan-out. Fixing the subscriber leak
would drop the RSS, and not because the retained objects were the problem.
That is the correction worth carrying out of this post: a leak's memory graph is not a measurement of the leak. The 0.370 seconds for 1000 instrument calls, 0.37 ms each on a notification nobody subscribed to on purpose, is the measurement that matters here, and it is a latency bug wearing a memory bug's clothes.
Thread.current leaks where Current does not
Puma's threads live as long as the process, so Thread.current is process-lifetime storage with a
per-request name, and a Puma memory leak reported against an application that never writes to a
constant is very often this. It is the one leak in this post that looks completely safe in review. Two actions, identical except for where the array is kept:
def thread_leak
trail = (Thread.current[:trail] ||= [])
trail << ("x" * 10_000)
render plain: "thread #{Thread.current.object_id} holds #{trail.size}\n"
end
def current_leak
Current.trail ||= []
Current.trail << ("x" * 10_000)
render plain: "current holds #{Current.trail.size}\n"
end
Five thousand requests each, one thread, same 10 KB payload. The thread local ended at
rss_mb=150.5 with puma srv tp 001 holds 5000. Current, an ActiveSupport::CurrentAttributes
subclass with attribute :trail, ended at rss_mb=99.9 and answered current holds 1 on the five
thousandth request as it had on the first. With five Puma threads the thread local version spread the
same total across every worker thread by name:
5 threads holding a trail
puma srv tp 001 holds 998
puma srv tp 002 holds 1013
puma srv tp 003 holds 993
puma srv tp 004 holds 1008
puma srv tp 005 holds 988
What resets Current is not magic and it is one line:
activesupport-8.1.4/lib/active_support/railtie.rb:61 calls
ActiveSupport::CurrentAttributes.clear_all inside app.executor.to_complete, registered by the
active_support.reset_execution_context initializer at line 49 of the same file. Anything that runs
inside the Rails executor therefore gets Current emptied at the end, and that includes jobs and
Action Cable, not only requests. Nothing resets Thread.current[:trail], because nothing knows it
exists.
The rule that falls out of this: per-request state goes in CurrentAttributes or in a local, and a
bare Thread.current[...] write in application code is a leak unless the same method removes it. The
integration test at the end of this post asserts both counts.
Fixing the leak does not give the memory back
Clearing the leaking array is where the story usually goes wrong, because the fix demonstrably works
and the graph does not move. On the process that had just leaked 20000 entries, an endpoint that
calls RECENT_SEARCHES.clear and then a full GC.start:
cleared 20000
pid=89407 rss_mb=115.7 heap_live_slots=216837 heap_pages=557 old_objects=216647 malloc_increase_bytes=0 major_gc=13 minor_gc=122
old_objects fell from 313292 to 216647, which is its boot value. heap_live_slots fell to 216837.
The leak is gone by every measure Ruby has. RSS is 115.7 MB, which is what it was before the clear,
to the tenth of a megabyte. Then GC.compact, and then the full GC.stat keys that explain it:
heap_allocated_pages=557
heap_empty_pages=333
heap_live_slots=217426
old_objects=216677
old_objects_limit=433354
total_freed_pages=0
Three hundred and thirty-three of the 557 heap pages are completely empty, the process still owns
all 557, and total_freed_pages is 0 after 15 major collections and a compaction. Running the same
sequence again on a fresh worker shows GC.compact doing its own half of the job and no more:
heap_eden_pages 344 then 224 across the compaction, heap_empty_pages 224 then 344,
heap_allocated_pages 568 throughout, total_freed_pages 0 throughout, and RSS 117.5 MB on both
sides of it.
Ruby is not incapable of returning pages, which is the correction I had to make to my own first draft. A test that allocates two million small Arrays, clears them and collects freed 347 of the 1444 pages and brought RSS from 201.1 MB down to 179.4 MB, about a fifth of what the spike had cost:
before rss_mb=91.5 pages=240 empty=0 live_slots=210580 freed_pages=0
peak rss_mb=201.1 pages=1444 empty=0 live_slots=2210599 freed_pages=0
after rss_mb=179.4 pages=1097 empty=890 live_slots=210627 freed_pages=347
So page release is a heuristic that fires on a 1200 page swing and does not fire on a 250 page one,
and the 250 page one is the size real leaks come in. Plan on the measured behaviour: deploying the
fix stops the growth and only a restart gives the memory back. And GC.malloc_trim, the other lever
people reach for, answers false to GC.respond_to? on this build, so the malloc side of this was
not testable here at all.
Both halves of that are worth pinning in a test, because either is something a future Ruby could change:
test "clearing the leak returns the slots and not the resident memory" do
GC.start(full_mark: true, immediate_sweep: true)
before = GC.stat.merge(rss_kb: Probe.rss_kb)
keep = 2_000_000.times.map { |i| [ i ] }
GC.start(full_mark: true, immediate_sweep: true)
peak = GC.stat.merge(rss_kb: Probe.rss_kb)
keep.clear
keep = nil
GC.start(full_mark: true, immediate_sweep: true)
GC.compact
after = GC.stat.merge(rss_kb: Probe.rss_kb)
# The slots come back.
assert_in_delta before[:heap_live_slots], after[:heap_live_slots],
0.2 * before[:heap_live_slots]
# The resident memory does not.
grown = peak[:rss_kb] - before[:rss_kb]
kept = after[:rss_kb] - before[:rss_kb]
assert_operator kept, :>, grown * 0.5,
"more than half the resident memory the leak cost should still be resident"
end
Four tests, ten assertions, green, and the other three assert that RECENT_SEARCHES keeps every
request, that Current.trail is nil after each one, and that Thread.current[:trail] counts 1, 2, 3
across three requests on the same thread.
The position, and what would change it
Restart your web workers on a schedule, and stop treating that as an admission of defeat. A Rails
process cannot give back heap pages it stopped needing, so even a perfectly clean application ratchets
up to the peak of its worst hour and stays there, and a periodic restart is the only thing that
collects that. Puma has no memory limit of its own to do it with: grep -i memory across
puma-8.0.2/lib/puma/dsl.rb returns one comment and no setting.
The cost of that position, and it is the reason to be honest about it: a scheduled restart hides a
real leak perfectly until traffic grows enough that the leak outruns the schedule, and then it
arrives as an outage rather than as a slope. So the restart is only defensible next to a number.
Log GC.stat[:old_objects] and RSS from one worker once a minute, look at the two week shape, and
the distinction the first section of this post draws becomes a thing you can check rather than a
thing you argue about. If old_objects is flat across a restart interval, the ratchet is all you
have and the restart is the whole fix.
What would change the position: Ruby returning empty heap pages by default, which the
total_freed_pages=0 above says it does not do at the magnitudes that matter. A malloc_trim that
exists on this platform would change the malloc half of it, which I could not test.
What this post does not cover
The allocator underneath Ruby, which is most of the internet's advice on this subject and none of the
above. MALLOC_ARENA_MAX and jemalloc are glibc questions, this machine is arm64-darwin25 with the
system allocator and no --with-jemalloc in RbConfig::CONFIG["configure_args"], so nothing here
tests them; the measured version of that argument, in a Docker container where it applies, is in
Rails on a VPS, where jemalloc came out 4 percent worse than doing nothing three
runs in a row.
Also absent: job processes, where the same counters work and the unit is a job rather than a request,
and where a leak per job is usually far bigger; leaks inside C extensions, which are invisible to
ObjectSpace because the bytes were never Ruby objects and which show as rising RSS with flat
old_objects and flat heap_pages; derailed_benchmarks, which automates the dump diffing above and
which I did not install, so I have nothing measured to say about it; and the single-request memory
peak, which is a different problem with a different fix and is worked through on real data in
CSV import and export in Rails.
One leak shape deliberately left out because it needs its own page: a memoisation on the class rather
than the instance, def self.settings_for(tenant); @settings[tenant] ||= ...; end, which is a cache
with no eviction and therefore a leak whose size is the cardinality of your tenants. The memoisation
that is worth doing, and what it bought, is measured in
Rails performance improvements.
All the numbers above are one process on one laptop. The object counts reproduce exactly; the megabytes move by a percent or two between runs, and the requests-per-second figures move more than that.
Comments
No comments yet. Be the first.