Instrumenting LLM Calls in Rails with `Rails.event.notify`
Wiring an LLM into a Rails app takes a handful of lines. Understanding what it actually costs you (feature by feature, user by user) is harder. Most providers and SDKs already report tokens, latency, and even cost, but that data lives in their dashboard. It’s disconnected from your requests, your users, and the feature that made the call. And it sits apart from the APM and logs where you already watch the rest of your app.
In a previous post, we introduced Rails.event.notify(...) , the tool-agnostic Event Reporter shipping in Rails 8.1. In this post, we’ll put it to work on a real problem: instrumenting every LLM call in your app so token usage, latency, and cost become structured events you can log, graph, and forward to any APM or data warehouse.
Why instrument LLM calls?
Before writing any code, it helps to be specific about what we’re after. Four things, really. Cost, from the raw token counts up to the dollar figure on the invoice. Latency, so a slow model or a bloated prompt shows up before your users feel it. Failures, because a timeout or a rate limit shouldn’t just vanish. And, most of all, attribution: not just that a call happened, but which feature, user, and request set it off.
ActiveSupport::Notifications could get us there, but we’d be writing custom glue code for every APM vendor. The Event Reporter gives us one structured emission point instead, and lets each subscriber decide what to do with it.
Prerequisites: This tutorial assumes Rails 8.1 (for
Rails.event) and any LLM client that reports token usage. The examples use theruby_llmgem, but the pattern works with the officialanthropic/openaiSDKs or a rawNet::HTTPcall just as well.
The starting point: an uninstrumented service
Here’s a typical, uninstrumented service that summarizes some text:
# app/services/summarizer.rb
class Summarizer
MODEL = "claude-haiku-4-5"
def initialize(text)
@text = text
end
def call
chat = RubyLLM.chat(model: MODEL)
response = chat.ask("Summarize the following text:\n\n#{@text}")
response.content
end
end
It works, but it’s a black box. We have no idea how many tokens that summary consumed, what it cost, or how long the LLM took to respond. Let’s improve that.
Emitting a structured event around the call
The Event Reporter’s core method is Rails.event.notify(name, payload). We wrap the call, measure the duration with a monotonic clock, and emit everything we care about:
# app/services/summarizer.rb
class Summarizer
MODEL = "claude-haiku-4-5"
def initialize(text)
@text = text
end
def call
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
chat = RubyLLM.chat(model: MODEL)
response = chat.ask("Summarize the following text:\n\n#{@text}")
duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
Rails.event.notify(
"llm.completion",
model: MODEL,
input_tokens: response.input_tokens,
output_tokens: response.output_tokens,
total_tokens: response.input_tokens + response.output_tokens,
duration_ms: duration_ms
)
response.content
end
end
Every summary now publishes a single llm.completion event. Notice what we don’t put in the payload: no prompt text, no user content. Keep payloads small and free of PII, just IDs, token counts, and durations.
Listening with a subscriber
An event that no one listens to does nothing. Subscribers are plain Ruby objects that implement an #emit(event) method, where event is a hash containing :name, :payload, :tags, :context, :timestamp, and :source_location.
Let’s start with a subscriber that writes a compact key=value log line and forwards metrics to an APM. It lives under app/subscribers/, so Rails autoloads it:
# app/subscribers/llm_event_subscriber.rb
class LlmEventSubscriber
def emit(event)
return unless event[:name] == "llm.completion"
payload = event[:payload]
Rails.logger.info(
"[llm.completion] model=#{payload[:model]} " \
"tokens=#{payload[:total_tokens]} " \
"duration_ms=#{payload[:duration_ms]}"
)
# Forward to your APM of choice (Datadog, AppSignal, New Relic, etc.):
MyAPM.distribution(
"llm.duration_ms",
payload[:duration_ms],
model: payload[:model]
)
MyAPM.counter(
"llm.total_tokens",
payload[:total_tokens],
model: payload[:model]
)
end
end
Then register an instance once, at boot:
# config/initializers/llm_events.rb
Rails.event.subscribe(LlmEventSubscriber.new)
The same event now lands in your logs and your dashboards, without the service object knowing anything about AppSignal, Datadog, or New Relic. Swap the subscriber’s body and you’ve swapped vendors.
From tokens to dollars
Tokens are only half the story. The number your finance team cares about is cost. Providers charge different rates for input and output tokens: the prompt you send costs less, the text the model generates back costs more. The exact rates vary by model. So we keep a small lookup table that maps each model to its two rates, in the same units the pricing pages use: U.S. dollars per one million tokens.
There’s a catch, though: a hand-maintained table like this goes stale fast. Providers release new models and adjust prices regularly, and every change means another edit to keep in sync. If you’re using ruby_llm, you can skip the table entirely. The gem keeps its own pricing registry and calculates cost for you : a response exposes response.cost.total, and model.cost_for(response.tokens) returns a RubyLLM::Cost broken down by component, input, output, cache_read, cache_write, thinking and the total across all of them. When your client hands you cost directly, emit that value and let the gem take care of staying current.
Not every LLM client reports cost, though, so it’s worth knowing how to compute it yourself. In that case we keep the table and the math inside the subscriber, so cost is computed once and every consumer of the event gets it for free:
# app/subscribers/llm_event_subscriber.rb
class LlmEventSubscriber
# Per-model pricing in USD per 1 million tokens.
# input => rate for the tokens you send (the prompt)
# output => rate for the tokens the model generates back
# Copy these numbers from your provider's pricing page and keep them in sync.
PRICING = {
"claude-haiku-4-5" => { input: 1.00, output: 5.00 },
"claude-sonnet-5" => { input: 3.00, output: 15.00 }
}.freeze
def emit(event)
return unless event[:name] == "llm.completion"
payload = event[:payload]
cost = estimate_cost(payload)
Rails.logger.info(
"[llm.completion] model=#{payload[:model]} " \
"tokens=#{payload[:total_tokens]} " \
"duration_ms=#{payload[:duration_ms]} " \
"cost_usd=#{cost.round(6)}"
)
# The duration_ms and total_tokens metrics are
# omitted to keep the example focused on cost.
MyAPM.distribution(
"llm.cost_usd",
cost,
model: payload[:model]
)
end
private
def estimate_cost(payload)
prices = PRICING[payload[:model]]
return 0.0 unless prices
input_cost = payload[:input_tokens] * prices[:input] / 1_000_000.0
output_cost = payload[:output_tokens] * prices[:output] / 1_000_000.0
input_cost + output_cost
end
end
The registration in the initializer stays exactly the same. Now every completion carries a dollar figure you can sum, chart, and alert on.
Attributing calls to features and users
Aggregate token counts are useful, but “which feature is eating my budget?” and “how much did this user cost this month?” are the questions that actually drive decisions. The Event Reporter answers both with two mechanisms: tags and context.
Tags help identify which feature or operation triggered the call. Wrap the call in Rails.event.tagged, using a keyword so the tag carries a real value rather than just a label:
# app/services/summarizer.rb
def call
Rails.event.tagged(feature: "summarization") do
# ...measure and notify as before...
end
end
Context describes the environment an event happens in, and it’s typically set once per request. A great place is a controller before_action or the base controller:
# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
before_action :set_llm_context
private
def set_llm_context
Rails.event.set_context(
request_id: request.request_id,
user_id: current_user&.id
)
end
end
From now on, every llm.completion event automatically carries the tags and context, with no extra work in the service object. A full event handed to your subscriber now looks like this:
{
name: "llm.completion",
payload: {
model: "claude-haiku-4-5",
input_tokens: 2000,
output_tokens: 500,
total_tokens: 2500,
duration_ms: 842
},
tags: { feature: "summarization" },
context: { request_id: "abc123", user_id: 456 },
timestamp: 1738964843208679035,
source_location: { filepath: "app/services/summarizer.rb", lineno: 14, label: "Summarizer#call" }
}
Update the subscriber to read them:
def emit(event)
return unless event[:name] == "llm.completion"
payload = event[:payload]
# `tags` is a hash. Because we tagged with `feature:`, we can read the value
# straight from the :feature key, with no assumptions about ordering.
feature = event[:tags][:feature]
user_id = event[:context][:user_id]
Rails.logger.info(
"[llm.completion] feature=#{feature} user_id=#{user_id} " \
"model=#{payload[:model]} cost_usd=#{estimate_cost(payload).round(6)}"
)
end
A reusable instrumentation helper
By now we’ll want this measure-and-notify block around every LLM call, not just the summarizer. Let’s extract it into a helper so every call site stays consistent:
# app/services/llm_instrumentation.rb
module LlmInstrumentation
# Wraps any block that returns an object responding to
# #input_tokens and #output_tokens, and emits an llm.completion event.
def self.instrument(model:, feature:)
Rails.event.tagged(feature: feature) do
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
response = yield
duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
Rails.event.notify(
"llm.completion",
model: model,
input_tokens: response.input_tokens,
output_tokens: response.output_tokens,
total_tokens: response.input_tokens + response.output_tokens,
duration_ms: duration_ms
)
response
end
end
end
The service object shrinks back to almost what we started with, now fully instrumented:
# app/services/summarizer.rb
class Summarizer
MODEL = "claude-haiku-4-5"
def initialize(text)
@text = text
end
def call
response = LlmInstrumentation.instrument(model: MODEL, feature: "summarization") do
RubyLLM.chat(model: MODEL).ask("Summarize the following text:\n\n#{@text}")
end
response.content
end
end
Capturing the failures
Surfacing failed calls is just as important as tracking successful ones. Emit a separate event on error so timeouts and rate limits don’t disappear silently:
# app/services/llm_instrumentation.rb
def self.instrument(model:, feature:)
Rails.event.tagged(feature: feature) do
started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
begin
response = yield
rescue => error
duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
Rails.event.notify(
"llm.error",
model: model,
error_class: error.class.name,
duration_ms: duration_ms
)
raise
end
duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
Rails.event.notify(
"llm.completion",
model: model,
input_tokens: response.input_tokens,
output_tokens: response.output_tokens,
total_tokens: response.input_tokens + response.output_tokens,
duration_ms: duration_ms
)
response
end
end
Add a branch to your subscriber for "llm.error", and now your dashboards show a failure rate alongside cost and latency, three basic signals that you need to observe on every LLM-powered feature.
Feeding a data warehouse
Because the payloads are already structured, a warehouse-oriented subscriber is just another #emit implementation. Buffer the events and ship them to your analytics pipeline:
# app/subscribers/warehouse_subscriber.rb
class WarehouseSubscriber
def emit(event)
return unless event[:name].start_with?("llm.")
LlmEventBufferJob.perform_later(
name: event[:name],
payload: event[:payload],
tags: event[:tags],
context: event[:context],
timestamp: event[:timestamp]
)
end
end
Register it alongside the first subscriber, in the same initializer:
# config/initializers/llm_events.rb
Rails.event.subscribe(LlmEventSubscriber.new)
Rails.event.subscribe(WarehouseSubscriber.new)
Once the data lands, per-user cost reports, per-feature budgets, and model A/B comparisons are all just queries.
Best practices for LLM events
A few habits keep this instrumentation clean as it grows. Stick to a clear resource.action namespace, like llm.completion and llm.error, so related events group together. Keep payloads free of PII. Compute cost in the subscriber rather than at the call site, so pricing lives in one place and your service objects don’t need to know about it. Instrument once through a shared helper, so every call site emits consistent events. And always instrument failures, because an error rate is as important as a cost figure.
Conclusion
LLM features don’t have to be black boxes. One Rails.event.notify call in the right spot turns an opaque API request into an event you can log, cost out, graph, or ship to a warehouse, and none of your service objects have to know where it ends up. As AI keeps creeping into your latency budget and your monthly bill, that visibility is what stops it from getting away from you.
Building AI features into your Rails app, or getting to Rails 8.1 so you can use the Event Reporter? Let’s talk .