Wiring an LLM into a Rails app takes a handful of lines. Understanding what it actually costs you (feature by feature, user by user) is harder. Most providers and SDKs already report tokens, latency, and even cost, but that data lives in their dashboard. It’s disconnected from your requests, your users, and the feature that made the call. And it sits apart from the APM and logs where you already watch the rest of your app.

In a previous post, we introduced Rails.event.notify(...) opens a new window , the tool-agnostic Event Reporter shipping in Rails 8.1. In this post, we’ll put it to work on a real problem: instrumenting every LLM call in your app so token usage, latency, and cost become structured events you can log, graph, and forward to any APM or data warehouse.

At OmbuLabs.ai, we’ve explored the importance of identifying meaningful AI opportunities opens a new window before selecting a solution. Once a worthwhile opportunity has been identified, however, a new question emerges:

Is this problem worth solving in the first place?

Too often, teams focus on the technology before evaluating the value. AI can automate tasks, generate content, and process information at incredible speed, but if the underlying work doesn’t matter, making it faster won’t create meaningful business outcomes.

Once a worthwhile opportunity has been identified, however, a new question emerges:

What should we build first?