Open-Source and Self-Hosted APM Tools for Rails

At FastRuby.io, we always recommend that clients set up an Application Performance Monitor. It’s not just useful for our Tune Report Service, but good practice for visibility into server health, user experience, and more.

One concern that we sometimes hear is that the application may have really sensitive information about the users and it may not be possible to send the information to a third-party provider, or that a managed service can be too expensive. In this article, we are going to explore some open source solutions that can be self-hosted.

The Ingredients (aka Glossary)

APM

An Application Performance Monitor (APM) is a system that provides different insights into the current and past state of the application and different events that have happened over time. There are 3 basic types of data that can be sent to an APM: a trace, a metric, and a log, each with associated metadata.

Trace

A “trace” is associated with an action that has a concrete start and end date and time. We usually generate a trace for each server request, or each background job being processed. A trace must include the duration of that action, the metadata that helps associate it with our application (endpoint, worker, etc), additional metadata for more details (headers, custom information about the current state of the app, etc), and can also include information that adds more levels of detail about the different parts of the processed action called “spans”.

For example: we can have a request GET / trace that includes information about the request parameters, the server that is handling it, the total duration, and more data. We can also then have other spans attached to the main request to show details about the code: a span for each database request, a span for an external API call, a span for some disk heavy operation, etc.

Metric

A “metric” is a singular value for a specific criterion at a given moment in time. Multiple values for the same criterion over time are a Time Series.

For example: the total RAM usage of the rails s process measured at regular intervals.

Log

We are all used to the Rails’ logger so this should not be a new term for most people, but a “log” is a registry of different actions performed by the application written in chronological order. This is typically information about when a request (or other action) starts, what controller is handling it, the received parameters, and any other information the application outputs.

Instrumentation

The very first thing we need when we want to measure performance and monitor our application is a way to actually collect the information from the running Rails application and server(s).

Rails has a built-in instrumentation framework opens a new window that we can use to subscribe to events from the application to then process that information.

It’s important to note that Rails only emits the events, but won’t store or send this information anywhere unless told to do so.

Export, Store and Process Traces

Here’s when we have to add the first piece: we can use OpenTelemetry and the opentelemetry-instrumentation-all gem opens a new window (or we can add only the gems we need from the list of dependencies opens a new window ) that will add listeners for different instrumentation events and will send the information to the tracing system of choice.

In order to store and process traces and spans, we need a tracing system that will receive the information sent by OpenTelemetry and allow the user to query and view the data. Some open-source tracing systems are Jaeger opens a new window and Zipkin opens a new window . OpenTelemetry also provides gems opens a new window to export to some of these tools without having to manually configure all the requests.

So, a basic OpenTelemetry setup for Rails would look like this:

# Gemfile
gem "opentelemetry-sdk"
gem "opentelemetry-instrumentation-all"

# Uncomment depending on the Tracing System you want to use
# gem "opentelemetry-exporter-jaeger"
# gem "opentelemetry-exporter-zipkin"

# in config/initializers/opentelemetry.rb

require "opentelemetry/sdk"
require "opentelemetry/instrumentation/all"

require "opentelemetry/exporter/jaeger"
ENV["OTEL_TRACES_EXPORTER"] = "jaeger"
ENV["OTEL_EXPORTER_JAEGER_ENDPOINT"] = "https://<jaeger_url>/api/v1/spans"

# or, if using Zipkin
# require "opentelemetry/exporter/zipkin"
# Select the zipkin exporter via environmental variables
# ENV["OTEL_TRACES_EXPORTER"] = "zipkin"

ENV["OTEL_SERVICE_VERSION"] = "0.15.0"

OpenTelemetry::SDK.configure do |c|
  c.service_name = "your_app_name"
  c.use_all # enables all available instrumentation!
end

Now we can run Zipkin’s or Jaeger’s servers and use their provided UIs to explore, query, and view traces or spans.

Collect Metrics

Now that we have information about each request and background job being processed, we also need to know information about the current state of the infrastructure at any point in time.

To start collecting metrics, we are going to need new pieces of the puzzle. We can use Prometheus as our metrics and monitoring system, and we can use yabeda-prometheus opens a new window or prometheus_exporter opens a new window to expose a /metrics endpoint that Prometheus can use to get the values of the different metrics periodically.

We can add a basic Yabeda Prometheus setup with this:

# Gemfile
gem "yabeda-prometheus"
# we can add more yabeda gems here if we want to collect more metrics for Rails, Sidekiq, and other parts of the system

# routes.rb

Rails.application.routes.draw do
  mount Yabeda::Prometheus::Exporter, at: "/metrics"
  ...

Aggregate Logs

Similar to the other elements we have described, we need 2 pieces: we have to listen to log events emitted by our application, and we need a service where we send the logs to be processed. We can use rails_loki_exporter opens a new window for the first piece, and loki opens a new window for the second. We can also create a custom Rails Logger by extending the ActiveSupport::Logger class opens a new window and assigning it as the application’s logger opens a new window , while making sure to send the logs to Loki’s URL.

Visualization

So far we’ve had to use one UI for traces, another UI for metrics, and we have no UI for the logs. We can now add the final piece of the puzzle: a unified UI that will connect to the other services and provide dashboards and interfaces to explore traces, metrics, logs, and alerts.

For this, we can use Grafana opens a new window , which only takes care of visualizing the data by querying information exposed by the other services.

Another popular UI that adds dashboards and unifies all the monitoring is Kibana opens a new window from the Elastic stack.

Elastic and Grafana stacks

In the previous sections we only mentioned Kibana and Grafana at the end as UIs, but both the Elastic stack opens a new window and the Grafana stack also provide solutions for tracing, metrics and logs. With the Grafana stack, we still need to add some instrumentation like opentelemetry, but the Elastic stack can be used on its own for a complete observability solution.

Mounted Rails Apps

rails_performance

We didn’t want to finish without mentioning the rails_performance gem opens a new window that can be added to a Rails application for a much simpler setup that can be good enough for smaller applications, without the complications of adding extra services, while still providing traces, metrics, custom events, and more. This stores the information in Redis. Something to keep in mind here is that this will be part of the Rails application, requiring more resources and extra work if scaling with more machines is needed.

solid_telemetry

As an alternative, if we need the information to be persistent, we can use a new gem called solid_telemetry opens a new window (still in alpha), that uses a Sqlite database (inspired by the Solid trifecta from Rails) and provides traces, metrics, and custom events.

Conclusion

We didn’t cover monitoring of the frontend side of the application, for which we can use tools like Grafana Faro opens a new window , or Elastic RUM to capture frontend errors, web vitals and more.

It is important to note that there’s no single best solution, while Elastic can provide all the elements of the stack, other tools can be cheaper to host or provide extra flexibility.

At the end of the day, what’s important is that we keep track of what’s happening in our application so we can make informed decisions (do we need to scale? where do performance bottlenecks currently exist?), debug production problems, and also identify bugs before they happen (with alerts and time series using metrics).

Do you have an Application Performance Monitor configured? We can help!