How we configured OpenTelemetry logs in Rails

Recently, we needed to add observability to a Rails project without getting locked into a single vendor. OpenTelemetry was the natural choice.

It generates three kinds of telemetry data, called signals: logs, metrics, and traces. Each signal is independent, so you can adopt one without the others.

All of these signals travel in a standard, vendor-agnostic format known as OTLP (OpenTelemetry Protocol). Because this format is standard across the industry, we can switch backends (like Datadog, New Relic, or Grafana Cloud) in the future without rewriting any application code.

In this post, we will explain how we configured the OpenTelemetry Ruby SDK to export logs directly to our vendor, Grafana Cloud. We will also discuss a few issues we encountered along the way and the upstream fixes we contributed.

Exporting directly to the vendor

The standard OpenTelemetry deployment involves running an OpenTelemetry Collector alongside your application. The Collector is a separate process, usually a sidecar container or a service on the same host. Your application sends telemetry to it over the local network, and the Collector then batches that data and forwards it to the vendor.

To keep our infrastructure simple, we bypassed the Collector and exported logs directly from the Ruby SDK to Grafana Cloud. This is not an unusual setup. The Scout APM logging gem sends logs directly from the app as well.

Our log volume is small, so the SDK's built-in batching is enough. A Collector is still the better choice if you need buffering, sampling, or scrubbing outside the app.

Setting up the Ruby SDK for logs

Add the following gems to your Gemfile:

gem "opentelemetry-sdk"
gem "opentelemetry-logs-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-exporter-otlp-logs"
gem "opentelemetry-instrumentation-all"
gem "opentelemetry-instrumentation-logger"

OpenTelemetry is highly modular, so each gem handles a specific responsibility:

• opentelemetry-sdk: The core OpenTelemetry framework (traces and the configuration entry point).
• opentelemetry-logs-sdk: Adds support for the logging signal (which is separate from traces).
• opentelemetry-exporter-otlp: Exports traces over the network via OTLP.
• opentelemetry-exporter-otlp-logs: Exports logs over the network via OTLP.
• opentelemetry-instrumentation-all: Bundles the instrumentation gems, including Rails, Rack, and Active Record.
• opentelemetry-instrumentation-logger: Hooks into the standard Ruby Logger so log messages become OpenTelemetry log records.

With these gems installed, the exporter needs to know where to send the data. Set your vendor's endpoint and authentication token as environment variables (the SDK automatically detects standard OTLP exporter environment variables):

OTEL_EXPORTER_OTLP_ENDPOINT="https://your-vendor.com/otlp"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-token>"

Finally, we need to initialize the SDK so it starts collecting and exporting data. Create an initializer (config/initializers/opentelemetry.rb):

return if ENV["OTEL_EXPORTER_OTLP_ENDPOINT"].blank?

OpenTelemetry::SDK.configure do |c|
  c.service_name = "our-rails-app"
  c.use_all
end

Here, c.use_all enables every instrumentation that has been required, which means the ones bundled in opentelemetry-instrumentation-all plus opentelemetry-instrumentation-logger.

Testing it locally

Now that the SDK is configured, you can test it by setting OTEL_LOGS_EXPORTER=console. This prints log records in your terminal instead of sending them to the vendor, which is the quickest way to confirm your setup works before deploying.

Issues we found and fixed in the Ruby SDK

When we exported logs to Grafana Cloud, we found two issues where the Ruby SDK behaved differently than other language SDKs and the OpenTelemetry specification.

1. Exporter dropped the base path

Some vendor backends require sending OTLP data to an endpoint with a specific base path, such as /otlp in our case with Grafana Cloud, but the exporter dropped it while appending the signal path.

Endpoint:  https://your-vendor.com/otlp
Expected:  https://your-vendor.com/otlp/v1/logs
Actual:    https://your-vendor.com/v1/logs

We reported this in issue #2157 and fixed it in PR #2158, which was released in opentelemetry-exporter-otlp-logs v0.5.1.

2. Handling HTTP 204 responses

Grafana Cloud returns 204 No Content after ingesting logs, but the exporter only treated 200 OK as success. The exports actually succeeded, but our app logged each one as a failure.

We raised issue #2043 and fixed it in PR #2044, which was released in opentelemetry-exporter-otlp-logs v0.4.0.

Next steps: structured logging

Now that logs are being successfully exported, the next logical step is structured logging. That's too much to cover here, but the rails_semantic_logger gem is a great place to start, and we might even cover the full OpenTelemetry setup for it in a future post!