cinc-otel

OpenTelemetry run tracing for Cinc Client.

cinc-otel emits an OpenTelemetry trace for each Cinc Client run: a chef.run root span, a child span for each run phase (node load, cookbook sync, compilation, converge, ...), and one span per resource action. The trace makes it easy to see where run time is spent and to correlate failed or slow runs with the rest of your distributed tracing in a backend such as Tempo or Jaeger.

It is implemented as a Chef::EventDispatch handler. The event stream is the only interface that delivers live, paired start/complete events for every phase and resource action, which map one-to-one onto span open/close with real wall-clock timestamps.

Requirements

cinc-otel is loaded inside a cinc-client process and relies on Chef's Chef::EventDispatch::Base, Chef::Log, and Chef::Config already being loaded. It does not depend on the chef gem itself and does not pull it in; Chef is expected to be present in the runtime that loads this gem. The gem is tested against Cinc Client 18 and 19.

The OpenTelemetry SDK and OTLP exporter (opentelemetry-sdk, opentelemetry-exporter-otlp) are runtime dependencies and are required when the handler loads, so they are always available.

Installation

Add it to the gems available to cinc-client. For example, in a Cinc installation that uses cinc gem install:

cinc gem install cinc-otel

Or add it to your bundle:

gem "cinc-otel"

Enabling tracing

Add the following to client.rb (or solo.rb):

require "cinc-otel"
CincOtel::Handler.register!

register! is idempotent: applications such as chef-solo evaluate the config file more than once, and only one handler is ever registered.

The exporter and endpoint are configured through the standard OTEL_* environment variables, for example:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

If span emission ever raises mid-run, the handler disables itself for the rest of the run rather than letting the error propagate — tracing can never break a converge.

If something else in the process has already configured the global OpenTelemetry SDK before the handler loads, that configuration (including its resource attributes) is used as-is.

Service name and tenant

The service name defaults to chef-client and can be overridden with OTEL_SERVICE_NAME.

Spans can be tagged with a __tenant resource attribute (used by Tempo multi-tenancy for search and tail-sampling routing) either through the standard env var:

OTEL_RESOURCE_ATTRIBUTES=__tenant=controlplane

or explicitly at registration, which takes precedence over the env var:

CincOtel::Handler.register!(tenant: "controlplane")

Connecting to a parent trace

When cinc-client is invoked by an OpenTelemetry-aware caller (a CI runner, otel-cli, an orchestrator), export the standard W3C Trace Context environment variables before the run and the chef.run span — and the whole run tree beneath it — will nest under the caller's span instead of starting a new trace:

export TRACEPARENT=00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
export TRACESTATE=vendor=value   # optional
cinc-client

TRACEPARENT/TRACESTATE are the W3C Trace Context format emitted by tools such as otel-cli, GitHub Actions, and Buildkite. If TRACEPARENT is absent or malformed, the run starts its own root trace as usual.

Span structure

chef.run                          chef.version, chef.run_id, chef.node.name, chef.environment
├─ chef.registration
├─ chef.node_load
├─ chef.cookbook_resolution
├─ chef.cookbook_clean
├─ chef.cookbook_sync
├─ chef.cookbook_gems
├─ chef.cookbook_compilation
│  ├─ chef.library_load
│  ├─ chef.ohai_plugin_load
│  ├─ chef.compliance_load
│  ├─ chef.attribute_load
│  ├─ chef.lwrp_load
│  ├─ chef.definition_load
│  └─ chef.recipe_load
├─ chef.converge
│  └─ one span per resource action, named e.g. file[/etc/motd]
└─ chef.handlers

Each resource span carries chef.resource.type, chef.resource.name, chef.resource.action, chef.resource.cookbook and chef.resource.recipe, plus outcome detail:

  • chef.resource.updated — true/false for converged resources
  • chef.resource.skipped and chef.resource.skip_reason — for resources skipped by a guard or action :nothing
  • chef.resource.notification_type and chef.resource.notifying_resource — when the action ran due to a notification
  • failed resources record the exception on the span and set error status; retries appear as retry span events

Testing locally

The SDK's console exporter prints spans to stdout, which is handy for verifying output without a collector:

OTEL_TRACES_EXPORTER=console cinc-solo -c solo.rb -o 'recipe[my_cookbook]'

Development

bundle install
bundle exec rake        # runs chefstyle + rspec
bundle exec rspec       # tests only
bundle exec rake style  # lint only

The unit suite runs without a Chef installation beyond the chef dev gem. There is also an end-to-end integration test that runs a real chef-solo/cinc-solo converge and asserts the emitted OpenTelemetry spans over the OTLP exporter. It is excluded from the default run and is skipped unless a solo binary is present:

CINC_OTEL_INTEGRATION=1 bundle exec rspec --tag integration spec/integration
# or
bundle exec rake integration

CI runs it in a dedicated job that installs Cinc Client and installs the built gem into its embedded Ruby.

License

Apache-2.0. See LICENSE.