feat: opt-in Datadog instrumentation of every KB call (1.2.0)

[fatbeard2]

Basecamp: https://app.basecamp.com/3934852/buckets/27820160/todos/10331579143

Why?

After Funnel moved to barkibu-kb 1.0.0, connect timeouts against KB (Faraday::ConnectionFailed: Failed to open TCP connection to knowledge-base-production.herokuapp.com:443 (execution expired)) reached Rollbar but never the Tech ops dashboard. The kb.client.errors span metric is built on the apps' *-http spans, which come from the Datadog Net::HTTP tracer patching Net::HTTP#request. faraday-net_http 1.0.1 opens the socket with http.start before that method, so a connect failure produces no http span at all. The exception lands on whatever span is the parent (the rails.cache span for reads through KB::Cache, the controller or Sidekiq span for writes), and when the app rescues it, Datadog sees nothing.

We were observing KB calls by instrumenting the transport underneath the gem. What we saw depended on the adapter, the Faraday version and the tracer's patch point. Findings and traces: https://app.basecamp.com/3934852/buckets/27820160/todos/10192244285#__recording_10331530328

Changes

• KB::Client: every public method (request, all, find, create, update, destroy, upsert) now goes through one private perform seam, which wraps the call in an ActiveSupport::Notifications event, request.kb_client (KB::Client::REQUEST_EVENT). Payload: verb, path, base_url, cache_hit (GET only), status, plus ActiveSupport's exception/exception_object when the call raised. Same cache keys, same params, same error classes; the existing 176 examples pass unchanged. Anything can subscribe (a log line, StatsD, a test); the gem itself has no Datadog dependency.
• New opt-in KB::Instrumentation::Datadog (lib/kb/instrumentation/datadog.rb): a start/finish subscriber to that event. subscribe! registers it once; the span opens on event start and closes on finish, so the tracer's Net::HTTP spans nest under it. No prepend, no class modification, so it composes with connected_health's circuit-breaker prepend without ordering concerns. The span:
  • no service: given, so the span inherits the app's service (c.service); nothing new appears in the APM service list
  • resource is low-cardinality: GET /v1/pets/birthdays, GET /v1/pets/?/contracts (UUID and numeric segments collapsed to ?)
  • tags: peer.hostname (which KB host the caller used), kb.method, kb.cache_hit (GET only), http.status_code (also on 4xx/5xx raises), plus Datadog's own error.type/error.message when the call raises. Deliberately no span.kind:client and no peer.service: the span covers the client's whole call (cache lookup, connect, parse), so Datadog must not infer it onto the knowledge-base service page as one of KB's operations (verified on staging: with those tags it did)
  • works with ddtrace 1.x (Funnel) and datadog 2.x (Global Admin, connected_health); the tracer gem stays the app's dependency (datadog ~> 2.0 is a development dependency here for the specs only)
  • with tracing disabled (Funnel and Global Admin outside production) the tracer yields a placeholder span and the client behaves exactly as before
• Version 1.2.0, CHANGELOG, README section "Datadog instrumentation".

How to test

docker compose run --rm kb bundle exec rspec
docker compose run --rm kb bundle exec rubocop lib spec

spec/instrumentation/datadog_spec.rb runs the real tracer in test mode with a transport that discards traces, and covers: resource templating, a successful GET, cache hit on the second find, a 404 (span error with http.status_code:404), a POST, tracing disabled, and the motivating case: a connect failure against a closed local port, recorded on the kb.client.request span with error.type:Faraday::ConnectionFailed.

Rollout

Apps opt in with two lines after Datadog.configure:

require 'kb/instrumentation/datadog'
KB::Instrumentation::Datadog.subscribe!

Global Admin and Funnel first, then repoint the kb.client.errors span metric to operation_name:kb.client.request status:error and the Tech ops dashboard widget.

🤖 Generated with Claude Code