Skip to main content

Distributed Tracing with OpenTelemetry

Flyte uses the otelutils package in flytestdlib to provide a unified interface for distributed tracing and metrics across its microservices. This infrastructure allows you to track requests as they flow through components like FlyteAdmin, FlytePropeller, and the DataProxy, using standard OpenTelemetry (OTel) protocols.

Configuring OpenTelemetry​

You configure tracing and metrics through the otel configuration section. This section defines which exporter to use (e.g., Jaeger or OTLP) and how to sample traces.

The following example shows a typical configuration for exporting traces to a Jaeger collector using a 10% sampling rate:

otel:
  type: jaeger
  jaeger:
    endpoint: http://jaeger-collector.flyte.svc.cluster.local:14268/api/traces
  sampler:
    parentSampler: traceid
    traceIdRatio: 0.1

Supported Exporters​

The ExporterType (defined in flytestdlib/otelutils/config.go) determines where telemetry data is sent:

  • noop: Disables all tracing and metrics (default).
  • file: Writes traces to a local file (configured via FileConfig.Filename).
  • jaeger: Sends traces to a Jaeger collector endpoint.
  • otlpgrpc: Uses the OTLP protocol over gRPC.
  • otlphttp: Uses the OTLP protocol over HTTP.

Sampling Strategies​

Sampling is managed via SamplerConfig. Flyte supports two primary SamplerType values:

  • always: Every request is traced.
  • traceid: Traces a percentage of requests based on the TraceIDRatio.

Internally, otelutils wraps these in a ParentBased sampler. This ensures that if a parent span is already being sampled, the child spans in Flyte will also be sampled regardless of the local ratio.

Initializing the Providers​

To enable tracing in a service, you must register the tracer and meter providers during startup. Use RegisterProvidersWithContext to initialize the global state based on your configuration.

import (
    "github.com/flyteorg/flyte/v2/flytestdlib/otelutils"
)

func main() {
    ctx := context.Background()
    cfg := otelutils.GetConfig()
    serviceName := "flyteadmin"

    err := otelutils.RegisterProvidersWithContext(ctx, serviceName, cfg)
    if err != nil {
        // Handle initialization error
    }
}

The RegisterProvidersWithContext function in flytestdlib/otelutils/factory.go performs several key tasks:

  1. Instantiates the appropriate SpanExporter and MetricExporter.
  2. Creates a telemetryResource that automatically includes service information and versioning.
  3. Sets up a TracerProvider with a Batcher for efficient exporting.
  4. Stores the providers in a global map indexed by the service name.

Manual Instrumentation​

When you need to trace specific logic within a function, use the NewSpan helper. This function simplifies span creation by automatically extracting log fields from the context and adding them as span attributes.

func (s *myService) DoWork(ctx context.Context) error {
    // Creates a span named "myService.DoWork" for the "flyteadmin" tracer
    ctx, span := otelutils.NewSpan(ctx, "flyteadmin", "myService.DoWork")
    defer span.End()

    // ... implementation logic ...
    return nil
}

NewSpan uses contextutils.GetLogFields(ctx) to find metadata (like project, domain, or workflowID) and attaches them to the span, ensuring that your traces are searchable by the same dimensions as your logs.

Automatic Integrations​

Flyte provides built-in wrappers to automatically trace common infrastructure components.

Kubernetes Client Tracing​

If your service interacts with the Kubernetes API, you can wrap the standard controller-runtime client to get automatic spans for every Get, List, Create, and Update call.

import "github.com/flyteorg/flyte/v2/flytestdlib/otelutils"

// Wrap an existing k8s client
tracedClient := otelutils.WrapK8sClient(k8sClient)

// Every call now generates a span under the "k8s-client" tracer
err := tracedClient.Get(ctx, key, obj)

The K8sClientWrapper in flytestdlib/otelutils/k8s.go intercepts these calls and creates spans with the prefix controller-runtime.pkg.client.

RPC Tracing with ConnectRPC​

For services using ConnectRPC (like the Flyte app service), you can inject the OTel providers into interceptors to trace incoming and outgoing RPC calls:

otelInterceptor, err := otelconnect.NewInterceptor(
    otelconnect.WithTracerProvider(otelutils.GetTracerProvider("flyte-service")),
    otelconnect.WithMeterProvider(otelutils.GetMeterProvider("flyte-service")),
)

This ensures that the trace context is propagated across network boundaries between Flyte services.