Context-Aware Labeled Metrics

When you need to track metrics like request failures or operation latencies across different Flyte projects, domains, or workflows, manually passing labels to every metric call is error-prone. Flyte provides a context-aware labeled metrics system in the labeled package that automatically extracts these labels from a context.Context.

Global Configuration

Before instantiating any labeled metrics, you must define which context keys should be mapped to Prometheus labels. This is typically done once at the application entry point (e.g., main.go).

import (
    "github.com/flyteorg/flyte/v2/flytestdlib/contextutils"
    "github.com/flyteorg/flyte/v2/flytestdlib/promutils/labeled"
)

func main() {
    // Define global keys that will be extracted from context for ALL labeled metrics
    labeled.SetMetricKeys(
        contextutils.ProjectKey,
        contextutils.DomainKey,
        contextutils.WorkflowIDKey,
        contextutils.TaskIDKey,
    )
}

[!CAUTION] labeled.SetMetricKeys must be called exactly once before any labeled metrics are created. Calling it multiple times with different keys will cause a panic.

Defining and Instantiating Metrics

Labeled metrics are usually grouped within a struct. You can use labeled.Counter, labeled.Gauge, labeled.Summary, and labeled.HistogramStopWatch.

import (
    "github.com/flyteorg/flyte/v2/flytestdlib/promutils"
    "github.com/flyteorg/flyte/v2/flytestdlib/promutils/labeled"
)

type MyServiceMetrics struct {
    Scope           promutils.Scope
    RequestCount    labeled.Counter
    ActiveRequests  labeled.Gauge
    Latency         labeled.HistogramStopWatch
}

func NewMyServiceMetrics(scope promutils.Scope) *MyServiceMetrics {
    return &MyServiceMetrics{
        Scope: scope,
        // NewCounter creates a metric that will look for the global keys in the context
        RequestCount: labeled.NewCounter("requests_total", "Total requests handled", scope),
        
        // NewGauge tracks current state
        ActiveRequests: labeled.NewGauge("active_requests", "Current active requests", scope),
        
        // NewHistogramStopWatch is used for timing operations
        Latency: labeled.NewHistogramStopWatch("request_latency", "Latency of requests", scope),
    }
}

Using Metrics with Context

To emit a metric with labels, ensure the context contains the required metadata using contextutils.

func (s *MyService) HandleRequest(ctx context.Context) {
    // 1. Inject metadata into context (usually done in middleware or upstream)
    ctx = contextutils.WithProjectDomain(ctx, "flytesnacks", "development")
    ctx = contextutils.WithWorkflowID(ctx, "my_workflow")

    // 2. Use the metric - labels are automatically extracted from ctx
    s.metrics.RequestCount.Inc(ctx)
    
    // 3. Timing an operation
    timer := s.metrics.Latency.Start(ctx)
    defer timer.Stop()
    
    // ... perform work ...
}

Advanced Metric Options

The labeled package provides MetricOption to customize how metrics are emitted.

Tracking System-Wide Totals

If you want to track both granular labeled metrics and a global "unlabeled" total (to avoid expensive Prometheus aggregations), use labeled.EmitUnlabeledMetric. This creates an additional metric with the suffix _unlabeled.

// Found in flytestdlib/storage/stow_store.go
metrics := &stowMetrics{
    ReadFailure: labeled.NewCounter("read_failure", "Indicates failure in GET", scope, labeled.EmitUnlabeledMetric),
}

When metrics.ReadFailure.Inc(ctx) is called, it increments:

1. read_failure{project="...", domain="..."}
2. read_failure_unlabeled (no labels)

Adding Metric-Specific Labels

If a specific metric requires labels beyond the global set, use labeled.AdditionalLabelsOption.

// Found in flytestdlib/storage/stow_store.go
failureTypeOption := labeled.AdditionalLabelsOption{Labels: []string{"failure_type"}}

readFailure := labeled.NewCounter("read_failure", "Indicates failure in GET", scope, failureTypeOption)

Best Practices and Gotchas

High Cardinality

Avoid adding high-cardinality keys like contextutils.ExecIDKey or contextutils.RequestIDKey to the global SetMetricKeys unless absolutely necessary. This can significantly increase the memory footprint of your Prometheus instance.

Initialization Order

If you attempt to instantiate a labeled metric before calling SetMetricKeys, the application will panic with ErrNeverSet.

// flytestdlib/promutils/labeled/counter.go:53
if len(metricKeys) == 0 {
    panic(ErrNeverSet)
}

Context Defaults

If a key defined in SetMetricKeys is missing from the context provided at runtime, the labeled package uses an empty string "" as the label value. This prevents panics during metric emission but may lead to "empty" labels in your dashboard. Use contextutils.Values(ctx, keys...) to verify what values are being extracted.