Context-Aware Labeled Metrics

When building distributed systems with Flyte, granular observability is crucial for understanding system behavior, identifying bottlenecks, and debugging issues. Manually adding labels to every metric can be tedious and error-prone. Flyte addresses this by providing context-aware labeled metrics, which automatically extract relevant label values from the Go context.Context.

This mechanism allows you to define a set of global labels that are automatically applied to your metrics, ensuring consistency and reducing boilerplate. Additionally, you can specify metric-specific labels for even finer-grained control.

Configuring Global Metric Labels

To enable context-aware labeling, you first configure the global metric keys that Flyte should extract from the context.Context. This is typically done once at application startup using flytestdlib.promutils.labeled.SetMetricKeys.

For example, in the cache_service application, SetMetricKeys is called to establish ProjectKey, DomainKey, WorkflowIDKey, and TaskIDKey as global labels:

labeled.SetMetricKeys(contextutils.ProjectKey, contextutils.DomainKey, contextutils.WorkflowIDKey, contextutils.TaskIDKey)

Once these keys are set, any metric initialized with the labeled package will automatically look for corresponding values in the context.Context when its methods (e.g., Inc, Add, Start, Observe) are called. If a key is present in the context, its value will be added as a label to the emitted metric.

Counter Metrics

Counter metrics are used to track events that only ever increase, such as the number of requests served or errors encountered. Flyte