Instrumenting Code with Prometheus Metrics

Flyte uses the promutils package to manage Prometheus metrics through a hierarchical Scope system. This approach ensures that metrics are automatically prefixed and sanitized, allowing different components to define metrics without worrying about global name collisions.

Creating and Nesting Scopes

To begin instrumenting code, you create a root Scope and then derive sub-scopes for specific components.

import (
    "github.com/flyteorg/flytestdlib/promutils"
)

// Create a root scope
rootScope := promutils.NewScope("flyte_admin")

// Create a sub-scope for a specific component
// This will prefix all metrics with "flyte_admin:request_handler:"
handlerScope := rootScope.NewSubScope("request_handler")

The Scope interface handles metric name sanitization automatically. For example, characters like - are converted to _ to comply with Prometheus naming conventions.

Implementing Counters and Gauges

Standard Prometheus metrics like Counter and Gauge are created directly from a Scope. Use the MustNew* variants during initialization to ensure the application panics if metric registration fails (e.g., due to a duplicate name).

type Metrics struct {
    SuccessCounter prometheus.Counter
    ActiveRequests prometheus.Gauge
}

func NewMetrics(scope promutils.Scope) Metrics {
    return Metrics{
        // Creates "flyte_admin:success"
        SuccessCounter: scope.MustNewCounter("success", "Count of successful operations"),
        // Creates "flyte_admin:active_requests"
        ActiveRequests: scope.MustNewGauge("active_requests", "Current number of active requests"),
    }
}

Timing Operations with StopWatch

Flyte provides a StopWatch wrapper around Prometheus Summaries to simplify timing code blocks. When you create a StopWatch, you specify a time.Duration scale (e.g., time.Millisecond), and the metric name is automatically suffixed with the scale (e.g., _ms).

func ProcessData(scope promutils.Scope) {
    // Creates a summary metric named "flyte_admin:process_duration_ms"
    stopWatch := scope.MustNewStopWatch("process_duration", "Time taken to process data", time.Millisecond)

    // Start the timer
    timer := stopWatch.Start()
    
    // Defer Stop() to record the duration when the function exits
    defer timer.Stop()

    // Perform the operation
    doWork()
}

Using HistogramStopWatch for Aggregation

If you need to aggregate quantiles across multiple instances (which standard Summaries do not support well), use HistogramStopWatch. It uses a Prometheus Histogram backend and defaults to a scale of seconds.

// Creates a histogram metric named "flyte_admin:api_latency"
histStopWatch := scope.MustNewHistogramStopWatch("api_latency", "API request latency")

timer := histStopWatch.Start()
defer timer.Stop()

Configuring Histograms and Summaries

For fine-grained control over metric buckets or quantiles, use HistogramOptions or SummaryOptions.

Custom Histogram Buckets

You can define custom buckets for a HistogramVec using HistogramOptions.

latencyBuckets := []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}

requestLatency := scope.MustNewHistogramVecWithOptions(
    "k8s_client_request_latency",
    "Kubernetes client request latency in seconds",
    promutils.HistogramOptions{Buckets: latencyBuckets},
    "verb", // Label name
)

// Usage with labels
requestLatency.WithLabelValues("GET").Observe(0.15)

Custom Summary Objectives

Use SummaryOptions to define specific quantiles and their allowed error margins.

options := promutils.SummaryOptions{
    Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
}

summary := scope.MustNewSummaryWithOptions("data_size", "Distribution of data sizes", options)
summary.Observe(1024)

Using Metric Vectors

Metric vectors allow you to add dimensions to your metrics using labels. Flyte supports vectors for all standard types as well as StopWatchVec.

// Create a StopWatchVec with a "method" label
apiTimerVec := scope.MustNewStopWatchVec("api_call", "Duration of API calls", time.Millisecond, "method")

func CallAPI(method string) {
    // Start a timer for a specific label value
    timer := apiTimerVec.WithLabelValues(method).Start()
    defer timer.Stop()
    
    // ... API logic ...
}

Troubleshooting and Gotchas

• Metric Name Suffixes: StopWatch and StopWatchVec automatically append a scale suffix (like _ms or _s) to the metric name. Do not include these suffixes manually in the name parameter.
• Sanitization: The Scope implementation replaces hyphens (-) with underscores (_) and removes other invalid characters. If your metric name looks different in Prometheus than in your code, check the SanitizeMetricName logic in flytestdlib/promutils/scope.go.
• Registration Panics: MustNew* methods will panic if you attempt to register a metric with a name that already exists in the same scope. Ensure metrics are initialized once (usually in a constructor) rather than inside a loop or request handler.
• StopWatch Creation: Never instantiate StopWatch or HistogramStopWatch structs manually. Always use the factory methods on a Scope to ensure proper registration and scaling.