Configuring Logging and Formatters

When you need to adjust the verbosity of a Flyte service or ensure logs are correctly parsed by cloud providers like Google Cloud Platform, you configure the global logger using the logger.Config struct. Flyte uses a centralized logging system built on top of logrus that supports multiple output formats and context-aware logging.

Configuring the Global Logger

To initialize or update the logger configuration programmatically, use the logger.SetConfig function. This updates the global state used by all logging calls in the application.

import (
	"context"
	"github.com/flyteorg/flyte/v2/flytestdlib/logger"
)

func main() {
	ctx := context.Background()

	// Configure the logger for production JSON output
	err := logger.SetConfig(&logger.Config{
		Level:             logger.InfoLevel,
		IncludeSourceCode: false,
		Formatter: logger.FormatterConfig{
			Type: logger.FormatterJSON,
		},
	})
	if err != nil {
		// Handle configuration error
		panic(err)
	}

	logger.Infof(ctx, "Logger initialized with level: %v", logger.GetConfig().Level)
}

Configuration via YAML

Flyte services typically manage this configuration through a Logger section in their configuration files (e.g., config.yaml):

logger:
  level: 4
  show-source: false
  mute: false
  formatter:
    type: "json"

Adjusting Log Severity Levels

The logger.Level type defines the minimum severity required for a message to be logged. Flyte defines the following levels in flytestdlib/logger/config.go:

Constant	Value	Description
logger.PanicLevel	0	Highest severity. Logs the message and then calls panic().
logger.FatalLevel	1	Logs the message and calls os.Exit(1).
logger.ErrorLevel	2	Used for errors that require attention.
logger.WarnLevel	3	Non-critical entries that deserve investigation.
logger.InfoLevel	4	General operational entries (Default).
logger.DebugLevel	5	Very verbose logging for development and troubleshooting.

Choosing a Formatter

Flyte supports three primary FormatterType values defined in flytestdlib/logger/config.go:

JSON Formatter (Default)

Sets type: "json". This produces structured logs suitable for most log aggregation systems. It uses ts as the timestamp key and disables HTML escaping by default.

Text Formatter

Sets type: "text". This produces human-readable logs, which is useful during local development.

GCP Formatter

Sets type: "gcp". This uses the GcpFormatter class to produce logs compatible with GCP Stackdriver/Cloud Logging. It maps Flyte log levels to specific GcpSeverity strings:

// Mapping found in flytestdlib/logger/gcp_formatter.go
var logrusToGcp = map[logrus.Level]GcpSeverity{
    logrus.DebugLevel: GcpSeverityDebug,    // "DEBUG"
    logrus.InfoLevel:  GcpSeverityInfo,     // "INFO"
    logrus.WarnLevel:  GcpSeverityWarning,  // "WARNING"
    logrus.ErrorLevel: GcpSeverityError,    // "ERROR"
    logrus.FatalLevel: GcpSeverityCritical, // "CRITICAL"
    logrus.PanicLevel: GcpSeverityAlert,    // "ALERT"
}

Performance and Debugging Options

Including Source Code Location

Setting IncludeSourceCode: true (or show-source: true in YAML) adds a src field to every log entry containing the file and line number (e.g., config_test.go:25).

Warning: This feature uses runtime.Caller(3) to determine the call site. This incurs a performance penalty and is only recommended for debug or development builds.

Muting Logs

The Mute field in logger.Config allows you to suppress all log output, including panics.

cfg := &logger.Config{
    Mute: true, // Suppresses all output
}

Warning: This is intended primarily for benchmarks and unit tests. Using this in production will prevent any error visibility, including critical system panics.

Context-Aware Logging

Once configured, use the context-aware functions to ensure log fields stored in the context.Context (such as request IDs) are included in the output.

func Process(ctx context.Context) {
    // Standard info log
    logger.Info(ctx, "Starting process")

    // Formatted error log
    if err := doWork(); err != nil {
        logger.Errorf(ctx, "Process failed: %v", err)
    }
}

These functions retrieve the current configuration via logger.GetConfig() and check logger.IsLoggable() before dispatching to the underlying logrus.StandardLogger().