Centralized Configuration Management

Flyte uses a centralized configuration system built on top of flytestdlib/config that provides type-safe, hierarchical settings management. This system allows different components to register their own configuration sections, which can then be populated from files, environment variables, or command-line flags.

Defining and Registering Configuration Sections

To add configuration to a Flyte component, you define a struct that represents your settings and register it as a Section. Registration is typically done at the package level using MustRegisterSection to ensure the section is available as soon as the package is initialized.

import "github.com/flyteorg/flyte/v2/flytestdlib/config"

type MyComponentConfig struct {
    Endpoint string `json:"endpoint"`
    Retries  int    `json:"retries"`
}

var (
    // Section keys are case-insensitive and must be unique across the application.
    configSection = config.MustRegisterSection("my_component", &MyComponentConfig{
        Retries: 3, // Default values
    })
)

func GetConfig() *MyComponentConfig {
    return configSection.GetConfig().(*MyComponentConfig)
}

Internally, a Section acts as a node in a configuration tree. When you call MustRegisterSection, Flyte adds a new entry to the rootSection (a global instance of Section). The Config type is a simple alias for interface{}, allowing any struct to be used as a configuration container.

Type-Safe Configuration Wrappers

Standard Go types like time.Duration or url.URL do not always unmarshal cleanly from JSON or YAML strings (e.g., "5m" or "http://localhost:8080"). Flyte provides specialized wrappers in flytestdlib/config that implement UnmarshalJSON and Set (for PFlags) to handle these conversions automatically.

Duration

The Duration struct wraps time.Duration and supports parsing strings like "20s" or "1h".

type Config struct {
    Timeout config.Duration `json:"timeout"`
}

URL

The URL struct wraps net/url.URL and validates that the provided string is a valid URL during unmarshaling.

type Config struct {
    BaseURL config.URL `json:"base-url"`
}

Port

The Port struct ensures that a port number is a valid integer between 0 and 65535. It can unmarshal from both numeric values and strings.

type Config struct {
    ListenPort config.Port `json:"listen-port"`
}

Command Line Integration with PFlags

If you want your configuration settings to be overridable via command-line flags, your config struct can implement the PFlagProvider interface. This allows the configuration system to automatically bind CLI flags to your struct fields.

import "github.com/spf13/pflag"

func (c MyComponentConfig) GetPFlagSet(prefix string) *pflag.FlagSet {
    cmdFlags := pflag.NewFlagSet("MyComponent", pflag.ExitOnError)
    cmdFlags.StringVar(&c.Endpoint, prefix+"endpoint", "localhost", "The service endpoint")
    cmdFlags.IntVar(&c.Retries, prefix+"retries", 3, "Number of retries")
    return cmdFlags
}

When the Accessor initializes flags, it traverses the registered sections and calls GetPFlagSet on any section that implements PFlagProvider.

Loading Configuration via Accessor

The Accessor interface is the primary mechanism for loading configuration into the registered sections. It handles file discovery, environment variable mapping, and flag initialization.

func main() {
    ctx := context.Background()
    
    // Options define how the configuration should be discovered and parsed.
    options := config.Options{
        StrictMode:  true, // Fail if the config file contains unknown keys
        SearchPaths: []string{"/etc/flyte/config", "."},
    }

    // AccessorProvider is typically implemented by a Viper-backed provider.
    accessor := config.GetConfigProvider()(options)

    // Initialize flags (e.g., with pflag)
    accessor.InitializePflags(pflag.CommandLine)
    pflag.Parse()

    // Load the config from files and environment variables
    if err := accessor.UpdateConfig(ctx); err != nil {
        panic(err)
    }
}

The Accessor uses the SearchPaths in Options to look for config.yaml (or other supported formats). If StrictMode is enabled, the UpdateConfig call will return an error if the configuration file contains keys that do not correspond to any registered Section.

Runtime Updates and Handlers

Flyte supports reacting to configuration changes at runtime. When registering a section, you can provide a SectionUpdated callback that is triggered whenever the configuration is refreshed (e.g., via a file change or a manual call to UpdateConfig).

In flytestdlib/logger/config.go, this pattern is used to update the global logger level whenever the configuration changes:

var (
    configSection = config.MustRegisterSectionWithUpdates(configSectionKey, defaultConfig, 
        func(ctx context.Context, newValue config.Config) {
            // newValue is the unmarshaled struct for this section
            onConfigUpdated(*newValue.(*Config))
        },
    )
)

The Section interface tracks whether a configuration has changed using an atomic bit. The GetConfigChangedAndClear() method allows the system to determine if an update handler needs to be invoked by comparing the new configuration against the existing one using reflect.DeepEqual.