Advanced Configuration: Viper and Complex Types

Flyte uses a sophisticated configuration system built on top of the spf13/viper library, extended to handle complex data structures, multi-file merging, and custom types. The core of this implementation resides in the flytestdlib/config/viper package, which bridges Flyte's internal configuration abstractions with the Viper ecosystem.

The Viper Accessor

The viperAccessor class in flytestdlib/config/viper/viper.go is the primary implementation of the config.Accessor interface. It manages the lifecycle of configuration loading, including environment variables, command-line flags (pflags), and file watching.

When you initialize a new accessor using viper.NewAccessor(opts config.Options), Flyte searches for configuration files in the provided SearchPaths. It then creates a viperAccessor that wraps these files into a CollectionProxy.

func newAccessor(opts config.Options) *viperAccessor {
	vipers := make([]Viper, 0, 1)
	configFiles := files.FindConfigFiles(opts.SearchPaths)
	for _, configFile := range configFiles {
		v := viperLib.New()
		v.SetConfigFile(configFile)
		vipers = append(vipers, v)
	}
    // ...
	return &viperAccessor{
		strictMode:         opts.StrictMode,
		rootConfig:         r,
		viper:              &CollectionProxy{underlying: vipers},
		watcherInitializer: &sync.Once{},
	}
}

Multi-File Merging with CollectionProxy

The CollectionProxy class in flytestdlib/config/viper/collection.go allows Flyte to treat multiple Viper instances as a single configuration source. This is critical for scenarios where a base configuration is provided by the system, but a user needs to provide local overrides.

The MergeAllConfigs method iterates through all underlying Viper instances and merges them into a single viperLib.Viper object. It also ensures that environment variables and pflags are correctly bound to the final merged configuration.

func (c CollectionProxy) MergeAllConfigs() (all Viper, err error) {
	combinedConfig := viperLib.New()
    // ... bind env and pflags ...
	for _, v := range c.underlying {
		if len(v.ConfigFileUsed()) == 0 {
			continue
		}
		combinedConfig.SetConfigFile(v.ConfigFileUsed())
		reader, err := os.Open(v.ConfigFileUsed())
		if err != nil {
			return nil, err
		}
		err = combinedConfig.MergeConfig(reader)
		if err != nil {
			return nil, err
		}
	}
	return combinedConfig, nil
}

Handling Complex Types

Standard Viper has limitations when dealing with case-sensitive map keys and custom Go types. Flyte overcomes these using custom mapstructure decode hooks defined in flytestdlib/config/viper/viper.go.

Case-Sensitive Maps: sliceToMapHook

Viper typically folds all keys to lowercase, which breaks maps where keys are case-sensitive (such as environment variable names or specific resource IDs). Flyte provides a workaround where maps can be defined as a list of single-key maps in YAML. The sliceToMapHook then converts this slice back into a single map.

func sliceToMapHook(f reflect.Kind, t reflect.Kind, data interface{}) (interface{}, error) {
	if f == reflect.Slice && t == reflect.Map {
		res := map[interface{}]interface{}{}
		asSlice := data.([]interface{})
		for _, item := range asSlice {
			if asMap, casted := item.(map[interface{}]interface{}); casted {
				for key, value := range asMap {
					res[key] = value
				}
			}
		}
		return res, nil
	}
	return data, nil
}

Custom Types: jsonUnmarshallerHook

For types that require custom parsing logic, such as config.Duration or config.URL, Flyte uses the jsonUnmarshallerHook. This hook checks if a target type implements the json.Unmarshaler interface. If it does, Flyte marshals the raw data to JSON and then unmarshals it into the target type, leveraging the type's custom UnmarshalJSON implementation.

This is used extensively in OtherComponentConfig (found in flytestdlib/config/tests/types_test.go) to handle fields like:

• DurationValue: config.Duration
• URLValue: config.URL
• NamedType: A custom enum that implements json.Unmarshaler

Configuration Structures in Practice

Flyte's test suite in flytestdlib/config/tests/accessor_test.go demonstrates how these complex types are structured in Go:

Nested Lists and Maps

The ConfigWithLists and ConfigWithMaps classes show how Flyte handles deep nesting:

type ComplexType struct {
	IntValue int `json:"int-val"`
}

type ConfigWithLists struct {
	ListOfStuff []ComplexType `json:"list"`
	StringValue string        `json:"string-val"`
}

type ConfigWithMaps struct {
	MapOfStuff     map[string]ComplexType `json:"m"`
	MapWithoutJSON map[string]ComplexType
}

Custom JSON Types

The ConfigWithJSONTypes class demonstrates the use of Flyte's custom duration type, which provides more flexibility than the standard time.Duration when parsing from various configuration formats:

type ConfigWithJSONTypes struct {
	Duration config.Duration `json:"duration"`
}

Strict Mode Validation

The viperAccessor supports a strictMode. When enabled, the parseViperConfigRecursive method will return an error if it encounters keys in the configuration file that do not correspond to any registered configuration section. This prevents silent failures caused by typos in configuration keys.

if v.strictMode {
    if newKeys := discoveredKeys.Difference(v.existingFlagKeys); newKeys.Len() > 0 {
        errs.Append(errors.Wrap(
            config.ErrStrictModeValidation,
            fmt.Sprintf("strict mode is on but received keys [%+v] to decode...", newKeys)))
    }
}

This strict validation is applied during the UpdateConfig and RefreshFromConfig calls, ensuring that the application's internal state always matches the intended configuration.