Secret Management Architecture

Flyte tasks often require access to sensitive credentials, such as API keys or database passwords. Securely providing these secrets to the task's execution environment (a Kubernetes pod) without hardcoding them or exposing them unnecessarily is critical. Flyte addresses this by implementing a flexible secret management architecture that allows different mechanisms for injecting secrets.

The SecretsInjector Interface: Defining Injection Mechanisms

At the core of Flyte's secret management is the SecretsInjector interface. This interface defines a contract for any component responsible for injecting secrets into a Kubernetes pod. It abstracts the specific method of injection, allowing Flyte to support various secret storage backends and injection techniques (e.g., mounting secrets as files, injecting them as environment variables).

Each SecretsInjector implementation must provide two methods:

• Type() config.SecretManagerType: Identifies the specific type of secret manager (e.g., a global manager, a Kubernetes-native manager). This allows Flyte to select the appropriate injector based on configuration.
• Inject(ctx context.Context, secrets *core.Secret, p *corev1.Pod) (newP *corev1.Pod, injected bool, err error): This is the primary method where the actual injection logic resides. It takes the current pod specification (p) and the secret details (secrets). It returns a modified pod (newP) if injection occurred, a boolean indicating if any changes were made, and an error if the process failed.

Here is the definition of the SecretsInjector interface:

type SecretsInjector interface {
	Type() config.SecretManagerType
	Inject(ctx context.Context, secrets *core.Secret, p *corev1.Pod) (newP *corev1.Pod, injected bool, err error)
}

Different implementations of this interface handle the specifics of how secrets are retrieved and then integrated into the pod's configuration, ensuring that tasks have access to the necessary sensitive data.

Orchestrating Secret Injection with SecretsPodMutator

When a Flyte task is launched, it runs within a Kubernetes pod. Before this pod starts, Flyte must ensure that all required secrets are correctly provisioned into its environment. This orchestration is handled by the SecretsPodMutator.

The SecretsPodMutator acts as a central manager that holds and coordinates multiple SecretsInjector implementations. It contains:

• enabledSecretManagerTypes: A list of config.SecretManagerType values indicating which secret managers are active.
• injectors: A map where each key is a config.SecretManagerType and its value is the corresponding SecretsInjector implementation.

type SecretsPodMutator struct {
	// Secret manager types in order that they should be used.
	enabledSecretManagerTypes []config.SecretManagerType

	// It is expected that this map contains a key for every element in enabledSecretManagerTypes.
	injectors map[config.SecretManagerType]SecretsInjector
}

When Flyte initializes, it constructs a SecretsPodMutator instance using the NewSecretsMutator function. This function reads the configured SecretManagerTypes from Flyte's configuration and instantiates the appropriate SecretsInjector for each type.

func NewSecretsMutator(ctx context.Context, cfg *config.Config, podNamespace string, scope promutils.Scope) (*SecretsPodMutator, error) {
	enabledSecretManagerTypes := []config.SecretManagerType{
		config.SecretManagerTypeGlobal,
	}
	if len(cfg.SecretManagerTypes) != 0 {
		enabledSecretManagerTypes = append(enabledSecretManagerTypes, cfg.SecretManagerTypes...)
	} else {
		enabledSecretManagerTypes = append(enabledSecretManagerTypes, cfg.SecretManagerType) //nolint: staticcheck
	}

	injectors := make(map[config.SecretManagerType]SecretsInjector, len(enabledSecretManagerTypes))
	globalSecretManagerConfig := secretmanager.GetConfig()
	for _, secretManagerType := range enabledSecretManagerTypes {
		injector, err := newSecretsInjector(ctx, secretManagerType, cfg, globalSecretManagerConfig, podNamespace,
			scope.NewSubScope("secret_injector"))
		if err != nil {
			return nil, err
		}
		injectors[secretManagerType] = injector
	}

	return &SecretsPodMutator{
		enabledSecretManagerTypes: enabledSecretManagerTypes,
		injectors:                injectors,
	}
}

When a pod is about to be created, the Mutate method of SecretsPodMutator is invoked. This method iterates through the enabledSecretManagerTypes and calls the Inject method of each corresponding SecretsInjector. The SecretsPodMutator relies on annotations present on the pod to determine which secrets are required for injection.

For example, a test case demonstrates how a mock SecretsInjector is called by the SecretsPodMutator during the mutation process:

t.Run("added", func(t *testing.T) {
		mutator := &mocks.SecretsInjector{}
		mutator.EXPECT().Inject(mock.Anything, mock.Anything, mock.Anything).Return(&corev1.Pod{}, true, nil)
		mutator.EXPECT().Type().Return(config.SecretManagerTypeGlobal)
		ctx := context.Background()

		m := SecretsPodMutator{
			enabledSecretManagerTypes: []config.SecretManagerType{config.SecretManagerTypeGlobal},
			injectors: map[config.SecretManagerType]SecretsInjector{
				config.SecretManagerTypeGlobal: mutator,
			},
		}

		_, changed, err := m.Mutate(ctx, podWithAnnotations.DeepCopy())
		assert.Nil(t, err)
		assert.True(t, changed)
	})

SecretNameComponents: Standardizing Secret Naming

In a multi-tenant environment like Flyte, it is essential to uniquely identify and scope secrets to specific organizations, domains, and projects. This prevents naming collisions and ensures that secrets are only accessible where intended. Flyte achieves this standardization using the SecretNameComponents struct.

This struct breaks down a secret's fully qualified name into its constituent parts:

• Org: The organization the secret belongs to.
• Domain: The domain within the organization.
• Project: The specific project.
• Name: The bare name of the secret.

type SecretNameComponents struct {
	Org     string
	Domain  string
	Project string
	Name    string // Secret name
}

These components are crucial for deriving, encoding, and decoding secret names throughout the system, particularly for Kubernetes image pull secrets. For instance, when Flyte needs to create an image pull secret in a task's namespace, it uses SecretNameComponents to construct a unique and consistent Kubernetes secret name.

The DecodeSecretName function in flyteplugins/go/tasks/pluginmachinery/secret/utils.go illustrates how Flyte parses an encoded secret name string back into its SecretNameComponents. This function expects a specific, delimited format to correctly extract all the scoping information.

func DecodeSecretName(encodedSecretName string) (*SecretNameComponents, error) {
	parts := strings.Split(encodedSecretName, secretFieldSeparator)

	// We need at least 5 parts: u, org, <org>, domain, <secret-name>
	if len(parts) < 9 {
		return nil, errors.New(secretNameInvalidNotEnoughPartsMsg)
	}

	if parts[0] != secretsStorageUnionPrefix || parts[1] != secretsOrgDelimiter || parts[3] != secretsDomainDelimiter || parts[5] != secretsProjectDelimiter || parts[7] != secretsKeyDelimiter {
		return nil, errors.New(secretNameInvalidUnexpectedPartsMsg)
	}

	result := &SecretNameComponents{

Similarly, the ToImagePullK8sName function in flyteplugins/go/tasks/pluginmachinery/secret/imagepull_kubernetes_utils.go demonstrates how these components are used to generate a Kubernetes-compliant name for an image pull secret, ensuring consistency and proper scoping.

func ToImagePullK8sName(components SecretNameComponents) string {

Configuration and Considerations

Flyte's secret management behavior is configurable through various settings:

• SecretManagerType: Specifies the primary secret manager type to use (e.g., Global, K8s).
• SecretManagerTypes: Allows enabling multiple secret manager types simultaneously.
• SecretEnvVarPrefix: Defines the prefix for environment variables that will contain injected secrets (default: FLYTE_SECRETS_).
• ImagePullSecrets.Enabled: A boolean flag to enable or disable the mirroring of image pull secrets into task namespaces.

When working with Flyte's secret management, keep the following considerations in mind:

• Pod Annotations: The SecretsPodMutator relies on specific annotations on the pod to identify which secrets need to be injected. Incorrect or missing annotations will prevent secrets from being provisioned.
• Extensibility: If you introduce new secret manager types, the internal newSecretsInjector function must be updated to correctly instantiate the new SecretsInjector implementations.
• Init Containers: When secrets are injected as files, an init container is added to the pod. This adds an additional step to the pod's startup sequence, which might slightly increase pod launch times.
• SecretNameComponents Format: The encoding and decoding logic for SecretNameComponents is sensitive to the internal secretFieldSeparator and the expected order of components. Any deviation from this format will lead to incorrect secret name resolution.
• RBAC Permissions: The mechanism for mirroring image pull secrets creates Kubernetes secrets in the task execution namespace. This requires that the Flyte components have appropriate Role-Based Access Control (RBAC) permissions to perform these operations.