Security & Secret Management

Flyte manages sensitive credentials and secrets through a Kubernetes Admission Webhook that mutates task pods during creation. This system, centered around the SecretsPodMutator in flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go, intercepts pod requests and injects secrets based on annotations.

Core Architecture

The SecretsPodMutator unmarshals secret requests from pod annotations and iterates through a list of enabled SecretsInjector implementations. Each injector is responsible for a specific backend (e.g., K8s, AWS, GCP, Vault).

// From flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go
func (s *SecretsPodMutator) Mutate(ctx context.Context, pod *corev1.Pod) (newP *corev1.Pod, podChanged bool, errResponse *admission.Response) {
	secrets, err := secretUtils.UnmarshalStringMapToSecrets(pod.GetAnnotations())
	// ...
	for _, secret := range secrets {
		mutatedPod, injected, err := s.injectSecret(ctx, secret, pod)
        // ...
		pod = mutatedPod
	}
	return pod, len(secrets) > 0, nil
}

Kubernetes Secret Injection

The K8sSecretInjector (in flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go) is the default mechanism. It maps Flyte secret requests to native Kubernetes Secret objects.

Environment Variables

When a secret is requested as an environment variable (core.Secret_ENV_VAR), Flyte adds an EnvVar to the pod spec referencing the secret key.

// From flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go
case core.Secret_ENV_VAR:
    envVar := CreateEnvVarForSecret(secret, i.cfg.SecretEnvVarPrefix)
    p.Spec.InitContainers = AppendEnvVars(p.Spec.InitContainers, envVar)
    p.Spec.Containers = AppendEnvVars(p.Spec.Containers, envVar)

By default, environment variables are prefixed (e.g., _UNION_). The prefix is configurable via webhook.secretEnvVarPrefix.

File Mounts

When requested as a file (core.Secret_FILE), Flyte creates a volume and volume mount for the secret.

// From flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go
case core.Secret_FILE:
    volume := CreateVolumeForSecret(secret)
    p.Spec.Volumes = AppendVolume(p.Spec.Volumes, volume)

    mount := CreateVolumeMountForSecret(volume.Name, secret)
    p.Spec.InitContainers = AppendVolumeMounts(p.Spec.InitContainers, mount)
    p.Spec.Containers = AppendVolumeMounts(p.Spec.Containers, mount)

Files are typically mounted under /etc/flyte/secrets/<SecretGroup>/<SecretKey>.

Cloud Secret Managers (AWS, GCP, Azure)

For AWS, GCP, and Azure, Flyte uses an init-container sidecar approach. These injectors only support the FILE mount requirement.

AWS Secret Manager

The AWSSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/aws_secret_manager.go) adds an init-container that uses the AWS CLI (or a specialized sidecar image) to download the secret into a shared memory volume.

// From flyteplugins/go/tasks/pluginmachinery/secret/aws_secret_manager.go
func createAWSSidecarContainer(cfg config.AWSSecretManagerConfig, p *corev1.Pod, secret *core.Secret) corev1.Container {
	return corev1.Container{
		Image: cfg.SidecarImage,
		Name:  formatAWSInitContainerName(len(p.Spec.InitContainers)),
		VolumeMounts: []corev1.VolumeMount{
			{
				Name:      AWSSecretsVolumeName,
				MountPath: AWSInitContainerMountPath,
			},
		},
		Env: []corev1.EnvVar{
			{Name: AWSSecretArnEnvVar, Value: formatAWSSecretArn(secret)},
			{Name: AWSSecretFilenameEnvVar, Value: filepath.Join(string(filepath.Separator), strings.ToLower(secret.Group), strings.ToLower(secret.Key))},
		},
	}
}

GCP Secret Manager

Similarly, the GCPSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/gcp_secret_manager.go) uses a gcloud sidecar to fetch secrets.

Note: GCP secrets are stored as binary. If you attempt to mount a GCP secret as an environment variable using the Embedded injector, Flyte validates that the value is a valid UTF-8 string before injection.

Vault Integration

The VaultSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/vault_secret_manager.go) integrates with HashiCorp Vault by adding specific annotations to the pod. These annotations are then processed by a pre-existing Vault Agent Admission Webhook.

// From flyteplugins/go/tasks/pluginmachinery/secret/vault_secret_manager.go
commonVaultAnnotations := map[string]string{
    "vault.hashicorp.com/agent-inject":            "true",
    "vault.hashicorp.com/secret-volume-path":      filepath.Join(VaultSecretPathPrefix...),
    "vault.hashicorp.com/role":                    i.cfg.Role,
    "vault.hashicorp.com/agent-pre-populate-only": "true",
}

Restriction: Vault integration in Flyte does not support ENV_VAR mount requirements; it only supports FILE.

Embedded Secret Injection

The EmbeddedSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go) fetches secrets directly from cloud APIs (AWS, GCP, Azure) or the Kubernetes API during the webhook mutation phase.

• Environment Variables: Secrets are injected as literal strings directly into the pod spec's Env list.
• Files: Secrets are injected via a custom init-container that receives the secret values as base64-encoded environment variables and writes them to a shared memory volume.

Scoping and Labels

The Embedded injector requires specific labels on the pod to derive the secret scope:

• project
• domain
• organization

It searches for secrets in the following priority order:

1. project + domain scope
2. domain scope
3. organization scope

Global Secrets

Global secrets (in flyteplugins/go/tasks/pluginmachinery/secret/global_secrets.go) allow injecting credentials from the Flyte control plane's own environment or mounted files into task pods.

Restriction: Global secrets can only be injected as environment variables. If a secret request specifies FILE mount requirement, the injection will fail.

Configuration

Secret management is configured via the webhook section of the Flyte configuration.

Parameter	Default	Description
webhook.secretManagerType	K8s	Primary secret manager (Global, K8s, AWS, GCP, Vault, Embedded, Azure).
webhook.secretEnvVarPrefix	_UNION_	Prefix for secret environment variables.
webhook.awsSecretManager.sidecarImage	docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4	Image for AWS sidecar.
webhook.gcpSecretManager.sidecarImage	gcr.io/google.com/cloudsdktool/cloud-sdk:alpine	Image for GCP sidecar.
webhook.vaultSecretManager.role	-	Vault role to assume for secret retrieval.