Skip to main content

Security & Secret Management

Flyte provides a unified interface for managing sensitive data across multiple backends, including AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Hashicorp Vault, and Kubernetes Secrets. This system is implemented as a Kubernetes admission webhook that intercepts Pod creation requests and injects secrets based on Pod annotations.

Configuring Secret Managers​

To enable secret management in Flyte, you must configure the webhook section in your Flyte configuration. The SecretsPodMutator (found in flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go) iterates through a list of enabled secret managers to fulfill secret requests.

webhook:
  secretManagerTypes:
    - K8s
    - AWS
    - Vault
  secretEnvVarPrefix: "_FLYTE_"

The secretManagerTypes list defines the order of preference. Flyte will attempt to resolve a secret using each manager in the order specified until one succeeds.

Injecting Kubernetes Secrets​

The K8sSecretInjector (in flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go) allows you to mount Kubernetes Secrets as either environment variables or files.

Environment Variable Injection​

When a secret is requested as an environment variable, Flyte adds the secret to the env list of both the init-containers and the main containers.

// From flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go
case core.Secret_ENV_VAR:
    envVar := CreateEnvVarForSecret(secret, i.cfg.SecretEnvVarPrefix)
    p.Spec.InitContainers = AppendEnvVars(p.Spec.InitContainers, envVar)
    p.Spec.Containers = AppendEnvVars(p.Spec.Containers, envVar)

File Mount Injection​

For file mounts, Flyte creates a volume from the Kubernetes Secret and mounts it to the Pod.

// From flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go
case core.Secret_FILE:
    volume := CreateVolumeForSecret(secret)
    p.Spec.Volumes = AppendVolume(p.Spec.Volumes, volume)
    mount := CreateVolumeMountForSecret(volume.Name, secret)
    p.Spec.InitContainers = AppendVolumeMounts(p.Spec.InitContainers, mount)
    p.Spec.Containers = AppendVolumeMounts(p.Spec.Containers, mount)

Using the Vault Secret Manager​

The VaultSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/vault_secret_manager.go) integrates with the Hashicorp Vault Agent Injector. Instead of modifying the Pod spec directly with secret values, it adds specific annotations that trigger the Vault sidecar.

// From flyteplugins/go/tasks/pluginmachinery/secret/vault_secret_manager.go
commonVaultAnnotations := map[string]string{
    "vault.hashicorp.com/agent-inject":            "true",
    "vault.hashicorp.com/secret-volume-path":      filepath.Join(VaultSecretPathPrefix...),
    "vault.hashicorp.com/role":                    i.cfg.Role,
    "vault.hashicorp.com/agent-pre-populate-only": "true",
}
secretVaultAnnotations := CreateVaultAnnotationsForSecret(secret, i.cfg.KVVersion)
p.Annotations = utils.UnionMaps(secretVaultAnnotations, commonVaultAnnotations, i.cfg.Annotations, p.Annotations)

Note: The Vault injector only supports FILE or ANY mount requirements; it does not support ENV_VAR.

Using the Embedded Secret Manager​

The EmbeddedSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go) is a high-performance alternative to cloud-specific sidecars. It fetches secrets at mutation time and injects them directly into the Pod spec.

Hierarchical Scoping​

The Embedded manager resolves secrets using a hierarchical lookup based on Pod labels. It searches in the following order:

  1. Project + Domain: {{org}}/{{domain}}/{{project}}/{{key}}
  2. Domain: {{org}}/{{domain}}//{{key}}
  3. Organization: {{org}}///{{key}}

This requires the Pod to have project, domain, and organization labels.

Efficient File Mounting​

Unlike cloud sidecars that add an init-container for every secret, the Embedded manager uses a single init-container (init-embedded-secret) to write all requested file-based secrets to a shared memory volume (/etc/flyte/secrets).

// From flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go
func (i *EmbeddedSecretManagerInjector) injectAsFile(secret *core.Secret, secretValue []byte, pod *corev1.Pod) {
    initContainer, exists := i.getOrAppendFileMountInitContainer(pod)
    appendSecretToFileMountInitContainer(initContainer, secret.GetKey(), secretValue)
    // ... setup volume and volumeMounts ...
}

Cloud Sidecar Injectors (AWS/GCP/Azure)​

For AWS, GCP, and Azure, Flyte can inject cloud-provider sidecars. These sidecars run as init-containers to download secrets into a shared volume.

  • AWS: Uses docker.io/amazon/aws-secrets-manager-secret-sidecar.
  • GCP: Uses gcr.io/google.com/cloudsdktool/cloud-sdk:alpine.
  • Azure: Uses mcr.microsoft.com/azure-cli:cbl-mariner2.0.

These are configured via AWSSecretManagerConfig, GCPSecretManagerConfig, and AzureSecretManagerConfig in flyteplugins/go/tasks/pluginmachinery/secret/config/config.go.

Troubleshooting and Gotchas​

GCP Binary Secrets​

GCP Secret Manager stores values as binary. If you attempt to mount a GCP secret as an environment variable, Flyte validates that the binary is a valid UTF-8 string.

// From flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go
if !utf8.Valid(secretValue.BinaryValue) {
    return pod, false, fmt.Errorf(
        "secret %q is attempted to be mounted as an environment variable, "+
            "but has a binary value that is not a valid UTF-8 string; mount "+
            "as a file instead", secret.Key)
}

Environment Variable Prefixes​

Flyte uses several environment variables to communicate secret locations to the task code:

  • FLYTE_SECRETS_DEFAULT_DIR: The directory where file-mounted secrets are stored (default: /etc/flyte/secrets).
  • FLYTE_SECRETS_ENV_PREFIX: The prefix used for secret environment variables (default: _UNION_).

Missing Labels​

The EmbeddedSecretManager will fail to inject secrets if the Pod is missing the required organization, project, or domain labels, as it cannot construct the lookup path.