Skip to main content

Secret Management Integration

Flyte provides a flexible secret management system that allows tasks to securely access credentials from various providers, including AWS Secret Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault, and Kubernetes Secrets.

Secret management in Flyte is handled through two primary mechanisms:

  1. Pod Injection: A mutating webhook uses SecretsInjector implementations to modify task Pods (e.g., adding sidecars or environment variables) before they start.
  2. Programmatic Access: Plugins use the SecretManager interface to retrieve secrets during task setup or execution.

Accessing Secrets in Plugins​

Flyte plugins can retrieve secrets programmatically using the SecretManager provided in the SetupContext or TaskExecutionContext.

func (p myPlugin) Handle(ctx context.Context, tCtx core.TaskExecutionContext) (core.Transition, error) {
    // Get the secret manager from the execution context
    sm := tCtx.SecretManager()

    // Retrieve a secret by its key
    apiKey, err := sm.Get(ctx, "my_api_key")
    if err != nil {
        return core.UnknownTransition, err
    }

    // Use the secret...
    return core.DoSomethingWithKey(apiKey), nil
}

The SecretManager interface is defined in flyteplugins/go/tasks/pluginmachinery/core/secret_manager.go:

type SecretManager interface {
    Get(ctx context.Context, key string) (string, error)
}

Configuring Secret Injectors​

Flyte uses SecretsInjector implementations to satisfy secret requirements defined in a task's TaskTemplate. You configure the active injector via the webhook.secretManagerType configuration.

AWS Secret Manager​

The AWSSecretManagerInjector (found in flyteplugins/go/tasks/pluginmachinery/secret/aws_secret_manager.go) injects an AWS-provided sidecar as an init-container. This container downloads the secret and saves it to a shared volume.

Configuration:

  • webhook.secretManagerType: aws
  • webhook.awsSecretManager.sidecarImage: The image used for the AWS sidecar (default: docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4).

GCP Secret Manager​

The GCP injector handles secrets stored in Google Cloud. Note that GCP secrets are stored as binary data. If you attempt to mount a GCP secret as an environment variable, Flyte validates that the value is a valid UTF-8 string.

Embedded Secret Manager​

The EmbeddedSecretManager (found in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go) provides direct API access to secrets without requiring a sidecar. It supports a hierarchical lookup strategy to resolve secrets based on the task's scope.

Hierarchical Lookup​

The EmbeddedSecretManager searches for secrets in the following order:

  1. Project + Domain: {{organization}}/{{domain}}/{{project}}/{{key}}
  2. Domain: {{organization}}/{{domain}}//{{key}}
  3. Organization: {{organization}}///{{key}}

For this to work, the task Pod must have the following labels:

  • project
  • domain
  • organization

Injection Modes​

The EmbeddedSecretManagerInjector supports two mount requirements:

  • ENV_VAR: Injects the secret value directly into the container's environment variables.
  • FILE: Uses a specialized init container to write secrets into a memory-backed volume (/etc/flyte/secrets).
// Example of how the Embedded injector handles file mounts
// from flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go
func (i *EmbeddedSecretManagerInjector) injectAsFile(secret *core.Secret, secretValue []byte, pod *corev1.Pod) {
    initContainer, exists := i.getOrAppendFileMountInitContainer(pod)
    appendSecretToFileMountInitContainer(initContainer, secret.GetKey(), secretValue)
    
    // ... configures volumes and mount paths
}

Troubleshooting and Gotchas​

Binary Secrets in GCP​

GCP secrets are binary-only. If a secret contains non-UTF-8 data and you attempt to mount it as an ENV_VAR, the injection will fail. Use FILE mount requirements for binary secrets.

AWS Init-Container Overhead​

The AWSSecretManagerInjector adds an init-container for each secret requested. If a task requires many secrets, this can significantly increase the number of init-containers in the Pod spec.

Missing Labels for Embedded Mode​

The EmbeddedSecretManager will return a SecretRequirementsError if the Pod labels for project, domain, or organization are missing, as it cannot derive the lookup path without them.

Environment Variable Prefixes​

By default, Flyte prefixes secret environment variables with _UNION_. You can customize this via the webhook.secretEnvVarPrefix configuration. If a secret explicitly defines an EnvVar name in the TaskTemplate, that name is used instead of the prefixed key.