Secret Management and Injection

Flyte provides a robust secret management system that allows tasks to securely access credentials across various cloud providers and Kubernetes environments. This system supports two primary modes of operation: programmatic retrieval within plugins and automated injection into task execution environments (Pods) via a mutating webhook.

Programmatic Secret Retrieval in Plugins

When developing a Flyte plugin, you can retrieve secrets directly from the TaskExecutionContext. This is commonly used in WebAPI-based plugins that need to authenticate with external services.

The SecretManager interface in flyteplugins/go/tasks/pluginmachinery/core provides a simple Get method:

type SecretManager interface {
	Get(ctx context.Context, key string) (string, error)
}

In your plugin's Handle method, you can access the secret manager and retrieve a secret by its ID:

func (p WebApiPlugin) Handle(ctx context.Context, taskCtx core.TaskExecutionContext) (core.Transition, error) {
    secretID := "my_service_api_key"
    secretVal, err := taskCtx.SecretManager().Get(ctx, secretID)
    if err != nil {
        return core.UnknownTransition, fmt.Errorf("failed to get secret: %w", err)
    }
    
    // Use the secret value for authentication
    client := myapi.NewClient(secretVal)
    // ...
}

Pod Injection via Webhook

For containerized tasks, Flyte uses a SecretsPodMutator within a Kubernetes mutating webhook to inject secrets into the task's Pod. The mutator reads secret requirements from Pod annotations and uses one or more SecretsInjector implementations to modify the Pod spec.

Configuring Secret Managers

You can configure which secret managers are enabled in the webhook section of the Flyte configuration. The SecretManagerType enum in flyteplugins/go/tasks/pluginmachinery/secret/config defines the available strategies:

Type	Description
K8s	Mounts native Kubernetes secrets as volumes or environment variables.
AWS	Injects an AWS-provided sidecar init-container to pull secrets from AWS Secret Manager.
GCP	Injects a sidecar init-container to pull secrets from GCP Secret Manager.
Vault	Injects Hashicorp Vault sidecars or annotations.
Embedded	Fetches secrets during the mutation phase and injects them directly as environment variables or via a single shared init-container.

Example configuration in flyte-config.yaml:

webhook:
  secretManagerTypes:
    - K8s
    - AWS
  awsSecretManager:
    sidecarImage: "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4"

Using the Embedded Secret Manager

The EmbeddedSecretManager (type Embedded) is designed to avoid the overhead of multiple sidecar containers. It fetches secrets using a SecretFetcher and injects them into the Pod spec.

If a secret requires a file mount (core.Secret_FILE), the EmbeddedSecretManagerInjector in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go adds a single init-container (defaulting to busybox) that writes the secrets to a memory-backed volume:

func (i *EmbeddedSecretManagerInjector) injectAsFile(secret *core.Secret, secretValue []byte, pod *corev1.Pod) {
    // Adds or retrieves the shared init container
    initContainer, exists := i.getOrAppendFileMountInitContainer(pod)
    // Appends the secret to the SECRETS environment variable in base64 format
    appendSecretToFileMountInitContainer(initContainer, secret.GetKey(), secretValue)
    
    // ... mounts the volume to /etc/flyte/secrets
}

Sidecar Injection (AWS/GCP)

The AWSSecretManagerInjector and GCPSecretManagerInjector add an init-container for each secret requested. For AWS, this uses the createAWSSidecarContainer function:

func createAWSSidecarContainer(cfg config.AWSSecretManagerConfig, p *corev1.Pod, secret *core.Secret) corev1.Container {
	return corev1.Container{
		Image: cfg.SidecarImage,
		Name:  formatAWSInitContainerName(len(p.Spec.InitContainers)),
		Env: []corev1.EnvVar{
			{Name: "SECRET_ARN", Value: formatAWSSecretArn(secret)},
			{Name: "SECRET_FILENAME", Value: filepath.Join("/", strings.ToLower(secret.Group), strings.ToLower(secret.Key))},
		},
        // ...
	}
}

Troubleshooting and Gotchas

Sidecar Overhead

If a task requests many secrets, the AWS and GCP sidecar injectors will add a corresponding number of init-containers. This can increase Pod startup time and resource consumption. Consider using the Embedded manager if you have many secrets.

Binary Secrets in Environment Variables

The EmbeddedSecretManager and GlobalSecretManager require that any secret injected as an environment variable must be a valid UTF-8 string. If a secret contains binary data (common in GCP Secret Manager), it must be mounted as a file.

if !utf8.Valid(secretValue.BinaryValue) {
    return pod, false, fmt.Errorf(
        "secret %q is attempted to be mounted as an environment variable, "+
        "but has a binary value that is not a valid UTF-8 string; mount "+
        "as a file instead", secret.Key)
}

Kubernetes Secret Keys

When using the K8sSecretInjector, the Key field in the secret request is mandatory. If the Group field is missing, Flyte defaults to using an MD5 hash of the key to identify the Kubernetes secret.

Environment Variable Prefixes

By default, Flyte prefixes secret environment variables with _UNION_. This can be customized via the secretEnvVarPrefix configuration setting in the webhook section. The SecretEnvVarPrefix environment variable is also injected into the container to allow applications to discover the prefix at runtime.