Skip to main content

The Embedded Secret Manager

The Embedded Secret Manager in Flyte provides a high-performance, sidecar-less mechanism for injecting secrets into task pods. Unlike traditional secret management approaches that rely on long-lived sidecar containers to sync secrets, the Embedded Secret Manager resolves and injects secrets during the Kubernetes Pod admission phase via a webhook.

Architecture and Workflow​

The core of this system is the EmbeddedSecretManagerInjector, located in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go. When a Flyte task is scheduled, the Flyte Pod Webhook intercepts the pod creation request. The SecretsPodMutator then orchestrates the injection process by calling the Inject method of the enabled secret manager.

The workflow follows these steps:

  1. Request Parsing: The mutator unmarshals secret requests from pod annotations.
  2. Identity Resolution: The injector extracts the project, domain, and organization from the pod's labels.
  3. Hierarchical Lookup: The injector searches for the secret across multiple scopes using the SecretFetcher interface.
  4. Injection: Depending on the MountRequirement (Environment Variable or File), the injector modifies the Pod spec to include the secret data.

Hierarchical Secret Resolution​

Flyte implements a hierarchical lookup strategy to allow for flexible secret management across different organizational levels. The lookUpSecret method in EmbeddedSecretManagerInjector iterates through scopes in the following priority:

  1. Project + Domain: The most specific scope (e.g., my-project/development/my-secret).
  2. Domain: A broader scope for secrets shared across a domain (e.g., development/my-secret).
  3. Organization: The global scope for the entire Flyte installation (e.g., my-org/my-secret).

For each scope, the manager first checks an internal cache (goCache.CacheInterface[SecretValue]) before falling back to the configured cloud provider fetcher.

// From flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go
func (i *EmbeddedSecretManagerInjector) lookUpSecret(ctx context.Context, components *SecretNameComponents) (*SecretValue, string, error) {
    // ... setup scopes ...
    for _, scope := range scopes {
        if cachedValue, err := i.secretCache.Get(ctx, scope.secretID); err == nil {
            return &cachedValue, scope.imagePullSecretName, nil
        }

        for _, secretFetcher := range i.secretFetchers {
            secretValue, err := secretFetcher.GetSecretValue(ctx, scope.secretID)
            if err == nil {
                i.secretCache.Set(ctx, scope.secretID, *secretValue)
                return secretValue, scope.imagePullSecretName, nil
            }
            // ... handle NotFound ...
        }
    }
    return nil, "", stdlibErrors.Errorf(ErrCodeSecretNotFoundAcrossAllScopes, ...)
}

Supported Backends​

The SecretFetcher interface abstracts the underlying secret storage. Flyte provides implementations for major cloud providers and Kubernetes:

  • AWS: AWSSecretFetcher interacts with AWS Secrets Manager.
  • GCP: GCPSecretFetcher interacts with Google Cloud Secret Manager.
  • Azure: AzureSecretFetcher interacts with Azure Key Vault.
  • Kubernetes: K8sSecretFetcher retrieves secrets from a designated namespace in the cluster.

The fetchers are instantiated in NewSecretFetcher (found in flyteplugins/go/tasks/pluginmachinery/secret/secret_fetcher.go) based on the SecretManagerType configured in the EmbeddedSecretManagerConfig.

Injection Mechanisms​

The Embedded Secret Manager supports two primary injection modes defined by the core.Secret_MountRequirement.

Environment Variables​

For ENV_VAR requirements, the injector directly adds environment variables to all containers in the pod. By default, these variables are prefixed with _UNION_ (configurable via SecretEnvVarPrefix).

// From flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go
func (i *EmbeddedSecretManagerInjector) injectAsEnvVar(secret *core.Secret, secretValue string, pod *corev1.Pod) {
    valueEnvVarName := i.parentCfg.SecretEnvVarPrefix + strings.ToUpper(secret.GetKey())
    // ...
    envVars := []corev1.EnvVar{
        {Name: valueEnvVarName, Value: secretValue},
    }
    pod.Spec.Containers = AppendEnvVars(pod.Spec.Containers, envVars...)
}

File Mounts​

For FILE requirements, Flyte avoids the overhead of a persistent sidecar by using a lightweight init container and an in-memory EmptyDir volume.

  1. The injector adds an init-embedded-secret container (typically using a busybox image).
  2. The secret data is base64-encoded and passed to the init container via the SECRETS environment variable.
  3. The init container runs a shell script that decodes these values and writes them to the shared volume at /etc/flyte/secrets/.
  4. The volume is mounted as read-only into the main task containers.

Security and Permissions​

A key design choice of the Embedded Secret Manager is that the Flyte Webhook pod (or the component running the injector) requires the IAM permissions to read from the cloud secret managers, rather than the individual task pods.

This approach offers several advantages:

  • Reduced IAM Complexity: You do not need to manage IAM roles for every possible task execution.
  • Security Isolation: Task pods never receive the credentials required to browse the secret manager; they only receive the specific secrets they requested.
  • Performance: Secret resolution happens once at pod creation, eliminating the need for tasks to perform network calls to secret providers during execution.

Configuration​

The manager is configured via the embeddedSecretManagerConfig block in the Flyte configuration. Key fields include:

  • type: The provider type (aws, gcp, azure, or k8s).
  • fileMountInitContainer: Configures the image (default public.ecr.aws/docker/library/busybox:latest) and resources for the file-injection init container.
  • secretCacheSize: Controls the size of the in-memory cache used to reduce provider API latency.

Note that for the injector to function, the task pod must have the project, domain, and organization labels correctly set, as these are used to construct the secret lookup keys. Missing labels will result in a SecretRequirementsError.