Skip to main content

Embedded Secret Management

Flyte implements an Embedded Secret Manager to provide a lightweight, sidecar-less mechanism for injecting secrets into task pods. Unlike traditional sidecar approaches that run a separate container for the duration of a task, the embedded manager fetches secrets during the Kubernetes admission phase via the Flyte Pod Webhook and injects them directly into the pod specification.

Design Philosophy​

The "Embedded" design is built to minimize resource overhead and simplify secret lifecycle management. By performing secret retrieval at pod creation time, Flyte avoids the need for long-running sidecar containers that would otherwise consume CPU and memory throughout the task's execution.

The implementation relies on the EmbeddedSecretManagerInjector class in flyteplugins/go/tasks/pluginmachinery/secret/embedded_secret_manager.go, which coordinates with various backend-specific fetchers to resolve and inject secret values.

Secret Retrieval and Fetchers​

The core of the retrieval logic is the SecretFetcher interface, which abstracts the underlying secret store (e.g., AWS Secrets Manager, GCP Secret Manager).

type SecretFetcher interface {
	GetSecretValue(ctx context.Context, secretID string) (*SecretValue, error)
}

Flyte provides several implementations of this interface:

  • AWSSecretFetcher: Communicates with AWS Secrets Manager.
  • GCPSecretFetcher: Communicates with Google Cloud Secret Manager.
  • AzureSecretFetcher: Communicates with Azure Key Vault.
  • K8sSecretFetcher: Retrieves secrets from Kubernetes Secret objects.

When a secret is requested, the SecretValue struct holds the retrieved data as either a string or raw bytes:

type SecretValue struct {
	StringValue string
	BinaryValue []byte
}

Hierarchical Secret Resolution​

The EmbeddedSecretManagerInjector uses a hierarchical lookup strategy to find the most specific secret available. It relies on the project, domain, and organization labels present on the Flyte Pod to construct potential secret IDs.

The lookUpSecret method in embedded_secret_manager.go searches in the following order of priority:

  1. Project + Domain: org/domain/project/secret_key
  2. Domain: org/domain//secret_key
  3. Organization: org///secret_key

This allows administrators to define default secrets at the organization or domain level while allowing specific projects to override them.

Injection Strategies​

Depending on the MountRequirement specified in the secret request, Flyte employs different injection techniques.

Environment Variables​

For core.Secret_ENV_VAR, the injector directly modifies the Env field of both the task's Containers and InitContainers. By default, environment variables are prefixed (e.g., _UNION_), but this can be overridden by the secret's configuration.

func (i *EmbeddedSecretManagerInjector) injectAsEnvVar(secret *core.Secret, secretValue string, pod *corev1.Pod) {
	valueEnvVarName := i.parentCfg.SecretEnvVarPrefix + strings.ToUpper(secret.GetKey())
	if secret.GetEnvVar() != "" {
		valueEnvVarName = secret.EnvVar
	}
    // ... adds to pod.Spec.Containers and pod.Spec.InitContainers
}

File Mounts via Init Container​

For core.Secret_FILE, Flyte uses a clever "busybox" init container approach to avoid long-running sidecars. The process works as follows:

  1. An emptyDir volume (memory-backed) is added to the Pod.
  2. A lightweight init container (configured via FileMountInitContainerConfig) is added.
  3. The secret values are base64-encoded and passed to this init container via a single environment variable named SECRETS.
  4. The init container runs a shell script that decodes these values and writes them to the shared volume.

The shell script used by the init container is defined in getOrAppendFileMountInitContainer:

fmt.Sprintf(`
    printf "%%s" "$%s" \
    | awk '/^.+=/ {
        i = index($0, "=");
        name = substr($0, 0, i - 1);
        value = substr($0, i + 1);
        output_file = "%s/" name;
        print value | "base64 -d > " output_file;
    }'
    `,
    EmbeddedSecretsFileMountInitContainerEnvVariableName,
    EmbeddedSecretsFileMountPath)

This ensures that by the time the main task container starts, the secrets are already present as files in /etc/flyte/secrets/.

Configuration​

The behavior of the embedded manager is controlled via the EmbeddedSecretManagerConfig struct in flyteplugins/go/tasks/pluginmachinery/secret/config/config.go. Key configuration points include:

  • Type: Determines which backend fetcher to initialize (AWS, GCP, etc.).
  • FileMountInitContainer: Configures the image (defaulting to busybox) and resource requirements for the secret-mounting init container.
  • ImagePullSecrets: If enabled, the manager can also mirror secrets into the pod's namespace to be used as ImagePullSecrets.
type FileMountInitContainerConfig struct {
	Image         string                      `json:"image" pflag:",Specifies init container image to use for mounting secrets as files."`
	Resources     corev1.ResourceRequirements `json:"resources" pflag:"-,Specifies resource requirements for the init container."`
	ContainerName string                      `json:"containerName" pflag:",Specifies the name of the init container that mounts secrets as files."`
}

Constraints and Considerations​

  • Label Dependency: The injector requires the Pod to have organization, project, and domain labels. If these are missing, secret resolution will fail.
  • Binary Secrets in Env Vars: If a binary secret (common in GCP) is requested as an environment variable, Flyte validates that it is a valid UTF-8 string. If it is not, the injection fails, and the user is prompted to mount it as a file instead.
  • Memory Usage: Since file-mounted secrets use a memory-backed emptyDir, very large secrets will consume the Pod's memory quota.