Skip to main content

Secret Injection Mechanics

Flyte manages secret injection into task execution pods through a Kubernetes Mutating Admission Webhook. This mechanism allows Flyte to intercept pod creation requests and modify the pod specification to include secret references, environment variables, or sidecar containers before the pod is scheduled.

The Mutation Orchestrator​

The core logic for pod modification resides in the SecretsPodMutator class within flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go. This component acts as an orchestrator that processes secret requests embedded in pod annotations.

When a pod is submitted to the Kubernetes API server, the PodMutator webhook handler (defined in executor/pkg/webhook/pod.go) decodes the request and delegates the mutation to SecretsPodMutator.Mutate.

The mutation process follows these steps:

  1. Annotation Parsing: It extracts secret requests from pod annotations using secretUtils.UnmarshalStringMapToSecrets. These annotations use the prefix flyte.secrets/s and contain Base32-encoded protobuf representations of the secrets.
  2. Injector Iteration: For each requested secret, the mutator iterates through a list of enabledSecretManagerTypes.
  3. Secret Injection: It calls the Inject method on the corresponding SecretsInjector implementation. The first injector that successfully handles the secret stops the iteration for that specific secret.
func (s *SecretsPodMutator) Mutate(ctx context.Context, pod *corev1.Pod) (newP *corev1.Pod, podChanged bool, errResponse *admission.Response) {
	secrets, err := secretUtils.UnmarshalStringMapToSecrets(pod.GetAnnotations())
	if err != nil {
		admissionError := admission.Errored(http.StatusBadRequest, fmt.Errorf("failed to unmarshal secrets from pod annotations: %w", err))
		return pod, false, &admissionError
	}

	for _, secret := range secrets {
		mutatedPod, injected, err := s.injectSecret(ctx, secret, pod)
		if !injected {
            // ... error handling ...
			return pod, false, &admissionError
		}
		pod = mutatedPod
	}
	return pod, len(secrets) > 0, nil
}

The SecretsInjector Interface​

The SecretsInjector interface, defined in flyteplugins/go/tasks/pluginmachinery/secret/secrets_injector.go, provides a standard contract for different secret management backends (Kubernetes, AWS, GCP, Vault, etc.) to modify the pod specification.

type SecretsInjector interface {
	Type() config.SecretManagerType
	Inject(ctx context.Context, secrets *core.Secret, p *corev1.Pod) (newP *corev1.Pod, injected bool, err error)
}

Native Kubernetes Secret Injection​

The K8sSecretInjector (in flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go) implements injection using native Kubernetes Secret objects. It supports two primary mount requirements:

File-based Injection​

When a secret is requested as a file (core.Secret_FILE), the injector:

  • Creates a Volume using SecretVolumeSource pointing to the Kubernetes secret.
  • Appends a VolumeMount to all containers and init-containers in the pod.
  • Sets the FLYTE_SECRETS_DEFAULT_DIR environment variable to /etc/flyte/secrets.
  • Mounts the secret at /etc/flyte/secrets/<SecretGroup>/<SecretKey>.

Environment Variable Injection​

When a secret is requested as an environment variable (core.Secret_ENV_VAR), the injector:

  • Creates an EnvVar using SecretKeyRef.
  • Names the variable using a prefix (defaulting to _UNION_) followed by the group and key (e.g., _UNION_MYGROUP_MYKEY).
  • Sets the FLYTE_SECRETS_ENV_PREFIX environment variable to the configured prefix.

Cloud Provider Injection via Sidecars​

For cloud-native secret managers like AWS Secrets Manager, Flyte uses an init-container sidecar pattern. The AWSSecretManagerInjector (in flyteplugins/go/tasks/pluginmachinery/secret/aws_secret_manager.go) demonstrates this approach:

  1. Shared Volume: It creates an EmptyDir volume with a memory medium (AWSSecretsVolumeName) shared across all containers.
  2. Init-Container: For each secret requested, it adds a new init-container (e.g., aws-pull-secret-0) using a specialized sidecar image.
  3. Secret Retrieval: The sidecar container is configured via environment variables (SECRET_ARN and SECRET_FILENAME) to pull the secret from AWS and write it to the shared volume.
  4. Mounting: The shared volume is mounted into the main containers at /etc/flyte/secrets.

This pattern ensures that secrets never touch the disk and are only available in memory during the pod's lifecycle.

Configuration and Precedence​

The order in which secret managers are attempted is determined by the SecretManagerTypes configuration. By default, Flyte includes a "Global" manager followed by any specifically configured types.

In flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go, the NewSecretsMutator function initializes the injectors:

func NewSecretsMutator(ctx context.Context, cfg *config.Config, podNamespace string, scope promutils.Scope) (*SecretsPodMutator, error) {
	enabledSecretManagerTypes := []config.SecretManagerType{
		config.SecretManagerTypeGlobal,
	}
	if len(cfg.SecretManagerTypes) != 0 {
		enabledSecretManagerTypes = append(enabledSecretManagerTypes, cfg.SecretManagerTypes...)
	} else {
		enabledSecretManagerTypes = append(enabledSecretManagerTypes, cfg.SecretManagerType)
	}
    // ... injector initialization ...
}

If a secret cannot be satisfied by any of the enabled injectors, the admission request is rejected with a 400 Bad Request, preventing the pod from starting without its required secrets.