Pod Mutation and Runtime Security

When you run a Flyte task that requires access to sensitive information—such as database credentials or API keys—you need a way to securely inject those secrets into the task's execution environment. If a task attempts to access a secret that hasn't been injected, it will fail with a "secret not found" error or a permission denied exception.

Flyte solves this using a Mutating Admission Webhook that intercepts Pod creation requests and modifies the Pod specification to include the necessary secret mounts or environment variables.

The Pod Mutation Lifecycle

The mutation process is handled by the PodMutator class in executor/pkg/webhook/pod.go. This class implements the controller-runtime WebHook interface. When a new Pod is scheduled, the Kubernetes API server sends an admission request to the Flyte executor.

The PodMutator.Handle method processes this request:

func (pm PodMutator) Handle(ctx context.Context, request admission.Request) admission.Response {
	obj := &corev1.Pod{}
	err := pm.decoder.Decode(request, obj)
	if err != nil {
		return admission.Errored(http.StatusBadRequest, err)
	}

	newObj, changed, admissionErr := pm.secretsMutator.Mutate(ctx, obj)
	if admissionErr != nil {
		return *admissionErr
	}

	if changed {
		marshalled, err := json.Marshal(newObj)
		if err != nil {
			return admission.Errored(http.StatusInternalServerError, err)
		}
		return admission.PatchResponseFromRaw(request.Object.Raw, marshalled)
	}

	return admission.Allowed("No changes")
}

Internally, the PodMutator delegates the actual modification to a SecretsPodMutator. This mutator identifies which secrets to inject by looking at the Pod's annotations.

Triggering Injection

The webhook does not mutate every Pod in the cluster. It only acts on Pods that meet two criteria:

1. Label: The Pod must have the label flyte-secrets-inject: true.
2. Annotations: The Pod must have annotations following the pattern flyte.secrets/s0, flyte.secrets/s1, etc.

These annotations contain marshaled core.Secret objects. The SecretsPodMutator.Mutate method in flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go unmarshals these annotations to determine which secret managers (K8s, AWS, GCP, etc.) should be used to fulfill the request.

Secret Injection Mechanisms

Flyte supports multiple secret injection strategies depending on the MountRequirement specified in the secret request. The K8sSecretInjector in flyteplugins/go/tasks/pluginmachinery/secret/k8s_secrets.go demonstrates the two primary methods:

1. File-based Injection

If a secret is requested as a file (core.Secret_FILE), Flyte creates a Kubernetes Volume and VolumeMount. This allows the secret to appear as a file within the task container.

case core.Secret_FILE:
    volume := CreateVolumeForSecret(secret)
    p.Spec.Volumes = AppendVolume(p.Spec.Volumes, volume)
    mount := CreateVolumeMountForSecret(volume.Name, secret)
    p.Spec.Containers = AppendVolumeMounts(p.Spec.Containers, mount)

2. Environment Variable Injection

If a secret is requested as an environment variable (core.Secret_ENV_VAR), Flyte adds an EnvVar to the container spec using ValueFrom to reference the Kubernetes secret.

case core.Secret_ENV_VAR:
    envVar := CreateEnvVarForSecret(secret, i.cfg.SecretEnvVarPrefix)
    p.Spec.Containers = AppendEnvVars(p.Spec.Containers, envVar)

By default, environment variables injected this way are prefixed with _UNION_ (configurable via webhook.secretEnvVarPrefix).

Webhook Security and TLS

Because the Kubernetes API server communicates with the webhook over HTTPS, the webhook must serve a valid TLS certificate. Flyte automates this setup during the executor's initialization phase.

The InitCerts function in executor/pkg/webhook/init_cert.go generates a self-signed CA and server certificate. These are stored in a webhookCerts struct:

type webhookCerts struct {
	CaPEM         *bytes.Buffer
	ServerPEM     *bytes.Buffer
	PrivateKeyPEM *bytes.Buffer
}

Flyte then creates a Kubernetes Secret (default name: flyte-pod-webhook) to store these certificates. When the executor starts, it mounts this secret to provide the TLS identity for the PodMutator server.

Configuration Options

You can customize the behavior of the mutation webhook through the webhook configuration block. Key settings include:

Parameter	Default	Description
secretName	flyte-pod-webhook	The K8s secret where TLS certificates are stored.
serviceName	flyte-pod-webhook	The K8s service name pointing to the webhook.
certDir	/etc/webhook/certs	The directory where the webhook looks for TLS files.
secretManagerType	k8s	The primary provider for secrets (e.g., k8s, aws, gcp, vault).
localCert	false	If true, certs are written to the local filesystem (used for local development).

At runtime, Flyte also sets environment variables like FLYTE_SECRETS_DEFAULT_DIR and FLYTE_SECRETS_ENV_PREFIX inside the task Pod to help the Flyte SDK locate the injected secrets.