Cloud-Native Secret Providers

When you run Flyte tasks that require access to sensitive information like API keys or database credentials, you often want to leverage cloud-native services like AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Flyte provides a sidecar-based injection mechanism that automatically fetches these secrets and mounts them as files into your task's pod, ensuring that your application code can access them securely without needing to interact directly with cloud APIs.

Sidecar-Based Injection Architecture

Flyte implements cloud-native secret retrieval using a Kubernetes admission webhook. When a pod is created for a Flyte task, the SecretsPodMutator (found in flyteplugins/go/tasks/pluginmachinery/secret/secrets_pod_mutator.go) inspects the pod's annotations for secret requests. If a cloud-native provider is configured, the mutator uses a specific SecretsInjector to modify the pod specification.

The injection process typically involves:

1. Adding a shared EmptyDir volume to the pod.
2. Injecting one or more init-containers (sidecars) that use cloud-specific CLIs or SDKs to download the secrets.
3. Mounting the shared volume into both the init-containers and the main task container.
4. Setting environment variables like FLYTE_SECRETS_DEFAULT_DIR to inform the task where the secrets are located.

AWS Secrets Manager

The AWSSecretManagerInjector allows you to pull secrets from AWS. It relies on the amazon/aws-secrets-manager-secret-sidecar image to perform the actual retrieval.

Usage

To request an AWS secret, you define a Secret object where the Group represents the ARN prefix and the Key is the secret name.

// Example of how Flyte internally represents the secret request
secret := &core.Secret{
    Group: "arn:aws:secretsmanager:us-west-2:123456789012:secret",
    Key:   "my-api-key",
    MountRequirement: core.Secret_FILE,
}

Internal Mechanism

The AWSSecretManagerInjector (in aws_secret_manager.go) formats the ARN by joining the Group and Key:

func formatAWSSecretArn(secret *core.Secret) string {
	return strings.TrimRight(secret.Group, ":") + ":" + strings.TrimLeft(secret.Key, ":")
}

It then creates an init-container that uses the SECRET_ARN environment variable to identify the secret and SECRET_FILENAME to specify the destination in the shared volume (defaulting to aws-secret-vol). The secret is mounted at /etc/flyte/secrets/<secret_group>/<secret_key>.

GCP Secret Manager

The GCPSecretManagerInjector uses the Google Cloud SDK (gcloud) to access secrets. This injector requires both a Group (the secret name) and a GroupVersion.

Usage

secret := &core.Secret{
    Group:        "my-gcp-secret",
    GroupVersion: "1",
    MountRequirement: core.Secret_FILE,
}

Internal Mechanism

In gcp_secret_manager.go, Flyte constructs a shell command for the gcloud sidecar. Because gcloud writes files with restrictive permissions (0600), the injector also appends a chmod command to ensure the main task container (which may run as a non-root user) can read the file.

func formatGCPSecretAccessCommand(secret *core.Secret) []string {
	secretDir := strings.ToLower(filepath.Join(GCPSecretMountPath, secret.Group))
	secretPath := strings.ToLower(filepath.Join(secretDir, secret.GroupVersion))
	args := fmt.Sprintf(
		"gcloud secrets versions access %[1]s/versions/%[2]s --out-file=%[4]s || gcloud secrets versions access %[2]s --secret=%[1]s --out-file=%[4]s; chmod +rX %[3]s %[4]s",
		secret.Group,
		secret.GroupVersion,
		secretDir,
		secretPath,
	)
	return []string{"sh", "-ec", args}
}

The secret is mounted at /etc/flyte/secrets/<SecretGroup>/<SecretGroupVersion>.

Azure Key Vault

The AzureSecretManagerInjector integrates with Azure Key Vault using the az CLI. It is designed to work with Azure Workload Identity Federation, requiring the pod to have access to a federated token file.

Usage

For Azure, the Group should be the full Vault URI of the secret.

secret := &core.Secret{
    Group:        "https://my-vault.vault.azure.net/secrets/my-secret",
    GroupVersion: "v1", // Optional: defaults to latest if empty
}

Internal Mechanism

The injector (in azure_secret_manager.go) performs an az login using the federated token before fetching the secret:

command := "az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token \"$(cat $AZURE_FEDERATED_TOKEN_FILE)\"; " +
    mkdirCmd + "az keyvault secret show --id \"%[1]s/%[2]s\" --query \"value\" -o tsv > %[3]s"

If GroupVersion is provided, the secret is stored at /etc/flyte/secrets/<secret_name>/<version>. If omitted, it is stored directly at /etc/flyte/secrets/<secret_name>.

Configuration

You can configure the sidecar images and resource requirements for each provider in the Flyte configuration. These settings are defined in flyteplugins/go/tasks/pluginmachinery/secret/config/config.go.

AWS Configuration

webhook:
  awsSecretManager:
    sidecarImage: "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4"
    resources:
      requests:
        cpu: "100m"
        memory: "100Mi"

GCP Configuration

webhook:
  gcpSecretManager:
    sidecarImage: "gcr.io/google.com/cloudsdktool/cloud-sdk:alpine"

Azure Configuration

webhook:
  azureSecretManager:
    sidecarImage: "mcr.microsoft.com/azure-cli:cbl-mariner2.0"

Security Requirements

For these injectors to function, the Kubernetes Pod must have the appropriate cloud permissions:

• AWS: The IAM Role associated with the Pod's ServiceAccount must have secretsmanager:GetSecretValue permissions for the requested ARNs.
• GCP: The Google Service Account (GSA) must have secretmanager.versions.access permissions. This is typically handled via Workload Identity.
• Azure: The User-Assigned Managed Identity must have permissions to pull secrets from the Key Vault, and the Pod must be configured for Workload Identity Federation.

Note that sidecar-based injection only supports FILE mount requirements. If a task requests a secret as an ENV_VAR, these cloud-native injectors will return an error, as they are designed to share secrets via a secure memory-backed volume.