Skip to main content

Configuring AWS Secrets Manager

Flyte supports two primary methods for integrating with AWS Secrets Manager: Sidecar-based and Embedded. Sidecar-based integration uses an AWS-provided init-container to fetch secrets, while Embedded integration fetches secrets directly within the Flyte Pod Webhook and injects them into the task pod.

Sidecar-based Integration​

The sidecar-based integration (SecretManagerTypeAWS) injects an AWS Secrets Manager sidecar as an init-container for every secret requested by a task. This method is robust but can increase pod startup time if many secrets are used, as each secret results in a separate init-container.

Configuration​

To enable sidecar-based integration, update the webhook section of your Flyte configuration. The configuration is defined in flyteplugins/go/tasks/pluginmachinery/secret/config/config.go.

webhook:
  secretManagerTypes:
    - AWS
  awsSecretManager:
    # The sidecar image used to pull secrets
    sidecarImage: "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4"
    resources:
      requests:
        cpu: 200m
        memory: 500Mi
      limits:
        cpu: 200m
        memory: 500Mi

Usage and Permissions​

When using sidecar mode, secrets are mounted as files at /etc/flyte/secrets/<group>/<key>.

  1. IAM Permissions: The Task Pod's ServiceAccount must have the secretsmanager:GetSecretValue permission for the target secrets.
  2. Secret Definition: Define the secret using the ARN as the group and the secret name as the key. The AWSSecretManagerInjector joins these to form the full identifier.
from flytekit import task, Secret

# The sidecar will use the group as the ARN and key as the filename
@task(secret_requests=[Secret(group="arn:aws:secretsmanager:region:account:secret:my-secret", key="my-key")])
def my_task():
    # Secret will be available at /etc/flyte/secrets/arn:aws:secretsmanager:region:account:secret:my-secret/my-key
    ...

Embedded Integration​

The embedded integration (SecretManagerTypeEmbedded) is more efficient as it avoids multiple sidecars. The Flyte Pod Webhook fetches the secret values during the pod mutation phase using the AWSSecretFetcher and injects them directly as environment variables or via a single shared init-container for file mounts.

Configuration​

To enable embedded integration, configure the webhook to use the Embedded type and specify AWS as the provider:

webhook:
  secretManagerTypes:
    - Embedded
  embeddedSecretManagerConfig:
    type: AWS
    awsConfig:
      region: "us-west-2"
    fileMountInitContainer:
      image: "public.ecr.aws/docker/library/busybox:latest"
      resources:
        requests:
          cpu: 100m
          memory: 100Mi

Naming Convention and Scoping​

Embedded mode uses a hierarchical naming convention to look up secrets in AWS Secrets Manager. It searches for secrets in the following order of specificity (defined in flyteplugins/go/tasks/pluginmachinery/secret/utils.go):

  1. u__org__<org>__domain__<domain>__project__<project>__key__<key>
  2. u__org__<org>__domain__<domain>__key__<key>
  3. u__org__<org>__key__<key>

Usage and Permissions​

  1. IAM Permissions: The Flyte Pod Webhook's ServiceAccount must have the secretsmanager:GetSecretValue permission for the secrets it needs to fetch.
  2. Secret Definition: Define the secret in your task using the key. The group is ignored in embedded mode as the lookup is based on the pod's labels (org, domain, project).
from flytekit import task, Secret

@task(secret_requests=[Secret(key="my-api-key", mount_requirement=Secret.MountRequirement.ENV_VAR)])
def my_task():
    # Secret will be available as an environment variable (default prefix _UNION_)
    # e.g., _UNION_MY-API-KEY
    ...

Configuration Reference​

The following configuration keys are available under the webhook section:

KeyDefaultDescription
awsSecretManager.sidecarImagedocker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4Image for the AWS sidecar init-container.
embeddedSecretManagerConfig.typeAWSThe provider for embedded secret management.
embeddedSecretManagerConfig.awsConfig.region""The AWS region for the embedded fetcher.
embeddedSecretManagerConfig.fileMountInitContainer.imagepublic.ecr.aws/docker/library/busybox:latestImage used to write secrets to files in embedded mode.
secretEnvVarPrefix_UNION_Prefix for secret environment variables.

Troubleshooting​

  • Sidecar Failures: If the sidecar init-container fails, check the Task Pod's IAM permissions. The sidecar logs will typically indicate an AccessDenied error if permissions are missing.
  • Embedded Lookup Failures: If a secret is not found in embedded mode, ensure the secret name in AWS Secrets Manager exactly matches the u__org__... format. The Flyte Pod Webhook logs will show the exact IDs it attempted to fetch.
  • File Mount Paths:
    • In Sidecar mode, secrets are mounted under /etc/flyte/secrets/<group>/<key>.
    • In Embedded mode, secrets are mounted under /etc/flyte/secrets/<key>.
  • Binary Secrets: In embedded mode, if a secret is stored as binary data in AWS, it must be a valid UTF-8 string to be mounted as an environment variable; otherwise, it must be requested as a FILE mount.