Generating Signed URLs

Flyte provides a mechanism to generate signed URLs for secure, temporary, and direct access to objects in cloud storage backends like S3, GCS, and Azure Blob Storage. This is primarily handled through the DataStore and its CreateSignedURL method.

Generating an Upload URL

To allow a client to upload a file directly to storage, generate a signed URL with stow.ClientMethodPut scope. You should include a ContentMD5 hash to ensure data integrity during the upload.

import (
    "context"
    "encoding/base64"
    "time"

    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/stow"
)

func GetUploadURL(ctx context.Context, store *storage.DataStore, path storage.DataReference, md5Sum []byte) (string, map[string]string, error) {
    expiresIn := 1 * time.Hour
    
    props := storage.SignedURLProperties{
        Scope:                 stow.ClientMethodPut,
        ExpiresIn:             expiresIn,
        ContentMD5:            base64.StdEncoding.EncodeToString(md5Sum),
        AddContentMD5Metadata: true,
    }

    signedResp, err := store.CreateSignedURL(ctx, path, props)
    if err != nil {
        return "", nil, err
    }

    // The client must include signedResp.RequiredRequestHeaders in their PUT request
    return signedResp.URL.String(), signedResp.RequiredRequestHeaders, nil
}

Generating a Download URL

To provide temporary access to a private object, generate a signed URL with stow.ClientMethodGet scope.

import (
    "context"
    "time"

    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/stow"
)

func GetDownloadURL(ctx context.Context, store *storage.DataStore, path storage.DataReference) (string, error) {
    props := storage.SignedURLProperties{
        Scope:     stow.ClientMethodGet,
        ExpiresIn: 30 * time.Minute,
    }

    signedResp, err := store.CreateSignedURL(ctx, path, props)
    if err != nil {
        return "", err
    }

    return signedResp.URL.String(), nil
}

Handling Required Headers

When generating signed URLs, especially for uploads (PUT), some storage backends require specific headers to be present in the final request. These are returned in the RequiredRequestHeaders field of SignedURLResponse.

Common headers include:

• Content-MD5: If ContentMD5 was provided in SignedURLProperties.
• x-amz-content-sha256: Often required by AWS S3 for certain configurations.

If these headers are not included by the client making the request to the signed URL, the storage backend will reject the request.

Configuring External Access

In many deployments, Flyte services access storage via an internal endpoint (e.g., a VPC endpoint or a local Minio service), but clients need signed URLs that point to an external, publicly accessible endpoint.

Use SignedURLConfig to override the stow configuration specifically for URL generation:

storage:
  type: s3
  stow:
    kind: s3
    config:
      endpoint: http://minio.flyte.svc.cluster.local:9000
      region: us-east-1
  signedUrl:
    stowConfigOverride:
      endpoint: https://s3.external-domain.com

In the code, this is represented by the SignedURLConfig struct:

// From flytestdlib/storage/config.go
type SignedURLConfig struct {
    StowConfigOverride map[string]string `json:"stowConfigOverride,omitempty" pflag:"-,Configuration for stow backend."`
}

Troubleshooting and Limitations

Unsupported Backends

The CreateSignedURL method is implemented by the StowStore. Backends that do not support pre-signing (such as the Redis-based store) will return an error when this method is called.

MD5 Encoding

The ContentMD5 field in SignedURLProperties must be a Base64-encoded string of the 128-bit MD5 hash. Providing a hex-encoded string or raw bytes will cause signature mismatch errors in the cloud provider.

Metadata Persistence

If AddContentMD5Metadata is set to true, Flyte will attempt to store the MD5 hash as metadata on the object itself. This is useful for subsequent integrity checks using Head requests, which return StowMetadata containing the ContentMD5.