Skip to main content

Secure External Access with Signed URLs

When external clients or browser-based tools need to access raw data in Flyte's storage without full cloud credentials, you can generate time-limited, secure URLs. Flyte provides this capability through the DataStore.CreateSignedURL method, which delegates the signing process to the underlying storage provider (such as AWS S3 or Google Cloud Storage).

Generating an Upload URL​

To allow an external client to upload data directly to a specific storage path, generate a signed URL with stow.ClientMethodPut. It is strongly recommended to include a ContentMD5 hash to ensure data integrity during the upload.

import (
    "context"
    "encoding/base64"
    "time"

    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/stow"
)

func GetUploadURL(ctx context.Context, store *storage.DataStore, path storage.DataReference, md5Sum []byte) (string, map[string]string, error) {
    // Define properties for the signed URL
    props := storage.SignedURLProperties{
        Scope:                 stow.ClientMethodPut,
        ExpiresIn:             time.Hour,
        ContentMD5:            base64.StdEncoding.EncodeToString(md5Sum),
        AddContentMD5Metadata: true,
    }

    // Generate the signed URL and required headers
    resp, err := store.CreateSignedURL(ctx, path, props)
    if err != nil {
        return "", nil, err
    }

    // The client MUST include resp.RequiredRequestHeaders in their HTTP PUT request
    return resp.URL.String(), resp.RequiredRequestHeaders, nil
}

Generating a Download URL​

For read-only access to artifacts or reports, use stow.ClientMethodGet. This is commonly used in the Flyte DataProxy service to provide temporary links for downloading task outputs.

import (
    "context"
    "time"

    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/stow"
)

func GetDownloadURL(ctx context.Context, store *storage.DataStore, path storage.DataReference) (string, error) {
    props := storage.SignedURLProperties{
        Scope:     stow.ClientMethodGet,
        ExpiresIn: 30 * time.Minute,
    }

    resp, err := store.CreateSignedURL(ctx, path, props)
    if err != nil {
        return "", err
    }

    return resp.URL.String(), nil
}

Handling Required Headers​

The SignedURLResponse includes a RequiredRequestHeaders map. For many storage providers (especially S3 when using metadata or MD5 checks), the request will be rejected if these headers are not present in the final HTTP call made by the client.

When building a response for an external client, ensure these headers are passed along:

// Example from dataproxy/service/dataproxy_service.go
resp := &dataproxy.CreateUploadLocationResponse{
    SignedUrl: signedResp.URL.String(),
    NativeUrl: storagePath.String(),
    Headers:   signedResp.RequiredRequestHeaders, // Mandatory for the client to use
}

Configuring Expiration Limits​

Flyte allows you to enforce maximum expiration durations for signed URLs to prevent overly long-lived access tokens. These are configured in the DataProxy service settings.

In your configuration file:

dataproxy:
  download:
    maxExpiresIn: 1h
  upload:
    maxExpiresIn: 1h

These limits are mapped to the DataProxyDownloadConfig struct in dataproxy/config/config.go.

Storage Backend Overrides​

If you need to use different credentials or a specific endpoint for generating signed URLs (e.g., a public-facing endpoint vs. an internal one), use SignedURLConfig. This allows you to override the default stow configuration specifically for the signing operation.

// flytestdlib/storage/config.go
type SignedURLConfig struct {
    // StowConfigOverride allows overriding credentials or endpoints for signed URL generation.
    StowConfigOverride map[string]string `json:"stowConfigOverride,omitempty"`
}

Troubleshooting and Limitations​

Unsupported Backends​

Not all storage backends supported by Flyte implement signed URLs. Specifically, the Redis implementation (TypeRedis) does not support this feature. Calling CreateSignedURL against a Redis-backed DataStore will return an error.

MD5 Mismatches​

If you provide a ContentMD5 in SignedURLProperties, the storage provider will validate the uploaded content against this hash. If the client uploads data that does not match the hash, the storage provider (e.g., S3) will return a 400 Bad Request or SignatureDoesNotMatch error.

Header Requirements​

If a client receives a 403 Forbidden when using a generated URL, verify that they are sending all headers returned in SignedURLResponse.RequiredRequestHeaders. Common missing headers include Content-MD5 or x-amz-meta-* headers.