Configuring Storage Backends

Flyte uses a unified storage abstraction called DataStore to interact with various backends like S3, Minio, Redis, and local filesystems. You configure these backends using the Config struct from the flytestdlib/storage package.

Configuring S3 or Minio Storage

To configure Flyte with S3 or Minio, use the StowConfig within the main Config object. While a legacy ConnectionConfig exists, the StowConfig is the modern approach for defining backend-specific parameters.

import (
    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/flyte/v2/flytestdlib/promutils"
    "github.com/flyteorg/stow/s3"
)

func createS3Store() (*storage.DataStore, error) {
    cfg := &storage.Config{
        Type: storage.TypeStow,
        Stow: storage.StowConfig{
            Kind: s3.Kind,
            Config: map[string]string{
                s3.ConfigAccessKeyID:     "my-access-key",
                s3.ConfigSecretKey:       "my-secret-key",
                s3.ConfigRegion:          "us-east-1",
                s3.ConfigEndpoint:        "http://minio:9000", // Optional for Minio
                s3.ConfigDisableSSL:      "true",              // Optional for local dev
            },
        },
        InitContainer: "my-flyte-bucket",
    }
    
    return storage.NewDataStore(cfg, promutils.NewTestScope())
}

Configuring Local Filesystem Storage

For local development or testing without a cloud provider, you can use the local stow backend. This maps a local directory to a Flyte container.

import (
    "github.com/flyteorg/flyte/v2/flytestdlib/storage"
    "github.com/flyteorg/stow/local"
)

cfg := &storage.Config{
    Type: storage.TypeStow,
    Stow: storage.StowConfig{
        Kind: local.Kind,
        Config: map[string]string{
            local.ConfigKeyPath: "/tmp/flyte-storage",
        },
    },
    InitContainer: "testdata",
}

In this setup, a reference like file://testdata/config.yaml will resolve to /tmp/flyte-storage/testdata/config.yaml.

Configuring Redis for Metadata

Flyte can use Redis as a raw store for small objects (metadata). Objects are stored as Redis string values keyed by the path portion of the DataReference (e.g., redis://<addr>/<key>).

import "github.com/flyteorg/flyte/v2/flytestdlib/storage"

cfg := &storage.Config{
    Type: storage.TypeRedis,
    Redis: storage.RedisConfig{
        Addr:     "localhost:6379",
        DB:       0,
        Password: "optional-password",
    },
}

Note: Redis has a hard cap of 512 MiB for string values. Large raw data should remain in a blob store like S3.

Enabling Hybrid Storage Routing

Flyte supports routing different URI schemes to different backends. If you provide a RedisConfig alongside a different primary storage Type (like s3), Flyte automatically routes redis:// references to Redis while using the primary store for everything else.

cfg := &storage.Config{
    Type: storage.TypeS3,
    // Primary S3 config
    Connection: storage.ConnectionConfig{
        Region: "us-east-1",
    },
    // Secondary Redis routing
    Redis: storage.RedisConfig{
        Addr: "redis-host:6379",
    },
    InitContainer: "my-bucket",
}

Configuring In-Memory Storage for Testing

For unit tests that require a DataStore without external dependencies, use TypeMemory.

import "github.com/flyteorg/flyte/v2/flytestdlib/storage"

store, err := storage.NewDataStore(&storage.Config{
    Type: storage.TypeMemory,
}, promutils.NewTestScope())

Key Configuration Parameters

Parameter	Description
InitContainer	The initial bucket or container name. This is required even if MultiContainerEnabled is true.
MultiContainerEnabled	If true, Flyte will automatically open connections to new containers/buckets as they are encountered in DataReference strings.
Limits.GetLimitMegabytes	Sets the maximum allowed download size (in MBs) per call to prevent excessive memory usage. Defaults to 2MB.

Troubleshooting Common Issues

• Missing InitContainer: Even when MultiContainerEnabled is set to true, you must provide an InitContainer. The storage layer uses this to initialize the base connection to the provider.
• Redis Signed URLs: Redis storage does not support Signed URLs. If your application logic requires GetSignedURL, you must use a blob store backend (S3, GCS, Azure).
• Stow Local Paths: The local stow backend expects the full path to include the container name. If your config path is /tmp/storage and your InitContainer is my-container, the physical path on disk will be /tmp/storage/my-container.