Path Management with DataReferences

In Flyte, managing data locations across different cloud providers (AWS S3, Google GCS, Azure Blob Storage) and local filesystems requires a consistent way to build and parse URIs. Flyte uses the DataReference type and the ReferenceConstructor interface to ensure that paths are constructed safely without manual string manipulation that often leads to errors with trailing slashes or scheme-specific nuances.

In this tutorial, you will learn how to initialize a storage provider, build nested data paths, and parse existing references into their constituent parts.

Prerequisites

To follow this tutorial, you need the following packages from Flyte:

• github.com/flyteorg/flyte/v2/flytestdlib/storage
• github.com/flyteorg/flyte/v2/flytestdlib/promutils

1. Initialize the DataStore

The DataStore is the primary entry point for all storage operations in Flyte. It embeds a ReferenceConstructor, which provides the logic for building paths.

import (
	"context"
	"github.com/flyteorg/flyte/v2/flytestdlib/promutils"
	"github.com/flyteorg/flyte/v2/flytestdlib/storage"
)

func main() {
	ctx := context.Background()
	testScope := promutils.NewTestScope()

	// Configure a local storage backend for this example
	cfg := &storage.Config{
		Type:          storage.TypeLocal,
		InitContainer: "my-bucket",
	}

	// NewDataStore returns a DataStore which includes a ReferenceConstructor
	store, err := storage.NewDataStore(cfg, testScope.NewSubScope("tutorial"))
	if err != nil {
		panic(err)
	}
}

The DataStore uses a URLPathConstructor by default for most storage types, which treats paths as URL-compatible strings.

2. Construct Nested References

Instead of using fmt.Sprintf or manual string joining, use the ConstructReference method. This method ensures that separators are handled correctly, regardless of whether your base path or nested keys have leading or trailing slashes.

// Define a base reference
baseRef := storage.DataReference("s3://my-bucket/metadata")

// Construct a nested path: s3://my-bucket/metadata/workflows/execution_1/inputs.pb
nestedRef, err := store.ConstructReference(ctx, baseRef, "workflows", "execution_1/", "/inputs.pb")
if err != nil {
    panic(err)
}

fmt.Println(nestedRef.String()) 
// Output: s3://my-bucket/metadata/workflows/execution_1/inputs.pb

The URLPathConstructor implementation in flytestdlib/storage/url_path.go uses url.ResolveReference internally. It ensures the base reference ends with a / before resolving, preventing common bugs where the last segment of a base path is accidentally replaced.

3. Parse a DataReference

When you receive a DataReference (which is a type alias for string), you often need to extract the bucket (container) or the specific key. The Split() method handles this, including special logic for cloud providers like Azure.

ref := storage.DataReference("s3://my-bucket/path/to/data")
scheme, container, key, err := ref.Split()
if err != nil {
    panic(err)
}

fmt.Printf("Scheme: %s, Bucket: %s, Key: %s\n", scheme, container, key)
// Output: Scheme: s3, Bucket: my-bucket, Key: path/to/data

Handling Azure ADLS Gen2

Flyte's Split() implementation in flytestdlib/storage/storage.go contains specific logic for Azure abfs and abfss schemes. In Azure, the filesystem is often encoded in the userinfo position (e.g., abfs://container@account.dfs.core.windows.net/path). Flyte correctly identifies the container from the username field in these cases.

4. Enable Multi-Container Access

By default, Flyte may restrict operations to the InitContainer defined in your configuration. If you need to build and access references across different buckets, you must enable MultiContainerEnabled in your storage.Config.

cfg := &storage.Config{
    Type:                  storage.TypeS3,
    InitContainer:         "default-bucket",
    MultiContainerEnabled: true, // Allows references to any bucket
}

If this is set to false, the storage layer will reject requests to references that do not belong to the InitContainer.

Summary

You have now built a safe path management flow using Flyte's storage utilities:

1. Initialized a DataStore to gain access to path construction logic.
2. Constructed nested paths using ConstructReference to avoid slash-related errors.
3. Parsed URIs using Split() to extract metadata safely across different cloud schemes.
4. Configured the system to allow cross-bucket data references.

For next steps, explore the RawStore interface within the DataStore to perform ReadRaw and WriteRaw operations using the references you've constructed.