Data & Storage Abstraction

Flyte provides a unified data and storage abstraction layer that allows the platform to interact with various cloud and local storage backends through a consistent interface. This system is primarily implemented in the flytestdlib/storage package and extended by the dataproxy service for high-level data translation.

Unified Storage with DataStore

The DataStore struct in flytestdlib/storage/storage.go is the central entry point for all storage operations. It abstracts away the complexities of different cloud providers (AWS S3, GCP GCS, Azure Blob Storage) and local filesystems by aggregating low-level byte access and high-level object serialization.

type DataStore struct {
	ComposedProtobufStore
	ReferenceConstructor
	metrics *dataStoreMetrics
}

A DataStore is typically initialized using a Config object that specifies the storage type (e.g., s3, local, mem, redis) and connection details.

DataReference Addressing

Flyte uses DataReference (a string type) as a universal URI scheme to address data across different backends. A typical reference looks like s3://my-bucket/path/to/data. The DataStore provides a Split() method to decompose these URIs into their constituent parts:

func (r DataReference) Split() (scheme, container, key string, err error) {
	u, err := url.Parse(string(r))
	// ... logic to extract scheme, container (host), and key (path)
	return u.Scheme, container, strings.Trim(u.Path, "/"), nil
}

Layered Access Interfaces

The storage abstraction is divided into two primary layers: byte-level access and Protobuf-level access.

RawStore (Byte-Level)

The RawStore interface defines methods for low-level operations on raw bytes. This is used for reading and writing large data blobs, logs, or any non-structured data.

type RawStore interface {
	ReadRaw(ctx context.Context, reference DataReference) (io.ReadCloser, error)
	WriteRaw(ctx context.Context, reference DataReference, size int64, opts Options, raw io.Reader) error
	Head(ctx context.Context, reference DataReference) (Metadata, error)
	Delete(ctx context.Context, reference DataReference) error
	// ... other methods like List and CopyRaw
}

ProtobufStore (Object-Level)

The ProtobufStore interface provides a typed way to interact with storage by automatically serializing and deserializing Protobuf messages.

type ProtobufStore interface {
	ReadProtobuf(ctx context.Context, reference DataReference, msg proto.Message) error
	WriteProtobuf(ctx context.Context, reference DataReference, opts Options, msg proto.Message) error
}

The default implementation of ProtobufStore uses an underlying RawStore to perform the actual I/O, handling the marshaling and unmarshaling of messages transparently.

Data Translation and Proxying

While the storage layer handles how data is persisted, the dataproxy service handles how Flyte's internal data types (Literals) are presented to external consumers like the Flyte Console (UI).

Literal to JSON Translation

The TranslatorService in dataproxy/translator.go uses the DataStore to retrieve offloaded data and then converts it into a JSON format compatible with React JSON Schema Form (RSJF). This is critical for rendering launch forms in the UI.

For example, when the UI needs to display a launch form for a workflow with offloaded inputs, the readOffloadedLiterals method fetches the data:

func (s *TranslatorService) readOffloadedLiterals(
	ctx context.Context,
	req *workflow.LiteralsToLaunchFormJsonRequest,
) ([]*task.NamedLiteral, error) {
	uri := req.GetLiteralsUri()
	var inputsOrOutputs task.Inputs
	if err := s.dataStore.ReadProtobuf(ctx, storage.DataReference(uri), &inputsOrOutputs); err != nil {
		return nil, connect.NewError(connect.CodeInternal, fmt.Errorf("failed to read literals from %s: %w", uri, err))
	}
	return inputsOrOutputs.GetLiterals(), nil
}

The actual conversion logic resides in dataproxy/converter/literal_json_converter.go, where LiteralsToLaunchFormJson maps complex Flyte types (like Blob, Schema, or Union) to JSON schemas.

RSJF Compliance

The converter handles specific Flyte types by adding custom metadata to the JSON schema. For instance, a Flyte Blob is converted into an object with uri, format, and dimensionality properties, marked with a custom format: "blob" for the UI to recognize:

case *core.LiteralType_Blob:
    // ...
    return map[string]any{
        "type":       "object",
        "format":     "blob",
        "properties": properties,
    }, nil

Configuration and Backends

Flyte supports multiple storage backends through the stow library and custom implementations. Configuration is managed via the StorageConfig struct, which includes:

• Type: The backend to use (s3, minio, local, mem, stow, redis).
• Connection: Backend-specific settings like endpoint, auth, and region.
• Limits: maxDownloadMBs (defaulting to 2 MiB) to prevent excessive memory usage during data retrieval.
• Multi-container: enable-multicontainer allows a single DataStore to access multiple buckets/containers if the underlying credentials permit it.

When using the redis backend, Flyte derives keys directly from the path portion of the DataReference (e.g., redis://<addr>/<key>), allowing it to be used as a high-speed metadata or small-object cache.