Storage Performance and Caching

When Flyte components like the Propeller or Admin frequently access the same metadata or small blobs from remote storage, high latency and API costs can accumulate. Flyte addresses this by providing an optional in-memory caching layer in flytestdlib that wraps the raw storage provider (such as S3 or GCS) to accelerate data retrieval.

In-Memory Caching Architecture

Flyte implements caching using a decorator pattern. The cachedRawStore class in flytestdlib/storage/cached_rawstore.go wraps any implementation of the RawStore interface. It uses freecache to store raw bytes in-memory, keyed by their DataReference.

The cache intercepts three primary operations:

• ReadRaw: Checks the cache first. On a miss, it fetches from the underlying RawStore, populates the cache, and returns the data.
• WriteRaw: Writes data to the underlying RawStore and simultaneously updates the cache.
• Delete: Removes the entry from both the cache and the underlying RawStore.

Cache Population Logic

In ReadRaw, Flyte reads the entire object into memory before returning it to the caller if a cache miss occurs. This is visible in the implementation:

func (s *cachedRawStore) ReadRaw(ctx context.Context, reference DataReference) (io.ReadCloser, error) {
    // ... check cache ...
    reader, err := s.RawStore.ReadRaw(ctx, reference)
    if err != nil {
        return nil, err
    }
    
    // Read entire content to populate cache
    b, err := ioutils.ReadAll(reader, s.metrics.FetchLatency.Start())
    if err != nil {
        return nil, err
    }

    err = s.cache.Set(key, b, 0)
    if err != nil {
        // Wrap error if cache population fails (e.g. object too large)
        err = errors.Wrapf(ErrFailedToWriteCache, err, "Failed to Cache the metadata")
    }

    return ioutils.NewBytesReadCloser(b), err
}

Configuration and GC Tuning

Caching is configured via the CachingConfig struct in flytestdlib/storage/config.go. It is disabled by default and must be explicitly enabled by setting a non-zero size.

Field	Description
MaxSizeMegabytes	The total memory allocated for the freecache instance. If set to 0, caching is disabled.
TargetGCPercent	Adjusts the Go runtime's garbage collection target. This affects the entire process.

When TargetGCPercent is provided, Flyte calls debug.SetGCPercent during the initialization of the cachedRawStore. This is often used to reduce GC overhead in memory-intensive components by allowing the heap to grow larger before triggering a collection.

Observability and Metrics

Flyte provides comprehensive metrics for storage operations, aggregated in the dataStoreMetrics struct. These metrics allow operators to monitor cache efficiency and backend storage performance.

Cache Metrics

The cacheMetrics class tracks the effectiveness of the in-memory layer:

• CacheHit / CacheMiss: Counters for ReadRaw and Delete operations.
• CacheWriteError: Incremented when freecache fails to store an entry (e.g., due to size limits).
• FetchLatency: A stopwatch measuring the time taken to retrieve data from the remote backend on a cache miss.

Backend Storage Metrics (stowMetrics)

For the underlying storage (typically S3 or GCS via the stow library), Flyte tracks detailed operation latencies and failure rates in stowMetrics:

• Head/List/Read/Write/Delete Latency: Measured using both labeled.StopWatch and labeled.HistogramStopWatch.
• Failure Counters: Specific counters for each operation type (e.g., WriteFailure, ReadFailure).

Copy Metrics

The copyMetrics class in flytestdlib/storage/copy_impl.go tracks the performance of the CopyRaw operation:

• CopyLatency: Total time for the copy operation.
• ComputeLengthLatency: Time spent determining the source file size before copying.

Performance Considerations and Constraints

The 1/1024 Size Limit

Flyte uses freecache, which imposes a strict limit on individual object sizes. If an object's size exceeds 1/1024 of the total MaxSizeMegabytes, the entry will not be stored in the cache. When this happens, ReadRaw or WriteRaw will return an error wrapped with ErrFailedToWriteCache.

Error Handling

Because a cache failure is often non-fatal for the overall storage operation, Flyte provides a utility to identify these specific errors. In flytestdlib/storage/utils.go:

func IsFailedWriteToCache(err error) bool {
	return errors.IsCausedBy(err, ErrFailedToWriteCache)
}

Applications can use this to log a warning rather than failing a task if the cache is full or the object is too large.

Naive Copy Implementation

The current implementation of CopyRaw in Flyte is "naive"—it does not use cloud-native multi-part copies or server-side copying. Instead, it downloads the entire source object into a buffer and then uploads it to the destination. This can lead to high memory consumption when copying large files that are not already in the cache.

Memory Management

Since the cache resides in the application's heap, large cache sizes will increase the memory footprint of Flyte components. Operators should balance MaxSizeMegabytes against the available container memory limits and use TargetGCPercent to tune how aggressively the Go runtime reclaims memory.