Caching and Performance Optimization

Flyte implements a TTL-based caching strategy within its control plane to optimize performance and reduce the overhead of cross-plane RPC calls. By intercepting requests in the AppService before they reach the data plane, Flyte minimizes redundant data retrieval for frequently accessed application metadata.

The Proxy Architecture

The AppService in app/service/app_service.go acts as a caching proxy for the InternalAppService (the data plane). It implements the AppServiceHandler interface and maintains an internal client to forward requests when a cache miss occurs.

type AppService struct {
	appconnect.UnimplementedAppServiceHandler
	internalClient appconnect.AppServiceClient
	// cache is nil when cacheTTL=0 (caching disabled).
	cache *expirable.LRU[string, *flyteapp.App]
}

This design separates the concerns of data persistence and state management (handled by the data plane) from the performance optimizations required by the control plane and UI.

TTL-Based Caching Strategy

The primary optimization occurs during Get requests. When a request for a specific application arrives, the AppService attempts to retrieve the application from its in-memory LRU (Least Recently Used) cache using a key derived from the application's project, domain, and name.

Cache Lookup and Population

The Get implementation follows a standard "read-through" pattern but adds a specific check for the application's state before caching:

func (s *AppService) Get(
	ctx context.Context,
	req *connect.Request[flyteapp.GetRequest],
) (*connect.Response[flyteapp.GetResponse], error) {
	appID, ok := req.Msg.GetIdentifier().(*flyteapp.GetRequest_AppId)
	if ok && appID.AppId != nil && s.cache != nil {
		if app, hit := s.cache.Get(cacheKey(appID.AppId)); hit {
			return connect.NewResponse(&flyteapp.GetResponse{App: app}), nil
		}
	}

	resp, err := s.internalClient.Get(ctx, req)
	if err != nil {
		return nil, err
	}
    
    // Only cache if the state is not transitional
	if ok && appID.AppId != nil && s.cache != nil && !isTransitionalState(resp.Msg.GetApp()) {
		s.cache.Add(cacheKey(appID.AppId), resp.Msg.GetApp())
	}
	return resp, nil
}

The Transitional State Guard

A critical design choice in Flyte's caching logic is the exclusion of "transitional states." If an application is in the process of starting up—where the user has requested an ACTIVE state but the deployment status is still STOPPED—the AppService refuses to cache the result.

func isTransitionalState(app *flyteapp.App) bool {
	if app == nil {
		return false
	}
	if app.GetSpec().GetDesiredState() == flyteapp.Spec_DESIRED_STATE_STOPPED {
		return false
	}
	for _, cond := range app.GetStatus().GetConditions() {
		if cond.GetDeploymentStatus() == flyteapp.Status_DEPLOYMENT_STATUS_STOPPED {
			return true
		}
	}
	return false
}

This prevents a common UI synchronization issue: if the control plane cached a "Stopped" status immediately after a user clicked "Start," the UI would continue to show the application as stopped for the duration of the TTL (default 30s), even if the application successfully started within seconds. By bypassing the cache during these transitions, Flyte ensures the UI remains responsive to rapid state changes.

Cache Invalidation and Consistency

To maintain data consistency, the AppService explicitly invalidates cache entries during mutation operations. The Create, Update, and Delete methods all call s.cache.Remove() after a successful call to the internal data plane service.

For example, the Update method ensures that any subsequent Get request will fetch the fresh state:

func (s *AppService) Update(
	ctx context.Context,
	req *connect.Request[flyteapp.UpdateRequest],
) (*connect.Response[flyteapp.UpdateResponse], error) {
	resp, err := s.internalClient.Update(ctx, req)
	if err != nil {
		return nil, err
	}
	if s.cache != nil {
		s.cache.Remove(cacheKey(req.Msg.GetApp().GetMetadata().GetId()))
	}
	return resp, nil
}

Non-Cached Operations

Not all operations benefit from caching. The List and Watch methods always forward requests directly to the InternalAppService:

• List: Results vary significantly based on filters and pagination, making simple key-based caching ineffective.
• Watch: This is a server-streaming RPC designed for real-time updates, which inherently bypasses the need for a static snapshot cache.

Configuration

The caching behavior is controlled via AppConfig in app/config/config.go. Operators can tune the CacheTTL based on the expected load and the desired balance between performance and state consistency.

type AppConfig struct {
	// InternalAppServiceURL is the base URL of the InternalAppService (data plane).
	InternalAppServiceURL string `json:"internalAppServiceUrl" pflag:",URL of the internal app service"`

	// CacheTTL is the TTL for the in-memory app status cache.
	// Defaults to 30s. Set to 0 to disable caching.
	CacheTTL time.Duration `json:"cacheTtl" pflag:",TTL for app status cache"`
}

Setting CacheTTL to 0 completely disables the caching layer, forcing the AppService to act as a pure transparent proxy.