Flyte Internal Component Architecture
The Flyte internal architecture is centered around a unified manager that orchestrates several modular services. The runs.service acts as the primary entry point for users, managing the lifecycle of workflow runs and persisting state to a database. It delegates the execution of individual tasks to the actions.service, which translates them into Kubernetes Custom Resources (TaskActions).
The executor.pkg.controller is a Kubernetes-based reconciler that watches these TaskAction resources. It leverages the Task Plugin System machinery to execute the actual work, whether it's a standard Kubernetes Pod or a specialized job like Spark or Ray. During execution, the executor reports progress and terminal states through the events.service, which proxies these updates back to the runs service to maintain a consistent view of the system.
Data management is handled by flytestdlib.storage, providing a unified interface for interacting with object stores (S3, GCS, etc.). The dataproxy service facilitates log streaming and data movement, while the cache_service optimizes execution by managing task output memoization. This modular design allows Flyte to scale execution across Kubernetes clusters while maintaining a centralized management plane.
Key Architectural Findings:
- The 'manager' is a unified entry point that initializes and mounts all sub-services (runs, actions, events, cache, dataproxy, executor).
- The 'runs.service' implements both public workflow APIs and an 'InternalRunService' used for recording execution events.
- The 'actions.service' acts as a bridge between the Flyte API and Kubernetes, managing 'TaskAction' CRDs.
- The 'executor' is a Kubernetes controller that uses 'flyteplugins' to execute tasks and 'events.service' to report status back to the 'runs.service'.
- 'flytestdlib.storage' provides a common 'DataStore' abstraction used by the API, executor, and plugins for input/output handling.