Workflow Execution and Event Flow
This sequence diagram illustrates the asynchronous workflow execution and event reporting flow in the Flyte-like system.
The process begins with a CLI (represented by the test client) requesting a new run from the Run Service. The Runs Service handles the initial setup, including persisting the run to the database and offloading inputs to storage, before enqueuing the root action via the Actions Service.
The Actions Service translates this request into a TaskAction Custom Resource (CR) within Kubernetes Task Lifecycle. The Executor Controller, running as a Kubernetes operator, reconciles this CR by invoking the appropriate plugin (e.g., to launch a Pod).
As the task progresses, the Executor Controller reports state changes back through the Events Subsystem (Events Proxy Service), which forwards them to the Runs Service for final persistence in the database. Simultaneously, the Actions Service watches for CR updates in Kubernetes to keep the run state synchronized.
Key Architectural Findings:
- The Runs Service acts as the primary entry point, managing run metadata and input storage before delegating execution to the Actions Service.
- The Actions Service uses Kubernetes Custom Resources (TaskAction) as the source of truth for task execution, enabling a declarative approach.
- The Executor Controller is a Kubernetes operator that reconciles TaskAction CRs by invoking plugins that manage the lifecycle of underlying compute resources like Pods.
- Event reporting is decoupled: the Executor Controller pushes events to an Events Subsystem, which then proxies them back to the Runs Service to update the database.
- The Actions Service maintains a watch on Kubernetes resources to provide real-time updates back to clients and ensure the database reflects the cluster state.