Skip to main content

Architecture Overview

This section contains architecture diagrams and documentation for Flyte.

Available Diagrams​

Flyte System Context Diagram​

The Flyte System Context diagram illustrates the high-level interactions between the Flyte platform and its external environment.

At the center is the Flyte Service, a Go-based gRPC system (using ConnectionWrapper) that manages the lifecycle of workflows, tasks, and runs. It is composed of several sub-services including Admin, Runs, and Tasks.

Users (such as Data Scientists and ML Engineers) interact with the system primarily through CLI Tools (like flytectl) or a Web UI. Flyte Copilot is a specialized CLI tool used as a sidecar within Kubernetes pods to manage data movement.

The system relies on several external infrastructure components:

  • PostgreSQL Database: Stores persistent metadata about workflows, projects, and execution history.
  • Cloud Storage: Uses a multi-cloud abstraction (StowStore) to store large data artifacts in AWS S3, Google Cloud Storage, or Azure Blob Storage.
  • Kubernetes API: Flyte orchestrates workflow execution by interacting with Kubernetes to manage pods and custom resources. The Flyte Executor component specifically acts as a Kubernetes controller.
  • Secret Management Services: Integrates with AWS Secrets Manager, Google Secret Manager, Azure Key Vault, and HashiCorp Vault to securely inject credentials into tasks.
  • Observability Stack: Exports metrics to Prometheus and traces to OpenTelemetry/Jaeger for system monitoring and debugging.
  • External Frameworks: Supports distributed computing frameworks like Spark, Ray, and Dask via their respective Kubernetes operators.

Key Architectural Findings:

  • Flyte is a gRPC-based service using Connect RPC for its API layer.
  • It uses a PostgreSQL database for metadata persistence and Cloud Storage (S3/GCS/Azure) for data artifacts.
  • Kubernetes is the primary orchestration engine, with Flyte acting as a controller for custom resources.
  • The system supports multiple secret managers including AWS, GCP, Azure, and HashiCorp Vault.
  • Observability is handled through Prometheus for metrics and OpenTelemetry for distributed tracing.
  • Flyte Copilot serves as a sidecar/init container for data handling within execution pods.

Flyte Internal Component Architecture​

The Flyte internal component architecture follows a service-oriented design, often running in a unified process managed by the Manager. The system is divided into a control plane for workflow management, an execution plane for task reconciliation, and a data plane for handling large datasets and logs.

Key Components​

  • Manager: The central entry point that initializes and orchestrates all internal services, including database migrations, storage connections, and Kubernetes clients.
  • runs.service: The core workflow engine responsible for managing the lifecycle of runs, tasks, and triggers. It persists metadata in PostgresConfig and offloads large input data to the DataStore.
  • ActionsService: Acts as a bridge between the Runs Service and Kubernetes. It translates action requests into TaskAction Custom Resources (CRDs) and watches for status updates to report back to the core engine.
  • executor.pkg.controller: A Kubernetes controller that reconciles TaskAction CRDs. It uses Core Plugin Architecture to execute various task types (e.g., K8s Pods, Spark jobs) and reports execution events.
  • dataproxy: Provides a unified interface for data upload/download and log streaming, abstracting the underlying storage and Kubernetes log APIs.
  • cache_service: A dedicated service for task result caching, allowing the Executor to skip redundant computations by checking for previously successful executions.
  • Core Plugin Architecture: A library of execution logic used by the Executor to interact with external compute providers and orchestrate task-specific resources.

Data Flow​

  1. A User initiates a run through the Runs Service.
  2. The Runs Service persists the run metadata and calls the Actions Service to enqueue the root action.
  3. The Actions Service creates a TaskAction CRD in Kubernetes.
  4. The Executor detects the new CRD and uses FlytePlugins to launch the actual task (e.g., a Pod).
  5. As the task progresses, the Executor reports events to the Events Service, which proxies them back to the Runs Service to update the workflow state.
  6. The Actions Service also watches the CRD status and provides an independent update path for action-level transitions.
  7. The DataProxy facilitates data movement between the user, the execution environment, and the DataStore.

Key Architectural Findings:

  • The 'manager' component serves as a unified orchestrator that wires together all services, including database and storage initialization.
  • 'runs.service' manages high-level workflow state and uses 'actions.service' as a gRPC-to-K8s bridge.
  • 'executor.pkg.controller' is a standard K8s controller that implements the 'TaskAction' reconciliation loop.
  • 'flyteplugins' provides the actual execution logic for different task types, decoupled from the controller machinery.
  • 'dataproxy' and 'cache_service' provide specialized data and performance optimizations, interacting with both the control and execution planes.
  • The architecture uses a 'Events Service' proxy to decouple the Executor's event reporting from the Runs Service's internal update logic.

Workflow Execution and Event Flow​

This sequence diagram illustrates the asynchronous workflow execution and event reporting flow in the Flyte-like system.

The process begins with a flytecopilot.cmd (represented by the test client) requesting a new run from the RunService. The Runs Service handles the initial setup, including persisting the run to the database and offloading inputs to storage, before enqueuing the root action via the ActionsService.

The Actions Service translates this request into a TaskAction Custom Resource (CR) within Kubernetes Task Lifecycle. The executor, running as a Kubernetes operator, reconciles this CR by invoking the appropriate plugin (e.g., to launch a Pod).

As the task progresses, the Executor Controller reports state changes back through the events (Events Proxy Service), which forwards them to the Runs Service for final persistence in the database. Simultaneously, the Actions Service watches for CR updates in Kubernetes to keep the run state synchronized.

Key Architectural Findings:

  • The Runs Service acts as the primary entry point, managing run metadata and input storage before delegating execution to the Actions Service.
  • The Actions Service uses Kubernetes Custom Resources (TaskAction) as the source of truth for task execution, enabling a declarative approach.
  • The Executor Controller is a Kubernetes operator that reconciles TaskAction CRs by invoking plugins that manage the lifecycle of underlying compute resources like Pods.
  • Event reporting is decoupled: the Executor Controller pushes events to an Events Subsystem, which then proxies them back to the Runs Service to update the database.
  • The Actions Service maintains a watch on Kubernetes resources to provide real-time updates back to clients and ensure the database reflects the cluster state.

Flyte Domain Data Model​

The data model for Flyte revolves around the concepts of Runs, Actions, and Events, which are primarily managed within the runs.repository.models package and defined via gRPC/Protobuf in flyteidl2.

Core Entities​

  • Action: The central entity representing a unit of work. In the database, Runs are simply root actions (where ParentActionName is null). Actions can be of different types: Tasks, Traces, or Conditions. They store their state, specification, and execution details as serialized protobufs (ActionSpec, ActionDetails, DetailedInfo).
  • ActionEvent: Represents a phase transition or a significant event in the lifecycle of an action attempt. It provides an append-only history of what happened during execution, including error information and timestamps.
  • Project: A top-level container for organizing workflows, tasks, and runs.
  • Task: Represents the definition of a task, including its project, domain, name, and version. It serves as a template for Action executions.
  • Trigger: Defines how a task should be automatically initiated (e.g., on a schedule). Triggers are associated with specific tasks and maintain a revision history.
  • secret: Manages sensitive information required by tasks. Secrets are scoped to projects and domains and are typically backed by Kubernetes Secrets in the underlying infrastructure.

Relationships​

  • Action to Action: A self-referencing relationship where an action can have a ParentActionName, forming a hierarchy (Run -> Action -> Sub-action).
  • Action to ActionEvent: One action can have many events, each representing a state change or attempt.
  • Task to Action: An action (of type TASK) refers to a specific task definition.
  • Task to Trigger: A task can have multiple triggers defined for it.
  • Project to Run/Action: All runs and actions are scoped within a project and domain.
  • Secret to Project/Domain: Secrets are scoped to ensure they are only accessible to authorized workloads within the same project/domain.

Metadata and Specifications​

  • TaskActionSpec: Contains execution-time configuration like labels, annotations, environment variables, and resource overrides.
  • Run: Stores detailed runtime metadata such as storage URIs for inputs/outputs and the digest of the resolved task spec.
  • TaskExecutionMetadata: Provides high-level metadata for an action, including who executed it and the trigger that initiated it.

Key Architectural Findings:

  • Runs and Actions are stored in the same database table, distinguished by the presence of a parent action name.
  • ActionEvent serves as an append-only log of phase transitions for actions, enabling detailed execution history tracking.
  • The system uses a 'Project -> Domain -> Name -> Version' hierarchy for identifying tasks and actions.
  • Metadata like RunSpec and RunInfo are stored as serialized protobuf blobs within the Action record for efficiency and flexibility.
  • Secrets are managed as a separate service but are tightly integrated with the project/domain scoping of the rest of the system.

Flyte Production Infrastructure Deployment​

The deployment architecture of Flyte Production Infrastructure is centered around a unified gRPC/HTTP service called the Flyte Binary, which consolidates multiple backend services and background workers into a single deployment. This service interacts with a PostgreSQL database for metadata and an Object Storage backend (S3, GCS, or Azure) for large data artifacts.

Key components include:

  • Flyte Binary: A multi-functional container that serves as the API gateway (handling Runs, Tasks, Projects, etc.) and runs background processes like the Trigger Scheduler, Abort Reconciler, and the Executor. The Executor acts as a Kubernetes controller, managing the lifecycle of TaskAction custom resources and orchestrating worker pods.
  • Flyte Console: A separate web-based user interface that communicates with the Flyte Binary API.
  • Worker Pods: Dynamically created by the Executor to run user-defined tasks. These pods utilize Flyte Co-Pilot as both an init container (for downloading metadata) and a sidecar (for monitoring and uploading results).
  • Ingress Controller: Manages external access to both the console (HTTP) and the binary service (gRPC and HTTP).
  • External Services: Integrates with OIDC providers for authentication and optional external Flyte Connectors for specialized task execution.

The architecture follows a Data Movement with Copilot for task execution and a The Reconciler Architecture for resource management within the Kubernetes cluster.

Key Architectural Findings:

  • The 'flyte-binary' is a unified service that bundles API handlers (Runs, Tasks, Projects, DataProxy, Events, Cache, Actions, App, Secret, Auth) and background workers (Abort Reconciler, Trigger Scheduler, Executor, Garbage Collector).
  • The Executor is a Kubernetes controller that reconciles 'TaskAction' CRDs and manages worker pod lifecycles.
  • Flyte Co-Pilot is deployed as both an init container (downloader) and a sidecar (uploader) within worker pods to handle metadata and data transfer.
  • The system uses PostgreSQL for metadata storage and supports multiple object storage providers (S3, GCS, Azure) via the 'stow' library.
  • Traffic is routed via an Ingress controller, with separate paths for the web console and gRPC/HTTP API services.

Workflow Run Lifecycle State Machine​

The Workflow Run Lifecycle State Machine diagram illustrates the progression of a workflow run (represented as an 'Action' in the Flyte backend) through various execution phases.

The lifecycle begins in the Phase state when an action is first enqueued. It then moves through resource allocation (Phase) and environment setup (Phase) before entering the active Phase state.

For standard tasks, the run transitions from Phase to terminal states like Phase, Phase, Phase, or Phase. A key feature discovered in the code is the retry logic: if a run fails or times out but has remaining retry attempts, the TaskActionReconciler resets it back to the Phase state.

Condition actions introduce a Phase state where the execution waits for an external signal. This state can transition to Phase upon receiving a signal, or to Phase/Phase if the signal is not received or the run is cancelled.

The ActionRepo ensures state integrity by enforcing that phase transitions generally move forward, except during retries where it allows moving from a failed state back to queued.

Key Architectural Findings:

  • Workflow runs are primarily managed as 'Actions' using the ActionPhase enum defined in flyteidl2/common/phase.proto.
  • The TaskActionReconciler in executor/pkg/controller/taskaction_controller.go manages the core transitions based on plugin execution results.
  • Retry logic is implemented by transitioning from FAILED or TIMED_OUT back to QUEUED if the current attempt count is less than the maximum allowed.
  • Condition actions utilize a PAUSED state to await external signals, as documented in the ActionPhase proto comments.
  • The database layer (ActionRepo) implements a guard condition in SQL to prevent invalid backward phase transitions, except for specific retryable phases.