Skip to main content

Overview

Flyte is a scalable, cloud-native orchestrator designed to manage complex data and machine learning workflows. It provides a unified platform for defining, executing, and monitoring workflow runs, ensuring reproducibility and reliability through a Kubernetes-native execution model.

Why Flyte?​

Managing large-scale data pipelines and ML training jobs often leads to "infrastructure spaghetti"—manually managing Kubernetes pods, handling retries, tracking task dependencies, and ensuring data lineage. Flyte exists to abstract these complexities. It allows engineers to focus on task logic while the platform handles resource allocation, state persistence, and observability.

Core Concepts​

  • Run: A single execution of a workflow or task. In Flyte's architecture, a Run is technically a "root Action" that serves as the entry point for execution.
  • Action: The fundamental unit of work. Actions can be individual tasks (containerized code), traces, or conditional logic.
  • Task: A reusable definition of work, specifying the container image, command-line arguments, and resource requirements.
  • TaskAction CRD: A Kubernetes Custom Resource that represents the live state of an action. The Flyte Executor reconciles these CRDs to drive the actual work on the cluster.
  • Project & Domain: Hierarchical namespaces (e.g., my-project/development) used to isolate resources and manage permissions.

How It Works​

Flyte operates as a distributed system, often bundled into a single unified binary called the Flyte Manager for ease of deployment.

  1. Submission: A user submits a CreateRun request to the Runs Service.
  2. Persistence: The Runs Service saves the run metadata to a PostgreSQL (or SQLite) database.
  3. Enqueueing: The Actions Service enqueues the root action, which triggers the creation of a TaskAction Custom Resource in Kubernetes.
  4. Execution: The Executor (a Kubernetes controller) detects the new TaskAction and manages its lifecycle, transitioning it through phases like QUEUED, RUNNING, and SUCCEEDED.
  5. Monitoring: The Actions Service watches for CRD updates and notifies the Runs Service, which updates the database and streams real-time status back to the user.

Use Cases​

Creating a Containerized Run​

You can trigger a run by providing a task specification directly to the RunService.

import (
    "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/common"
    "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/core"
    "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/task"
    "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/workflow"
)

// Define a simple container task run
req := &workflow.CreateRunRequest{
    Id: &workflow.CreateRunRequest_RunId{
        RunId: &common.RunIdentifier{
            Org: "flyte", Project: "demo", Domain: "development", Name: "hello-world",
        },
    },
    Task: &workflow.CreateRunRequest_TaskSpec{
        TaskSpec: &task.TaskSpec{
            TaskTemplate: &core.TaskTemplate{
                Target: &core.TaskTemplate_Container{
                    Container: &core.Container{
                        Image: "alpine:latest",
                        Args:  []string{"echo", "Hello Flyte!"},
                    },
                },
            },
        },
    },
}
client.CreateRun(ctx, connect.NewRequest(req))

Listing Runs for a Project​

Filter and retrieve runs within a specific project and domain.

listReq := &workflow.ListRunsRequest{
    ScopeBy: &workflow.ListRunsRequest_ProjectId{
        ProjectId: &common.ProjectIdentifier{
            Organization: "flyte",
            Name:         "demo",
            Domain:       "development",
        },
    },
}
resp, _ := client.ListRuns(ctx, connect.NewRequest(listReq))
for _, run := range resp.Msg.Runs {
    fmt.Printf("Run: %s, Phase: %s\n", run.Action.Id.Run.Name, run.Action.Status.Phase)
}

Aborting an Active Run​

Stop execution immediately if a run is no longer needed.

abortReq := &workflow.AbortRunRequest{
    RunId: &common.RunIdentifier{
        Org: "flyte", Project: "demo", Domain: "development", Name: "stale-run",
    },
    Reason: connect.Ptr("Manual cancellation"),
}
client.AbortRun(ctx, connect.NewRequest(abortReq))

When to Use Flyte​

Use it when:​

  • You need to orchestrate complex, multi-step data pipelines.
  • You require strong reproducibility and versioning for ML models.
  • You want to leverage Kubernetes for scaling individual tasks independently.
  • You need built-in task caching to save time and compute costs.

Don't use it if:​

  • You only need simple cron jobs without dependencies.
  • You don't have access to a Kubernetes cluster.
  • Your tasks are extremely short-lived (sub-second) and the overhead of pod creation is prohibitive.

Stack Compatibility​

  • Primary Language: Go (Service implementation).
  • SDKs: Python, TypeScript, and Rust bindings available.
  • Infrastructure: Kubernetes (required for execution).
  • Storage: S3, GCS, or Azure Blob Storage (via DataProxy).
  • Database: PostgreSQL (production) or SQLite (local dev).
  • API: gRPC / HTTP/2 via Buf Connect.

Getting Started Pointers​

  • Explore the Run Service for managing workflow lifecycles.
  • Check the Executor documentation to understand how tasks are reconciled on Kubernetes.
  • Review the DataProxy for handling large-scale task inputs and outputs.

Limitations & Assumptions​

  • Kubernetes Dependency: Flyte assumes it is running in or has access to a Kubernetes cluster for task execution.
  • Flyte 2 Evolution: This version (Flyte 2) focuses on a unified binary architecture, which simplifies deployment but may differ in configuration from legacy Flyte 1 installations.
  • Storage: Requires an object store for offloading large task inputs/outputs.