Skip to main content

Flyte

Reliably orchestrate ML pipelines, models, and agents at scale — in pure Python.

Overview​

Flyte is an open-source, Kubernetes-native platform designed to automate and scale complex machine learning and data workflows. It allows developers to define workflows in pure Python, providing a seamless transition from local development to production-grade distributed execution. By abstracting away the underlying infrastructure, Flyte enables data scientists and engineers to focus on their core logic while ensuring reproducibility, reliability, and observability.

At its core, Flyte manages the lifecycle of executions, coordinating the movement of data and the scheduling of tasks across a distributed cluster. It features a powerful execution engine that handles resource allocation, state transitions, and retries. With a robust plugin system, Flyte integrates with a wide range of cloud-native tools and services, making it a versatile choice for modern AI and data platforms.

Key Concepts​

  • Workflow & Run Management Workflow & Run Management: Orchestrates the execution of complex directed acyclic graphs (DAGs) and tracks the state and progress of individual runs.
  • Task Plugin System Task Plugin System: Enables integration with diverse compute backends and external services like Spark, Ray, and AWS through a modular plugin architecture.
  • Data & Storage Abstraction Data & Storage Abstraction: Provides a consistent interface for data I/O, supporting multiple cloud storage providers and ensuring efficient data movement.
  • Caching & Memoization Caching & Memoization: Optimizes performance by reusing outputs from previous task executions, significantly reducing computation time and costs.
  • Security & Secret Management Security & Secret Management: Ensures secure execution by managing sensitive credentials and providing them to tasks via integrated secret providers.
  • Scheduling & Triggers Scheduling & Triggers: Automates workflow execution with support for cron-based schedules and event-driven triggers for continuous processing.

Common Use Cases​

  • ML Training Pipelines: Orchestrate multi-step machine learning workflows, from data preprocessing and feature engineering to model training and evaluation.
  • Data Engineering: Build scalable ETL pipelines that process large datasets across distributed computing frameworks like Spark or Dask.
  • Model Serving: Deploy and manage machine learning models as services with integrated monitoring and log streaming.
  • Distributed Computing: Leverage specialized compute resources for heavy-duty data processing tasks within a unified, version-controlled workflow.
  • AI Agents: Develop and orchestrate agents that interact with multiple APIs and services to perform complex, multi-stage tasks.

Getting Started​

To begin your journey with Flyte, check out the Getting Started guide to set up your local environment. For a deeper understanding of the platform's design, explore the Architecture Overview or learn about Task and Version Management to begin building and deploying your first workflows.