Skip to main content

Data Proxy and Translation Services

Flyte uses the Data Proxy and Translator services to offload data-heavy operations from the central control plane. By handling large literal translations and providing direct access to object storage via signed URLs, these services prevent large data payloads from causing congestion in the core workflow engine.

Translating Literals for the UI​

The TranslatorService (defined in dataproxy/translator.go) is responsible for converting between Flyte's internal protobuf-based Literal format and the JSON format used by the Flyte UI (Launch Forms).

Converting Inline Literals​

When you need to display a launch form for a task or workflow, the UI calls LiteralsToLaunchFormJson. This method takes a map of literals and their corresponding variable definitions to produce a JSON schema.

// Example from dataproxy/translator_test.go
svc := NewTranslatorService(nil, nil)

resp, err := svc.LiteralsToLaunchFormJson(context.Background(), connect.NewRequest(&workflow.LiteralsToLaunchFormJsonRequest{
    Literals:  testNamedLiterals(), // Map of Flyte Literals
    Variables: testVariableMap(),   // Variable definitions (types)
}))

Handling Offloaded Literals​

For large inputs or outputs that are already stored in object storage (S3/GCS), the service can read them directly using a LiteralsUri. To ensure security, this requires an ActionId.

Internally, readOffloadedLiterals verifies the request by calling the RunService:

  1. It fetches authorized data URIs for the specific ActionId via s.runClient.GetActionDataURIs.
  2. It validates that the requested LiteralsUri matches either the InputsUri or OutputsUri reported by the RunService.
  3. It reads the protobuf data directly from storage using the dataStore.

This mechanism ensures that the Data Proxy only translates data that the user is authorized to access for a specific execution.

Direct Storage Access via Signed URLs​

The Service in dataproxy/service/dataproxy_service.go provides a way for clients (like Flytekit or the UI) to interact directly with object storage without streaming data through the Flyte backend.

Requesting Upload Locations​

To upload data, you call CreateUploadLocation. The service returns a signed URL that allows a direct PUT request to the storage provider.

req := &connect.Request[dataproxy.CreateUploadLocationRequest]{
    Msg: &dataproxy.CreateUploadLocationRequest{
        Project:      "my-project",
        Domain:       "development",
        Filename:     "data.parquet",
        FilenameRoot: "unique-execution-id",
        ContentMd5:   []byte("..."), // Optional but recommended
        ExpiresIn:    durationpb.New(30 * time.Minute),
    },
}
resp, err := service.CreateUploadLocation(ctx, req)
// resp.Msg.SignedUrl is the URL for the client to use

The service constructs the storage path using the pattern: {storage_prefix}/{project}/{domain}/{filename_root}/{filename}

If FilenameRoot is not provided, it uses a URL-safe base32 encoding of the ContentMd5 hash as the directory name.

Safety and Overwrite Protection​

The Data Proxy performs a best-effort check to prevent accidental overwrites in checkFileExists. If a file already exists at the target location:

  • If no ContentMd5 was provided in the request, it rejects the upload with CodeAlreadyExists.
  • If a ContentMd5 was provided, it compares it against the existing file's metadata. It only allows the upload if the hashes match (effectively making the operation idempotent).

Input Management and Caching​

The UploadInputs method is a specialized utility for persisting task inputs to storage in a way that supports Flyte's caching mechanism.

Deterministic Hashing​

When you call UploadInputs, the service:

  1. Resolves the task template to identify variables marked in cache_ignore_input_vars.
  2. Filters out these ignored variables using filterInputs.
  3. Computes a deterministic FNV-64a hash of the remaining inputs using hashInputsProto.

This hash is used as part of the storage path: {storage_prefix}/{org}/{project}/{domain}/offloaded-inputs/{hash}/inputs.pb

By using a deterministic hash of the non-ignored inputs, Flyte ensures that identical input sets result in the same storage location, facilitating cache hits across different executions.

Configuration​

The behavior of these services is controlled via the DataProxyConfig (in dataproxy/config/config.go). Key configuration options include:

  • Upload Limits: MaxSize (default 100Mi) restricts the size of data that can be uploaded.
  • Expiration: MaxExpiresIn (default 1h) sets the upper bound for how long signed URLs remain valid.
  • Storage Prefix: StoragePrefix (default "uploads") defines the root directory in the object store where the Data Proxy manages files.

These settings are registered under the dataproxy configuration section and can be adjusted to match the storage policies of your Flyte deployment.