Tracking Action Progress and Errors

Flyte tracks the lifecycle of task executions using a dual-layered approach that combines Kubernetes-native conditions with a detailed, timestamped phase history. This ensures that both automated controllers and human operators have a clear view of an action's progress, while also providing structured error data that can be propagated back to the Flyte SDK.

Phase Transitions and History

The PhaseTransition struct in executor/api/v1/taskaction_types.go is the primary mechanism for recording the timeline of a task. Unlike a simple status field that only shows the current state, Flyte maintains a PhaseHistory slice within the TaskAction status.

// PhaseTransition records a phase change with its timestamp.
type PhaseTransition struct {
	// Phase is the phase that was entered (e.g. "Queued", "Initializing", "Executing", "Succeeded", "Failed").
	Phase string `json:"phase"`

	// OccurredAt is when this phase transition happened.
	OccurredAt metav1.Time `json:"occurredAt"`

	// Message is an optional human-readable message about the transition.
	Message string `json:"message,omitempty"`
}

The TaskActionReconciler in executor/pkg/controller/taskaction_controller.go manages this history. To prevent the status from growing indefinitely with redundant entries during frequent reconciliation loops, Flyte implements a deduplication check. A new PhaseTransition is only appended if the phase name differs from the most recent entry in the history:

// From executor/pkg/controller/taskaction_controller.go
if phaseName != "" {
    n := len(ta.Status.PhaseHistory)
    if n == 0 || ta.Status.PhaseHistory[n-1].Phase != phaseName {
        ta.Status.PhaseHistory = append(ta.Status.PhaseHistory, flyteorgv1.PhaseTransition{
            Phase:      phaseName,
            OccurredAt: metav1.Now(),
            Message:    msg,
        })
    }
}

Structured Error Propagation

When a task fails, Flyte needs to communicate more than just a boolean "failed" state. The ErrorState struct captures structured metadata from the underlying plugin (such as the K8s pod provider or a cloud service) to provide context to the user.

// ErrorState captures the structured error returned by the plugin.
type ErrorState struct {
	// Code is the plugin-defined error code (e.g. "OOMKilled").
	Code string `json:"code,omitempty"`

	// Kind is the error kind: "USER", "SYSTEM", or "" (unspecified).
	Kind string `json:"kind,omitempty"`

	// Message is the human-readable error message.
	Message string `json:"message,omitempty"`
}

This structure allows Flyte to distinguish between different failure modes. For example, an OOMKilled error code indicates a resource issue, while a SYSTEM kind might trigger different retry logic than a USER kind error.

The ActionsService in actions/service/actions_service.go converts this internal ErrorState back into a Flyte IDL ExecutionError. This ensures that the SDK receives a consistent error object regardless of which plugin executed the task:

func errorStateToExecutionError(es *executorv1.ErrorState) *core.ExecutionError {
	kind := core.ExecutionError_UNKNOWN
	switch es.Kind {
	case "USER":
		kind = core.ExecutionError_USER
	case "SYSTEM":
		kind = core.ExecutionError_SYSTEM
	}
	return &core.ExecutionError{
		Code:    es.Code,
		Kind:    kind,
		Message: es.Message,
	}
}

Kubernetes Native Conditions

Flyte follows Kubernetes conventions by using TaskActionConditionType and TaskActionConditionReason to report high-level status. These conditions allow external tools (like kubectl) to easily query the state of a TaskAction.

The system defines several standard types and reasons:

• Condition Types: Progressing, Succeeded, Failed.
• Condition Reasons: Queued, Initializing, Executing, Completed, RetryableFailure, PermanentFailure.

The mapPhaseToConditions function in the controller acts as the bridge between the plugin's internal phase and these public-facing conditions. For instance, when a plugin reports PhaseRunning, the controller updates the Progressing condition to True with the reason Executing:

case pluginsCore.PhaseRunning:
    phaseName = string(flyteorgv1.ConditionReasonExecuting)
    msg = info.Reason()
    setCondition(ta, flyteorgv1.ConditionTypeProgressing, metav1.ConditionTrue,
        flyteorgv1.ConditionReasonExecuting, msg)

System Failure Tracking

Flyte distinguishes between user-level task failures and system-level infrastructure issues. Infrastructure issues (like transient Kubernetes API errors) are tracked via a system failure count. If the number of system failures exceeds a configured threshold (defaulting to MaxSystemFailures = 3), Flyte marks the action as a permanent failure with the specific error code MaxSystemFailuresExceeded. This prevents a single flaky infrastructure component from causing infinite retries while still providing resilience against transient blips.