Garbage Collection of Terminal Actions

To prevent terminal TaskAction resources from accumulating in your Kubernetes cluster and impacting performance, you can configure the Flyte GarbageCollector to automatically delete them after they reach a specific age.

Configuring Garbage Collection

The garbage collector is configured via the GCConfig struct in executor/pkg/config/config.go. You can control how often the collector runs and how long terminal resources should persist before being deleted.

// executor/pkg/config/config.go

type GCConfig struct {
	// Interval is how often the garbage collector runs. 0 disables GC.
	Interval stdconfig.Duration `json:"interval" pflag:",How often the garbage collector runs. 0 disables GC."`

	// MaxTTL is the time-to-live for terminal TaskActions before deletion.
	MaxTTL stdconfig.Duration `json:"maxTTL" pflag:",Time-to-live for terminal TaskActions before deletion."`
}

To enable garbage collection, set the interval to a non-zero duration. For example, to run the collector every 30 minutes and delete resources older than 1 hour:

executor:
  gc:
    interval: 30m
    maxTTL: 1h

How Garbage Collection Works

The GarbageCollector (defined in executor/pkg/controller/garbage_collector.go) runs as a background worker within the Flyte executor. It implements the controller-runtime manager.Runnable interface and is added to the manager during the executor's setup phase in executor/setup.go.

Resource Identification

The collector identifies candidates for deletion using Kubernetes labels. The TaskActionReconciler in executor/pkg/controller/taskaction_controller.go is responsible for applying these labels when a TaskAction reaches a terminal state:

• flyte.org/termination-status: Set to terminated.
• flyte.org/completed-time: Set to the UTC completion time in 2006-01-02.15-04 format.

The Collection Cycle

During each interval, the GarbageCollector.collect method performs the following steps:

1. Pagination: It lists TaskAction resources in batches (defined by gcPageSize = 500) to avoid overloading the Kubernetes API server.
2. Filtering: It uses a label selector to find resources that have both the termination status and a completed time.
3. TTL Comparison: It calculates a cutoff time based on the current time minus the MaxTTL. Because the completed-time label uses a minute-precision format, the collector can perform a lexicographical string comparison to identify expired resources.

// executor/pkg/controller/garbage_collector.go

func (gc *GarbageCollector) collect(ctx context.Context) error {
	// ...
	cutoff := time.Now().UTC().Add(-gc.maxTTL).Format(labelTimeFormat)
    
	for {
		var taskActions flyteorgv1.TaskActionList
		listOpts := []client.ListOption{
			client.MatchingLabels{LabelTerminationStatus: LabelValueTerminated},
			client.HasLabels{LabelCompletedTime},
			client.Limit(gcPageSize),
		}
		// ...
		if err := gc.client.List(ctx, &taskActions, listOpts...); err != nil {
			return err
		}

		for i := range taskActions.Items {
			ta := &taskActions.Items[i]
			completedTime := ta.GetLabels()[LabelCompletedTime]
            
			// String comparison works due to lexicographical time format
			if completedTime < cutoff {
				if err := gc.client.Delete(ctx, ta); err != nil {
					// log error and continue
				}
			}
		}
        // ... handle pagination tokens
	}
}

Disabling Garbage Collection

If you need to preserve terminal TaskAction resources indefinitely for auditing or debugging, you can disable the garbage collector by setting the interval to 0.

executor:
  gc:
    interval: 0s

When the interval is 0, the executor setup logic in executor/setup.go skips the initialization of the GarbageCollector entirely:

// executor/setup.go

if cfg.GC.Interval.Duration > 0 {
    if cfg.GC.MaxTTL.Duration <= 0 {
        return fmt.Errorf("executor: gc.maxTTL must be positive when gc is enabled, got %v", cfg.GC.MaxTTL.Duration)
    }
    gc := controller.NewGarbageCollector(mgr.GetClient(), cfg.GC.Interval.Duration, cfg.GC.MaxTTL.Duration)
    if err := mgr.Add(gc); err != nil {
        return fmt.Errorf("executor: failed to add garbage collector: %w", err)
    }
}

Troubleshooting

Executor Fails to Start

If you enable garbage collection by setting an interval but provide a non-positive maxTTL (e.g., 0s or -1h), the Flyte executor will fail to start with the following error: executor: gc.maxTTL must be positive when gc is enabled, got 0s

Resources Not Being Deleted

If terminal TaskAction resources are not being cleaned up despite GC being enabled:

1. Check Labels: Verify that the TaskAction has the flyte.org/termination-status: terminated and flyte.org/completed-time labels. If these are missing, the TaskActionReconciler may be failing before it can label the resource.
2. Check Logs: Look for logs from the gc component. The collector logs "garbage collection cycle failed" if it encounters API errors and "garbage collection completed" with the count of deleted items when successful.
3. Verify Timezone: The collector uses time.Now().UTC(). Ensure your cluster nodes are not experiencing significant clock drift, as this could affect the TTL calculation.