Skip to main content

Customizing Task Log Links

Flyte allows you to customize the log links displayed in the console by implementing log plugins. These plugins use task execution metadata to generate URIs for external logging systems like CloudWatch, Datadog, or Azure Monitor.

Using TemplateLogPlugin for URI Templating​

The TemplateLogPlugin is the standard way to generate log links using regex-based templates. You define a URI template containing placeholders that Flyte replaces with actual execution data.

Basic Configuration​

To configure a log link for a service like Datadog, you define a TemplateLogPlugin with the appropriate URI template.

import (
    "github.com/flyteorg/flyte/v2/flyteplugins/go/tasks/pluginmachinery/tasklog"
    "github.com/flyteorg/flyte/v2/gen/go/flyteidl2/core"
)

// Example configuration for Datadog logs
datadogPlugin := tasklog.TemplateLogPlugin{
    DisplayName:   "Datadog Logs",
    TemplateURIs:  []tasklog.TemplateURI{"https://app.datadoghq.com/logs?event&from_ts={{ .podUnixStartTime }}&live=true&query=pod_name%3A{{ .podName }}&to_ts={{ .podUnixFinishTime }}"},
    MessageFormat: core.TaskLog_JSON,
}

Available Template Variables​

The Input struct provides the metadata used to populate templates. The following variables are available for use in your TemplateURIs:

VariableDescription
{{ .podName }}The name of the Kubernetes pod.
{{ .podUID }}The unique identifier of the pod.
{{ .namespace }}The Kubernetes namespace where the task is running.
{{ .containerName }}The name of the container within the pod.
{{ .containerID }}The ID of the container (prefixes like docker:// are automatically stripped).
{{ .logName }}The name assigned to the log stream.
{{ .podUnixStartTime }}Pod start time in Unix timestamp format.
{{ .podUnixFinishTime }}Pod finish time in Unix timestamp format.
{{ .taskRetryAttempt }}The current retry attempt number for the task.
{{ .generatedName }}A DNS-1123 compatible unique name for the task execution.
{{ .executionName }}The name of the workflow execution.

Dynamic log links allow you to generate URIs based on configuration provided within the task itself. This is useful for tools like VSCode or interactive debuggers where the connection details (like a port) might change per task.

To use dynamic links, the task's TaskTemplate must include a link_type in its config map that matches the plugin's Name.

// Plugin definition for VSCode interactive logs
vscodePlugin := tasklog.TemplateLogPlugin{
    Name:                "vscode",
    DisplayName:         "VSCode",
    DynamicTemplateURIs: []tasklog.TemplateURI{"vscode://flyteinteractive:{{ .taskConfig.port }}/{{ .podName }}"},
    MessageFormat:       core.TaskLog_JSON,
}

In this example, {{ .taskConfig.port }} is resolved by looking up the key port in the TaskTemplate.Config map.

Specialized Log Plugins​

For logging systems that require complex encoding, Flyte provides specialized implementations.

Azure Monitor Logs​

The AzureLogsTemplatePlugin handles the specific requirements of Azure Monitor, where the Kusto query must be gzipped, Base64 encoded, and then URL encoded before being appended to the URI.

azurePlugin := tasklog.AzureLogsTemplatePlugin{
    TemplateLogPlugin: tasklog.TemplateLogPlugin{
        DisplayName:  "Azure Logs",
        TemplateURIs: []tasklog.TemplateURI{"https://portal.azure.com/#blade/Microsoft_Azure_Monitoring_Logs/LogsBlade/resourceId/YOUR_RESOURCE_ID/source/LogsBlade.AnalyticsShareLinkToQuery/q/"},
    },
    // Optional: Override the default Kusto query
    QueryFormat: &[]string{`ContainerLogV2 | where PodName == "{{.podName}}" | project TimeGenerated, LogContent`}[0],
}

Creating a Custom Log Plugin​

If the standard template plugins do not meet your needs, you can implement the Plugin interface. You must define a GetTaskLogs method that accepts an Input struct and returns an Output containing a slice of core.TaskLog objects.

type MyCustomLogPlugin struct {}

func (p MyCustomLogPlugin) GetTaskLogs(i tasklog.Input) (tasklog.Output, error) {
    // Construct a custom URI using Input metadata
    customURI := fmt.Sprintf("https://my-logs.com/search?pod=%s&ns=%s", i.PodName, i.Namespace)
    
    return tasklog.Output{
        TaskLogs: []*core.TaskLog{
            {
                Uri:           customURI,
                Name:          "My Custom Logs",
                MessageFormat: core.TaskLog_TEXT,
            },
        },
    }, nil
}

Implementation Details​

  • ContainerID Stripping: When using {{ .containerID }}, Flyte automatically strips the container runtime prefix (e.g., cri-o:// or docker://) to match how most logging agents (like Fluentd) index logs.
  • DNS Sanitization: The {{ .generatedName }} variable is passed through utils.ConvertToDNS1123SubdomainCompatibleString to ensure it is safe for use in URLs and Kubernetes identifiers.
  • Regex Matching: Template variable replacement is case-insensitive (e.g., {{ .podName }} and {{ .PodName }} are treated the same).