From ab92dac8e840cbe887447e4de679ad699b2491df Mon Sep 17 00:00:00 2001 From: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com> Date: Tue, 4 Oct 2022 14:18:37 +0530 Subject: [PATCH] update configuring_links (#905) * update configuring_links Fixes https://github.com/flyteorg/flyte/issues/2820 Signed-off-by: SmritiSatyanV smriti@union.ai * update configuring links Changes based on review Signed-off-by: SmritiSatyanV smriti@union.ai --- .../deployment/configure_logging_links.py | 30 +++++++++++++++---- 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/flytesncaks/cookbook/deployment/configure_logging_links.py b/flytesncaks/cookbook/deployment/configure_logging_links.py index c68b19579..8b857d964 100644 --- a/flytesncaks/cookbook/deployment/configure_logging_links.py +++ b/flytesncaks/cookbook/deployment/configure_logging_links.py @@ -4,19 +4,19 @@ Configuring Logging Links in UI ------------------------------- -Oftentimes to debug your workflows in production, you want to access logs from your tasks as they run. +To debug your workflows in production, you want to access logs from your tasks as they run. These logs are different from the core Flyte platform logs, are specific to execution, and may vary from plugin to plugin; for example, Spark may have driver and executor logs. Every organization potentially uses different log aggregators, making it hard to create a one-size-fits-all solution. Some examples of the log aggregators include cloud-hosted solutions like AWS CloudWatch, GCP Stackdriver, Splunk, Datadog, etc. -Flyte does not have an opinion here and provides a simplified interface to configure your log provider. Flyte-sandbox -ships with the Kubernetes dashboard to visualize the logs. This may not be safe for production; hence we recommend users +Flyte provides a simplified interface to configure your log provider. Flyte-sandbox +ships with the Kubernetes dashboard to visualize the logs. This may not be safe for production, hence we recommend users explore other log aggregators. -How Do I configure? -^^^^^^^^^^^^^^^^^^^^ +How to configure? +^^^^^^^^^^^^^^^^^ To configure your log provider, the provider needs to support `URL` links that are shareable and can be templatized. The templating engine has access to `these `_ parameters. @@ -60,6 +60,24 @@ - "https://some-other-source/home?region=us-east-1#logEventViewer:group=/flyte-production/kubernetes;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log" messageFormat: "json" # "unknown" | "csv" | "json" +.. tip:: + + Since helm chart uses the same templating syntax for args (like ``{{ }}``), compiling the chart results in helm replacing Flyte log link templates as well. To avoid this, you can use escaped templating for Flyte logs in the helm chart. + This ensures that Flyte log link templates remain in place during helm chart compilation. + For example: + + If your configuration looks like this: + + ``https://someexample.com/app/podName={{ "{{" }} .podName {{ "}}" }}&containerName={{ .containerName }}`` + + Helm chart will generate: + + ``https://someexample.com/app/podName={{.podName}}&containerName={{.containerName}}`` + + Flytepropeller pod would be created as: + + ``https://someexample.com/app/podName=pname&containerName=cname`` + This code snippet will output two logs per task that use the log plugin. -However, not all task types use the log plugin; for example, the Sagemaker plugin uses the log output provided by Sagemaker, and the Snowflake plugin will use a link to the snowflake console. +However, not all task types use the log plugin; for example, the SageMaker plugin uses the log output provided by Sagemaker, and the Snowflake plugin will use a link to the snowflake console. """