From b88840e94f80dc431160b09bc228cba3e708085c Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Wed, 1 May 2024 13:59:34 -0600 Subject: [PATCH 1/9] Create rfc --- docs/rfcs/logging-before-config-resolution.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 docs/rfcs/logging-before-config-resolution.md diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md new file mode 100644 index 00000000000..e179473776d --- /dev/null +++ b/docs/rfcs/logging-before-config-resolution.md @@ -0,0 +1,62 @@ +# How To Log Before Config Resolution + +## Overview + +The OpenTelemetry Collector supports configuring a primary logger that the collector and its components use to write logs. +This logger cannot be created until the user's configuration has been completely resolved. +There is a need to write logs during the collector start-up, before the primary logger is instantiated, such as during +configuration resolution or config validation. This document describes + +- why providing logging capabilities during startup is important +- the current (as of v0.99.0) behavior of the Collector +- different solutions to the problem +- the accepted solution + +## Why Logging During Startup is Important + +When the collector is starting it tries to resolve user configuration as quickly as possible. +But the Collector's configuration resolution strategy is not trivial - it allows for complex interactions between +multiple, different config sources that must all resolve without error. During this process important information could +be shared with users such as: +- [Warnings about deprecated syntax](https://github.com/open-telemetry/opentelemetry-collector/issues/9162) +- [Warnings about undesired, but handled, situations](https://github.com/open-telemetry/opentelemetry-collector/issues/5615) +- Debug information + +## Requirements for any solution + +1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. + +## Current behavior + +As of v0.99.0, the collector does not provide a way to log before the primary logger is instantiated. + +## Solutions + +### Buffer Logs in Memory and Write Them Once the Primary Logger Exists + +The Collector could provide a temporary logger that, when written to, keeps the logs in memory. These logs could then +be passed to the primary logger to be written out in the properly configured format/level. + +Benefits: +- Logs are written in the user-specified format/level + +Downsides: +- If an error occurs before the primary logger is created the buffered logs will not be written out +- If the primary logger is used to write any logs before the buffered logs are passed, logs will be out of order + +### Create a Logger Using the Primary Logger's Defaults + +When the user provides no primary logger configuration the Collector creates a Logger using a set of default values. +The Collector could, very early in startup, create a logger using these exact defaults and use it until the primary +logger is instantiated. + +Benefits: +- Logs order is preserved +- Logs are still writen when an error occurs before the primary logger can be instantiated + +Downsides: +- Logs may be written in a format/level that differs from the format/level of the primary logger + +## Accepted Solution + +I'll fill this out once we decide. From a5065bc1a2a57e5dec6d2dd1002d3783b76bba84 Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Wed, 1 May 2024 14:21:51 -0600 Subject: [PATCH 2/9] typo --- docs/rfcs/logging-before-config-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index e179473776d..0bc887aa938 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -52,7 +52,7 @@ logger is instantiated. Benefits: - Logs order is preserved -- Logs are still writen when an error occurs before the primary logger can be instantiated +- Logs are still written when an error occurs before the primary logger can be instantiated Downsides: - Logs may be written in a format/level that differs from the format/level of the primary logger From bd857fdffa51fedf7a33e127d97d3b5aaede6d6f Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Wed, 1 May 2024 14:51:44 -0600 Subject: [PATCH 3/9] Apply feedback --- docs/rfcs/logging-before-config-resolution.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index 0bc887aa938..71bd1b07851 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -25,6 +25,8 @@ be shared with users such as: ## Requirements for any solution 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. +2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to stdout/sterr +3. If an error occurs before the primary logger is created any previously written logs MUST be written to either stout or stderr if they have not already been written there. ## Current behavior From 3a15c9eaa7234cfbf207446f1721dfd86456902b Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Thu, 2 May 2024 10:51:20 -0600 Subject: [PATCH 4/9] Update docs/rfcs/logging-before-config-resolution.md Co-authored-by: Pablo Baeyens --- docs/rfcs/logging-before-config-resolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index 71bd1b07851..522e10bd5ce 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -25,8 +25,8 @@ be shared with users such as: ## Requirements for any solution 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. -2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to stdout/sterr -3. If an error occurs before the primary logger is created any previously written logs MUST be written to either stout or stderr if they have not already been written there. +2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s) +3. If an error occurs before the primary logger is created any previously written logs MUST be written to the user-specified location(s) if they have not already been written there. ## Current behavior From 2cbaf10732ab9959729f075d01517494af30c5e2 Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Thu, 2 May 2024 13:12:20 -0600 Subject: [PATCH 5/9] Add back stderr/stdout requirement --- docs/rfcs/logging-before-config-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index 522e10bd5ce..3634d44e8ad 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -26,7 +26,7 @@ be shared with users such as: 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. 2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s) -3. If an error occurs before the primary logger is created any previously written logs MUST be written to the user-specified location(s) if they have not already been written there. +3. If an error occurs before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. ## Current behavior From 1f6d859723ad14183df653e87b52a41799b32780 Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Thu, 2 May 2024 13:14:28 -0600 Subject: [PATCH 6/9] clarify requirement 3 --- docs/rfcs/logging-before-config-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index 3634d44e8ad..c7b12e83252 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -26,7 +26,7 @@ be shared with users such as: 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. 2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s) -3. If an error occurs before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. +3. If an error causes the collector to gracefully terminate before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. ## Current behavior From 5b176a6fab3cdf6ac5c1996661410ae8eacaaa56 Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Fri, 3 May 2024 11:42:08 -0600 Subject: [PATCH 7/9] Apply feedback --- docs/rfcs/logging-before-config-resolution.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index c7b12e83252..bb5fb3c12b0 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -26,7 +26,8 @@ be shared with users such as: 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. 2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s) -3. If an error causes the collector to gracefully terminate before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. +3. The EntryCaller (what line number the log originates from) must be accurate to where the log was written. +4. If an error causes the collector to gracefully terminate before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. ## Current behavior @@ -61,4 +62,6 @@ Downsides: ## Accepted Solution -I'll fill this out once we decide. +[Buffer Logs in Memory and Write Them Once the Primary Logger Exists](#Buffer-Logs-in-Memory-and-Write-Them-Once-the-Primary-Logger-Exists) + +This solution, while more complex, allows the collector to write out the logs in the user-specified format whenever possible. A fallback logger must be used in situations where the primary logger could not be created and the collector is shutting down, such as when encountering an error during configuration resolution, but otherwise the primary logger will be used to write logs that occurred before the primary logger existed. From b04f8e825dca339ed157fbbd31005a38b4318e8c Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Fri, 3 May 2024 11:45:38 -0600 Subject: [PATCH 8/9] Make anchor lowercase --- docs/rfcs/logging-before-config-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index bb5fb3c12b0..fe13f8a20bf 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -62,6 +62,6 @@ Downsides: ## Accepted Solution -[Buffer Logs in Memory and Write Them Once the Primary Logger Exists](#Buffer-Logs-in-Memory-and-Write-Them-Once-the-Primary-Logger-Exists) +[Buffer Logs in Memory and Write Them Once the Primary Logger Exists](#buffer-logs-in-memory-and-write-them-once-the-primary-logger-exists) This solution, while more complex, allows the collector to write out the logs in the user-specified format whenever possible. A fallback logger must be used in situations where the primary logger could not be created and the collector is shutting down, such as when encountering an error during configuration resolution, but otherwise the primary logger will be used to write logs that occurred before the primary logger existed. From 59d81738a37b7cce4a1dab7c753efdfaa51c2921 Mon Sep 17 00:00:00 2001 From: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com> Date: Mon, 6 May 2024 08:40:46 -0600 Subject: [PATCH 9/9] Apply suggestions from code review Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com> --- docs/rfcs/logging-before-config-resolution.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/rfcs/logging-before-config-resolution.md b/docs/rfcs/logging-before-config-resolution.md index fe13f8a20bf..0c947e83637 100644 --- a/docs/rfcs/logging-before-config-resolution.md +++ b/docs/rfcs/logging-before-config-resolution.md @@ -25,7 +25,7 @@ be shared with users such as: ## Requirements for any solution 1. Once the primary logger is instantiated, it should be usable anywhere in the Collector that does logging. -2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s) +2. Log timestamps must be accurate to when the log was written, regardless of when the log is written to the user-specified location(s). 3. The EntryCaller (what line number the log originates from) must be accurate to where the log was written. 4. If an error causes the collector to gracefully terminate before the primary logger is created any previously written logs MUST be written to the either stout or stderr if they have not already been written there. @@ -44,8 +44,7 @@ Benefits: - Logs are written in the user-specified format/level Downsides: -- If an error occurs before the primary logger is created the buffered logs will not be written out -- If the primary logger is used to write any logs before the buffered logs are passed, logs will be out of order +- If the primary logger is used to write any logs before the buffered logs are passed, logs may be out of order. There are no guarantees that logs will be written in order, so the log timestamps should be taken as the source of truth for ordering. ### Create a Logger Using the Primary Logger's Defaults