[telemetrySettting] Create sampled Logger

[antonjim-te]

Issue:

In one of the use cases which we are working on, we have identified that sampledLogger mechanism used in the Opentelemetry exporters (otlpexporter and otlphttpexporter), could become a memory bottleneck, when the number of exports in the same pipeline is high (we are talking about 100s of instances of exporters running on the same machine).

The sampledLogger is created inside the exporterhelper module and the sampling is always activated unless the collector is in debug mode:

• https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/queued_retry.go#L88

func newQueuedRetrySender(id component.ID, signal component.DataType, qCfg QueueSettings, rCfg RetrySettings, reqUnmarshaler internal.RequestUnmarshaler, nextSender requestSender, logger *zap.Logger) *queuedRetrySender {
	retryStopCh := make(chan struct{})
	sampledLogger := createSampledLogger(logger)

• https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/queued_retry.go#L269

func createSampledLogger(logger *zap.Logger) *zap.Logger {
	if logger.Core().Enabled(zapcore.DebugLevel) {
		// Debugging is enabled. Don't do any sampling.
		return logger
	}

	// Create a logger that samples all messages to 1 per 10 seconds initially,
	// and 1/100 of messages after that.
	opts := zap.WrapCore(func(core zapcore.Core) zapcore.Core {
		return zapcore.NewSamplerWithOptions(
			core,
			10*time.Second,
			1,
			100,
		)
	})
	return logger.WithOptions(opts)
}

Investigation:

The high memory usage has been discovered using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/pprofextension

• In the first scenario, default configuration with the sampledLogger active, based on the profiled data we could observe that sampling mechanism was responsible for allocating aprox. 75% of the memory of the running collector:
  

• In the second scenario, with the same number of running exporters, but with the sampledLogger mechanism disabled (using debug level configuration check configuration below), we were able to observe a much smaller memory footprint:

    service:
      telemetry:
        logs:
          level: debug

Given that the logging sampling mechanism could become a bottleneck in certain scenarios (our case), we would like to propose a contribution, which potentially can improve the memory footprint of the exporters in certain use cases (potentially making it more scalable)

Proposal:

Create an extra sampled logger in the telemetry configuration used to avoid flooding the logs with messages that are repeated frequently

The sampled logger is configured by LogsSamplingConfig

service:
    telemetry:
        logs:
            sampling:

Default configuration for the sampled logger

service:
    telemetry:
        logs:
            sampling:
	        initial: 1
	        thereafter: 100
	        tick: 10s

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: antonjim-te / name: Antonio Jimenez (4394496, 3192d88, 39f95e4, 5295e11, 9534fbb, 56b71f3, 5aec726, 5d2cdeb, 7ab0227, 9778c88, b6ac5ac, 6c2051a, f53561a, 310b747, 9540c55, 20536a1, 0b60ef8, 0add644, 37fcef7, 70f05d8, a5d054b, 868041f, 8a321ac, 0cb309c, bc250fb, 7180926, 15b3b42, 409f6ff, 66fa58e, 4b84a00, fcd2b83, ce3776a, bce885d, f9ca8a7, 8c3cddd)

[antonjim-te]

/review @pedro-te @sboychen-te @vutnguye-te @sfreitas-te

[antonjim-te]

Hi @bogdandrutu, could you take a look at the PR please?
Thank you.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[dmitryax]

Thank you, @antonjim-te. It looks good to me. Just minor comments

[dmitryax]

I'd like to have another look from any of @open-telemetry/collector-approvers before merging