Review and audit of EventSource logging for self-diagnostics

[alanwest]

OpenTelemetry .NET uses EventSource for internal logging. Each component defines an EventSource including the API, SDK, and each of the exporters and instrumentation components.

The purpose of this issue is threefold:

1. Review the usage of these EventSource's for correctness in what gets logged and where it gets logged.

Example of incorrectness:BaseExporter.Shutdown logs a span processor related error when there is a failure. This error should not be about spans or processors:

opentelemetry-dotnet/src/OpenTelemetry/BaseExporter.cs

Lines 123 to 127 in 6350288

	catch (Exception ex)
	{
	OpenTelemetrySdkEventSource.Log.SpanProcessorException(nameof(this.Shutdown), ex);
	return false;
	}

2. Improve coverage of diagnostic logging by identifying gaps.

Example: Sometimes (but not always) we have TODOs throughout the code highlighting gaps where a diagnostic log may be useful, but has not been implemented yet. Here is an example where an attempt to update a metric has failed:

opentelemetry-dotnet/src/OpenTelemetry/Metrics/AggregatorStore.cs

Line 292 in db8f0e7

// TODO: Measurement dropped due to internal exception.

3. Aim to make diagnostic logging actionable

Based on this comment #2525 (comment), we should seek to provide actionable guidance when there are errors. For example, if an error is due to misconfiguration, then the log message should give an indication for how to resolve the error.

Related in-flight work

The following PR was held off until the post-1.0 release. It may be beneficial to land this work prior to the review of self-diagnostics:

#1529

References

OpenTelemetry general error handling guidelines/self-diagnostics
OpenTelemetry .NET self-diagnostics guide

[github-actions]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.