Implement observer instrument

[mauriciovasquezbernal]

This PR implements the Observer instrument: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-metrics.md#observer.

An Observer is created by calling the new register_observer method on Meter. This method receives a callback that is invoked on collection time with the Observer instrument as parameter, this provides the observe function for capturing data. It is not possible to unregister an Observer yet.

Observers are aggregated using a MinMaxSumCountLast aggregator based on #422.

Fixes #403
Fixes #178

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@df01d09). Click here to learn what that means.
The diff coverage is 87.95%.

@@            Coverage Diff            @@
##             master     #425   +/-   ##
=========================================
  Coverage          ?   88.63%           
=========================================
  Files             ?       42           
  Lines             ?     2165           
  Branches          ?      245           
=========================================
  Hits              ?     1919           
  Misses            ?      171           
  Partials          ?       75

Impacted Files	Coverage Δ
opentelemetry-api/src/opentelemetry/util/loader.py	81.25% <ø> (ø)
...lemetry-sdk/src/opentelemetry/sdk/util/__init__.py	85.88% <ø> (ø)
...entelemetry-api/src/opentelemetry/util/__init__.py	83.33% <ø> (ø)
...try-ext-wsgi/src/opentelemetry/ext/wsgi/version.py	100% <100%> (ø)
...xt-zipkin/src/opentelemetry/ext/zipkin/__init__.py	84.41% <100%> (ø)
.../context/propagation/tracecontexthttptextformat.py	84.72% <100%> (ø)
...xt-jaeger/src/opentelemetry/ext/jaeger/__init__.py	86.54% <100%> (ø)
...sdk/src/opentelemetry/sdk/trace/export/__init__.py	89.2% <100%> (ø)
...-ext-flask/src/opentelemetry/ext/flask/__init__.py	88.88% <100%> (ø)
...telemetry-api/src/opentelemetry/context/context.py	100% <100%> (ø)
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df01d09...f56e583. Read the comment docs.

[toumorokoshi]

Overall the code changes look good. I'll wait for the conflict resolution before approving just to make sure I understand what has been added regarding testing the observe.

Aside from that, the API seems like it would make some scenarios very difficult for me to implement. I wonder if there's been discussion on this?

Batcher being attached to Meter.

It seems the batcher is implicitly responsible for choosing the aggregation, which also means that the batcher is responsible for choosing what sort of metrics to keep around the observations (min, max, count, etc).

I think it's a common scenario to want to have different types of metrics emitted for different types of observation. For example, if my observation is a single metric, there's no need for me to have a a min/max/count version of the metric. If I want to observe that single metric and, for example, CPU, where I want such groupings, I have a fairly complex path to actually configure that:

1. create Exporter
2. create BatcherA (single Aggregator), MeterA for my single metric
3. create PushControllerA with MeterA and Exporter
4. create BatcherB (sum/min/max/count Aggregator), MeterB for my second metric
5. create PushControllerB with MeterB and Exporter

So, effectively I need 2-3 additional data structures (batcher, pushcontroller, meter) for every variant of Aggregator I want to support. It's a lot of overhead.

I'd argue (and I'm guessing this is a spec level conversation) that it should be possible to tie aggregators directly to the observer, so I can choose what statistics I want to track on my observer specifically for my observer.

Observers of different frequencies

It looks like the Observer run rate is dependent on how often the exporter is responsible for collecting data, but it could be possible that I want to observe some metrics at a different rate than others. For example, maybe I want to capture CPU/Memory on a per minute basis, but I want # of coroutines in an event loop faster, maybe every 30 seconds.

In an exporter like prometheus this is irrelevant as the rate is determined by the poller. However, there are plenty of exporters that have a push model, and many metric storage systems support different granularity of time series. Again, it's not great to require a separate meter, exporter, and pushcontroller to support different poll frequencies.

[mauriciovasquezbernal]

I'd argue (and I'm guessing this is a spec level conversation) that it should be possible to tie aggregators directly to the observer, so I can choose what statistics I want to track on my observer specifically for my observer.

It's a more general discussion that applies to other instruments as well, specially Measure that could support multiple types of aggregations.

Currently, the aggregator is chosen based on the instrument type:

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/batcher.py

Lines 43 to 54 in 5a0dae9

	def aggregator_for(self, metric_type: Type[MetricT]) -> Aggregator:
	"""Returns an aggregator based on metric type.
	Aggregators keep track of and updates values when metrics get updated.
	"""
	# pylint:disable=R0201
	if issubclass(metric_type, Counter):
	return CounterAggregator()
	if issubclass(metric_type, Measure):
	return MinMaxSumCountAggregator()
	# TODO: Add other aggregators
	return CounterAggregator()

As far as I know this is an ongoing discussion on the spec about how to choose the aggregator for a given instrument.

Again, it's not great to require a separate meter, exporter, and pushcontroller to support different poll frequencies.

I agree, the current architecture makes it very hard to support observers with different frequencies. I think this limitation comes from the fact that observers are run on collection time, that is determined by the controller. I'm not aware of any discussion about it.

I think these are discussions for the specification.

[toumorokoshi]

LGTM, thanks!

[lzchen]

So, effectively I need 2-3 additional data structures (batcher, pushcontroller, meter) for every variant of Aggregator I want to support. It's a lot of overhead.

@toumorokoshi I am a bit confused as to why you need additional data structures for variants of Aggregators. Aggregators are generated from the metrics that require them, not from the type of Batcher.

[lzchen]

LGTM!