Add dynamic metric configuration support

[wtong98]

NOTE: this is an experimental feature, not blocking for GA

Follows from this otep

Goal

We introduce a system for dynamically configuring metric collection schedules. The system includes three major changes:

• collection periods may be changed during runtime via a configuration service
• collection periods are keyed per-metric, rather than collecting all metrics
• a configuration service may read config data from a local file or a remote (potentially third-party) backend

An example implementation is almost completed and will be sent to the collector-contrib and go-contrib repos. In the meantime, we'd love to hear your thoughts on this specification and the related protocol!

About

We're an intern group at Google. Members include:

• Ming Chen @vmingchen (host)
• Josh Montana @jdmontana (co-host)
• Churan Chen @churanchen (intern, now done)
• William Tong @wtong98 (me! also an intern)

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ William Tong (4542194, 98672e5, ece626e, e9364b3, 4e9ebff)
• ✅ Ming Chen (9fa0491)

[jmacd]

This is great. /I definitely support this and the notion of dynamic configuration for metric instruments and for metric and tracing SDKs in general. My only slight concern is that implementing dynamic or multiple Metric collection periods isn't the easiest place to start, but it's definitely good functionality to have. 😀

[jkwatson]

I think that per-metric push-intervals should be de-scoped for GA. That's a significant ask to be built in every language, and I would recommend removing it for now.

[wtong98]

@jkwatson Definitely, the per-metric collection periods are a sweeping change, and will likely require substantial work (although for the Go SDK, we came up with a relatively contained way to introduce the change -- will make a PR on go-contrib soon). Overall, I'd be fine if the spec remained experimental and no hard changes included at all until after GA. The plan on our end has been to introduce prototype implementations to the go-contrib and collector-contrib repos, and if feedback is positive and it seems like a good idea, to migrate changes into the main project (probably after GA). Does that sound like a good way forward?

[huyan0]

I think its a really interesting idea. Thanks.

[jmacd]

I believe we decided to call this an experiment so that it will not be a requirement for SDKs to implement. Given that, I don't feel the objections to the specific kind of configuration being delivered are very important.