Make the batch processor limit data points rather than metrics.

[kisieland]

Description:
Changes batch processor logic to limit data points rather than metrics.

Link to tracking Issue: #2754

Testing:
Ran go test ./...
I've added some edge case test and alerted existing tests.

[kisieland]

@dashpole @bogdandrutu can you take a look?

[dashpole]

This was approved during the 3/24 sig meeting: #2754 (comment)

[dashpole]

I think this needs a release note in the CHANGELOG.md under breaking changes

[kisieland]

I think this needs a release note in the CHANGELOG.md under breaking changes

Done.
I guess my editor removed trailing spaces from other lines.

[bogdandrutu]

@dashpole please help us to ensure we don't expose unnecessary public APIs if possible, any public API will cost us in the future.

[tigrannajaryan]

Is there any measurable performance impact due to this change?

[kisieland]

@tigrannajaryan I've added new version of benchmark, It looks like there is about 20% increase on the time it takes to process data. But, as the batch does not take a significant time of the pipeline we can live with this, WDYT?
(Feel free to re-run and verify it)
On main branch I got:

$ GOMAXPROCS=12 go test -bench=BenchmarkBatchMetricProcessor -benchtime=20000x
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/batchprocessor
cpu: Intel(R) Xeon(R) CPU @ 2.30GHz
BenchmarkBatchMetricProcessor-12           20000             78924 ns/op
PASS
ok      go.opentelemetry.io/collector/processor/batchprocessor  22.330s

On change-batch-processing I've got:

$ GOMAXPROCS=12 go test -bench=BenchmarkBatchMetricProcessor -benchtime=20000x
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/batchprocessor
cpu: Intel(R) Xeon(R) CPU @ 2.30GHz
BenchmarkBatchMetricProcessor-12           20000             94004 ns/op
PASS
ok      go.opentelemetry.io/collector/processor/batchprocessor  22.502s

NOTE: This comment was edited, once I updated the Benchmark to exclude data initialisation. With the data initialisation included there was no significant differences.

[dashpole]

@jrcamp @Aneurysm9 It would be helpful to have your reviews as well, if you have time.

[kisieland]

@bogdandrutu @Aneurysm9 please take a look

[tigrannajaryan]

Please resolve the merge conflicts.

[kisieland]

Merge conflicts resolved.
@tigrannajaryan @bogdandrutu please take a look, I would appreciate if this PR was included in next release.

[tigrannajaryan]

LGTM, thanks @kisieland

All requested changes addressed

[tigrannajaryan]

@bogdandrutu PTAL, I believe all your comments are addressed and we can merge.

[tigrannajaryan]

@kisieland please squash the commits and add a descriptive commit message. Please also include the results of before/after benchmarking in the commit message.

[kisieland]

@tigrannajaryan Done

[kisieland]

@bogdandrutu PTAL

[tigrannajaryan]

@kisieland please fix the changelog conflict one more time (sorry).

[kisieland]

@tigrannajaryan I don't see any conflicts, I resolved them two times today.

[bogdandrutu]

Small optimization to avoid creating a new metric then copy it to destination, but directly copy metadata and points to the destination metric.

[kisieland]

@bogdandrutu Please merge if everything is ok (merging is blocked for me)

[tigrannajaryan]

@kisieland Thank you for the contribution and for the patience.