receiver/prometheus: add store to track stale metrics

[odeke-em]

Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates #3413
Updates #2961

[odeke-em]

If you'd like the big picture, please see https://github.com/orijtech/opentelemetry-collector/pull/new/receiver-prometheus-add-staleness-tracking+emit-staleness-markers and after it the remotewrite compliance test for Staleness passes with

        --- PASS: TestRemoteWrite/otelcollector/Staleness (10.04s)

Kindly cc-ing related reviewers @dashpole @rakyll @Aneurysm9 @bogdandrutu @tigrannajaryan.

[odeke-em]

@bogdandrutu @tigrannajaryan the failed check seems to be flake from the CI itself and unrelated to code:

6d54c59f90a1c17832fdec2373b613351ac3046a5aa62f66c0e127895e61e5f7
Installing otel-collector_0.28.0-post_amd64.deb ...
Error response from daemon: Error processing tar file(exit status 1): lchown /tmp/otel-collector_0.28.0-post_amd64.deb: no such file or directory
Error: Process completed with exit code 1.

[odeke-em]

'The PR to finish things up is #3423 but unfortunately Github doesn't stack/link changes so that PR has code from this one, but after this one is merged then it'll get to its normal size.

[tigrannajaryan]

Please add a comment to explain the purpose and the intended usage. (Yes, I know the functions below have descriptions. I am looking for something that is more of an overall description of how the store should be used and why).
Links to previous discussions or issues which describe the problem can be also useful.

[odeke-em]

Done, thanks @tigrannajaryan!

[tigrannajaryan]

👍

[alolita]

@odeke-em some of the required tests are failing. Pl take a look and fix.

[odeke-em]

@alolita, the Windows test failure is a flake from an entirely different package that we haven't touched per https://github.com/open-telemetry/opentelemetry-collector/pull/3414/checks?check_run_id=2823559150#step:6:179

ok  	go.opentelemetry.io/collector/config/configauth	0.029s
# go.opentelemetry.io/collector/config/configgrpc [go.opentelemetry.io/collector/config/configgrpc.test]
config\configgrpc\configgrpc_test.go:456:38: cannot use &grpcTraceServer{} (type *grpcTraceServer) as type pdatagrpc.TracesServer in argument to pdatagrpc.RegisterTracesServer:
	*grpcTraceServer does not implement pdatagrpc.TracesServer (wrong type for Export method)
		have Export(context.Context, pdata.Traces) (interface {}, error)
		want Export(context.Context, pdata.Traces) (pdatagrpc.TracesResponse, error)
config\configgrpc\configgrpc_test.go:501:36: cannot use &grpcTraceServer{} (type *grpcTraceServer) as type pdatagrpc.TracesServer in argument to pdatagrpc.RegisterTracesServer:
	*grpcTraceServer does not implement pdatagrpc.TracesServer (wrong type for Export method)
		have Export(context.Context, pdata.Traces) (interface {}, error)
		want Export(context.Context, pdata.Traces) (pdatagrpc.TracesResponse, error)

same for the lint failures being unrelated to this change

make[1]: Leaving directory '/home/runner/work/opentelemetry-collector/opentelemetry-collector'
config/configgrpc/configgrpc_test.go:456:38: cannot use &(grpcTraceServer literal) (value of type *grpcTraceServer) as pdatagrpc.TracesServer value in argument to pdatagrpc.RegisterTracesServer: wrong type for method Export (have func(context.Context, go.opentelemetry.io/collector/consumer/pdata.Traces) (interface{}, error), want func(context.Context, go.opentelemetry.io/collector/consumer/pdata.Traces) (go.opentelemetry.io/collector/internal/pdatagrpc.TracesResponse, error)) (typecheck)
			pdatagrpc.RegisterTracesServer(s, &grpcTraceServer{})
			                                  ^
config/configgrpc/configgrpc_test.go:501:36: cannot use &(grpcTraceServer literal) (value of type *grpcTraceServer) as pdatagrpc.TracesServer value in argument to pdatagrpc.RegisterTracesServer: wrong type for method Export (have func(context.Context, go.opentelemetry.io/collector/consumer/pdata.Traces) (interface{}, error), want func(context.Context, go.opentelemetry.io/collector/consumer/pdata.Traces) (go.opentelemetry.io/collector/internal/pdatagrpc.TracesResponse, error)) (typecheck)
	pdatagrpc.RegisterTracesServer(s, &grpcTraceServer{})
	                                  ^
make[2]: *** [Makefile.Common:26: lint] Error 1

[odeke-em]

All tests now passing on their own after the flakes.

[tigrannajaryan]

👍