Expand RecordingInstrucments to support collection of observers

[ywangd]

The support is needed for RecordingInstruments to be used in tests for guages with a collection of observers.

Relates: #110630

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

Thanks for adding this! A couple of thoughts:

1. it seems the collection variants of registerLongsAsyncCounter and registerDoubleAsyncCounter are still not handled; can you add that?
2. it also seems we have some duplication: RecordingDoubleCounter and RecordingAsyncDoubleCounter are identical. I think we can remove the second and just keep the former
3. similarly, I think we don't need both RecordingAsyncLongCounter and RecordingLongCounter - it's probably better to change them to be like the double counter.

[ywangd]

@ldematte Thanks for the review.

it seems the collection variants of registerLongsAsyncCounter and registerDoubleAsyncCounter are still not handled; can you add that?

Yep pushed b9c0bd0

it also seems we have some duplication: RecordingDoubleCounter and RecordingAsyncDoubleCounter are identical. I think we can remove the second and just keep the former

The two classes are quite different imo. The former is a synchronous counter with incrementXxx methods to actively publish metrics while the later is the synchronous version with callback to collect metrics. So I don't think we can remove either of them? Or do you mean RecordingDoubleGauge and RecordingAsyncDoubleCounter are fairly identical and you want only one of them? In this case, I'd prefer we keep them separate despite their implementations are mostly the same because they form different hierarchies of instruments. It feels rather confusing to me if we return the same class for a counter and a guage. It would also lead to many cascading changes to production code, e.g. APMMeterRegistry which uses related classes in method names and signatures.

[ldematte]

LGTM; dont' worry about the (maybe) duplicated classes, I can follow up with some cleanup if needed. Thanks!