[Discussion] Cache instances of MqttClientPublishResult

[gfoidl]

Continuing the profiling session from #1323 and #1324 this PR caches instances of MqttClientPublishResult, so that these 100.000 allocations are gone.

Note: in order to make this work, the client user code needs to be adapted:

-await mqttClient.PublishAsync(publishMessage, stoppingToken);
+using var result = await mqttClient.PublishAsync(publishMessage, stoppingToken);

(Same test program for profiling as in the linked PR on top)

The same technique for caching -- or other maybe better caches -- could be expanded.
For "internal" types (they are all public) it shouldn't have any user visible effect. I tried it for MqttPublishPacket, but didn't know when the instances aren't needed anymore, so wasn't able to return them to the cache. Thus this comit got reverted.

---

Caching / pooling is quite hard to get right, and in essence the GC also can be seen as pool where we rent objects from. Or put in another view: with pooling we create a custom memory allocator.
So it's always a trade-off when we rent objects from our own pool -- especially as we have to return them to our pool to take advantage of pooling, which in case of "GC-pool" the GC does this on our behalf by scanning the objects-roots.

I'm not sure if this change is good or not, so I badged the title with "discussion" and made the PR a draft one, so we can see where it goes.

[chkr1011]

Interesting approach. But I am afraid that we might get cross thread access issues. Lets say the result is also consumed on other threads (like UI threads) the values might get cleared (in the Dispose call) from the receiving thread. So the user must copy those values as soon as possible.

[gfoidl]

For the MqttClientPublishResult this shouldn't be problem, as it depends on when the instance gets disposed.
But now I don't like the approach taken for MqttClientPublishResult as it's very unhandy.

The shown approach may be interesting for "internal types" like MqttPublishPacket, that aren't user facing, so the control of creation / rental and disposal / return to pool can be controled better.

[chkr1011]

@gfoidl I review some places in the v4 branch and managed to remove a lot of "useless" allocations. Not the MqttClientPublishResult but a huge amount of others. Now the memory footprint of my benchmark dropped from 1.6 GB to 400 MB.

Please let me know what you think. Do you have more recommendations?

[gfoidl]

I can do some profiling on v4 branch next week, and let you know then.

[chkr1011]

I will close this discussion for now because a lot of other classes were eliminated and overall memory consumption is optimized. But we can still discuss this topic in the future.

[gfoidl]

Looks much better now

Please let me know what you think. Do you have more recommendations?

Need to dig through the code (changes) first. When I get some ideas, I'll open an issue / PR to discuss them.