Add target based scaling support for Netherite

[bachuv]

This PR adds support for target based scaling. It builds upon Azure/azure-functions-durable-extension#2452. This PR is in draft mode since it hasn't been tested E2E yet.

Changes:

• Add NetheriteMetricsProvider and move CollectMetrics code from ScaleMonitor here
• Add NetheriteTargetScaler (implements ITargetScaler)
• Override TryGetTargetScaler in NetheriteProvider
• Add unit tests
• E2E testing

[sebastianburckhardt]

I have spent some time to modify the logic so it now computes a target instead of a scale recommendation. It still has the same overall structure but I had to use the data provided by the metrics to determine the current scale.

[bachuv]

Thanks Sebastian! I updated the PR and tested your changes locally.

[bachuv]

@sebastianburckhardt, can you point me to where I should add unit tests?

[sebastianburckhardt]

@sebastianburckhardt, can you point me to where I should add unit tests?

I think the best place to add the tests is the DurableTask.Netherite.Tests project.

The other tests in there are mostly end-to-end tests that use some special fixture. In your case, just plain normal unit tests are probably enough, so I don't think you need to use any of those.

[bachuv]

@sebastianburckhardt, can you help provide expected input and output values to test the target based scaling algorithm? The inputs for this test are maxConcurrentTaskActivityWorkItems and maxConcurrentTaskOrchestrationWorkItems.

[bachuv]

@sebastianburckhardt, I copied this from another test setup so let me know if I'm using it correctly. Should this information get sent as an input parameter with different values for each test case?

[sebastianburckhardt]

Is this PR still needed?

[davidmrdavid]

I believe we've parked this for the time being. Right @bachuv? I suspect this may become relevant again once Flex Consumption enter it's GA-prep phase, but we're not there yet.

@sebastianburckhardt: is it a problem if this remains open, just perhaps in draft mode?

[bachuv]

I believe we've parked this for the time being. Right @bachuv? I suspect this may become relevant again once Flex Consumption enter it's GA-prep phase, but we're not there yet.

@davidmrdavid, yes that's right.

[davidmrdavid]

I've taken the liberty of marking this as draft to help signal that it doesn't need active review, and is simply an idle PR meant for future completion. Hope that's ok!

[ericleigh007]

Not to be too obvious, but I guess this is why we were often peppered with log entries warning that target-based scaling could not be used. The log entries are quite noisy.

[davidmrdavid]

Hey @ericleigh007 - do you know if your app somehow "opted in" to target based scaling? If you're getting these warnings without opting in, that could be a problem (and possibly a reason to re-prioritize this work)

[ericleigh007]

I was under the impression that target-based scaling was the default, and you had to opt-out. So we have done nothing specifically for this.
This does seem to look different than the result we get with AzureStorage.
As another peripheral point, I thought that target-based scaling pivoted back to metric-based because mainly lack of management roles, but that may be an incorrect perception.