feat: add managed resource metrics

[teeverr]

Description of your changes

This PR adds two new metrics that can be used for exposing information about reconciliations and request to aws per managed resource:

• managed_resource_reconciles_total
• aws_api_calls_reconcile_managed_resource_total

Which is especially helpful for managed resources which use relatively expensive api calls, like service catalog.

This two metrics turn on by feature flag --enable-managed-resource-metrics or env variable ENABLE_MANAGED_RESOURCE_METRICS.
For now these metrics are implemented only in ProvisionedProduct.

Metrics sample:

aws_api_calls_reconcile_managed_resource_total{api_version="servicecatalog.aws.crossplane.io/v1alpha1",controller_operation_type="create",kind="ProvisionedProduct",resource_name="kirill-test-2"} 1
aws_api_calls_reconcile_managed_resource_total{api_version="servicecatalog.aws.crossplane.io/v1alpha1",controller_operation_type="delete",kind="ProvisionedProduct",resource_name="kirill-test-2"} 0
aws_api_calls_reconcile_managed_resource_total{api_version="servicecatalog.aws.crossplane.io/v1alpha1",controller_operation_type="observe",kind="ProvisionedProduct",resource_name="kirill-test-2"} 27415
aws_api_calls_reconcile_managed_resource_total{api_version="servicecatalog.aws.crossplane.io/v1alpha1",controller_operation_type="update",kind="ProvisionedProduct",resource_name="kirill-test-2"} 0
managed_resource_reconciles_total{api_version="servicecatalog.aws.crossplane.io/v1alpha1",kind="ProvisionedProduct",resource_name="kirill-test-2"} 5483

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable test to ensure this PR is ready for review.

[chlunde]

Could we avoid changing each manage resource (at least more than a few lines)? Perhaps this belongs in provider-runtime?

See also crossplane/crossplane-runtime#555

[teeverr]

Could we avoid changing each manage resource (at least more than a few lines)?

Do you have some ideas or tips? As I understand it would be possible if crossplane-runtime provided MRs metrics, as minimum managed_resource_reconciles_total could be implemented there, but i'm not sure about aws_api_calls_reconcile_managed_resource_total

[chlunde]

@teeverr I think it's a good idea to discuss it with sig-observability: https://github.com/crossplane/crossplane?tab=readme-ov-file#special-interest-groups-sig

re aws_api_calls_reconcile_managed_resource_total I wonder if the cardinality is too much. I don't have any suggestions on how to implement it with less code, worst case might be to have some information injected into context.Context (information about the managed resource GVK + name), but I have not looked into it, and discussing it with the SIG would be good I think.