Performance issue with observability metrics for unique URLs

[HeshanSudarshana]

Description:

Some of the metrics provided in the observability feature consist of tags such as http_url which is a string with the invocation URL. Therefore, when there are query params or path params of high entropy such as timestamp, the unique metric count rises which may lead to a performance issue.

The following metric is an example. (The URL provided is for the request https://localhost:9095/v2/pet/findByStatus?status=available)

http_response_time_seconds_value{http_method="GET", http_url="/v2/pet/findByStatus_status_available", instance="172.17.0.1:9000", job="Ballerina_Metrics", protocol="http", resource="get3f443a64f0bf4231b0bbe0435f4a4e73", service="Swagger_space_Petstore__1_0_5"}

[chamil321]

Ballerina 1.2.x branch fix is done via ballerina-platform/ballerina-lang#31706

[chamil321]

As per the discussion [P2] [INTERNAL] (STLSUB-8) - Micro Gateway 3.2.3 goes Out of Memory upon Load... client http.url metric made configurable via ballerina-platform/ballerina-lang#32048

[nadundesilva]

We will need to do the same change for Ballerina Swan Lake as well.

• In server side metrics (incoming requests), we can do either of the following
  • change the value added to the templated path (e.g. /employees/[string employeeId]/details) (prefered)
  • change the tag to a tracing only tag
• In client side metrics (outgoing requests), we can change the tag to a tracing only tag

Note: We can use the following methods to add a tag only to tracing data by using the falling functions

• In Ballerina Code: observe:addTagToSpan(tagKey, tagValue)
• In native Java code: observerContext.getSpan().addTag(tagKey, tagValue)