Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce number of tags for micrometer #3511

Merged
merged 4 commits into from
Mar 4, 2022

Conversation

tomekl007
Copy link
Contributor

Using the mechanism described in: https://access.redhat.com/documentation/en-us/red_hat_build_of_quarkus/1.11/html/collecting_metrics_in_your_quarkus_applications/con-http-metrics_quarkus-collecting-metrics

It reduces the number of tags reported by the metrics registry, by using {path_parameters}.

After the fix, tags look like this:

0 = "/api/v1/diffs/{diff_params}"
1 = "/api/v1/trees/tree/{ref}/entries"
2 = "/api/v1/trees"
3 = "/api/v1/trees/tree/commitLogExtended"
4 = "/api/v1/contents"
5 = "/api/v1/trees/tree"
6 = "/api/v1/trees/tag/testTag"
7 = "/api/v1/trees/tree/commitLogExtendedUnchanged"
8 = "/api/v1/trees/tree/main"
9 = "/api/v1/trees/branch/{ref}"
10 = "/api/v1/trees/tree/testTag"
11 = "/api/v1/trees/tree/testBranch"
12 = "/api/v1/trees/branch/{branchName}/commit"
13 = "/api/v1/trees/tree/{ref}/log"

so the number of tags is substantially reduced and the warning is no longer emitted.

@tomekl007 tomekl007 linked an issue Mar 3, 2022 that may be closed by this pull request
@tomekl007 tomekl007 requested a review from snazy March 3, 2022 10:56
Copy link
Member

@snazy snazy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach looks good. But I think there are a couple more places like

/content/{key}
/trees//tree/{refName}  (getReferenceByName)
/trees//tree/{refType}/{refName}    (assign/delete, can be a "tag" as well)
/trees//tree/branch/{refName}/transplant
/trees//tree/branch/{refName}/merge

@codecov
Copy link

codecov bot commented Mar 3, 2022

Codecov Report

Merging #3511 (be4e348) into main (790ed7d) will increase coverage by 0.20%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##               main    #3511      +/-   ##
============================================
+ Coverage     86.73%   86.94%   +0.20%     
+ Complexity     3046     2958      -88     
============================================
  Files           376      369       -7     
  Lines         16410    16013     -397     
  Branches       1214     1177      -37     
============================================
- Hits          14233    13922     -311     
+ Misses         1731     1664      -67     
+ Partials        446      427      -19     
Flag Coverage Δ
java 87.38% <ø> (+0.24%) ⬆️
javascript 82.55% <ø> (ø)
python 83.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ioned/persist/adapter/spi/DatabaseAdapterUtil.java 66.10% <0.00%> (-5.09%) ⬇️
...persist/nontx/NonTransactionalDatabaseAdapter.java 84.63% <0.00%> (-1.90%) ⬇️
...nessie/versioned/persist/tx/TxDatabaseAdapter.java 76.25% <0.00%> (-1.46%) ⬇️
...e/versioned/persist/tests/AbstractConcurrency.java 96.45% <0.00%> (-1.42%) ⬇️
...d/persist/adapter/spi/AbstractDatabaseAdapter.java 95.51% <0.00%> (-0.18%) ⬇️
.../main/java/org/projectnessie/model/ContentKey.java 93.54% <0.00%> (ø)
.../org/projectnessie/quarkus/runner/InputBuffer.java
...projectnessie/quarkus/gradle/QuarkusAppPlugin.java
...jectnessie/quarkus/gradle/QuarkusAppExtension.java
...org/projectnessie/quarkus/gradle/ProcessState.java
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 790ed7d...be4e348. Read the comment docs.

Copy link
Contributor

@nastra nastra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some simple test to make sure we're getting all of the required metrics from #3510?

Something like:

@QuarkusTest
@TestProfile(value = QuarkusTestProfileInmemory.class)
public class ITTestMetrics {

  @Test
  void smokeTestMetrics() {
    String body = RestAssured.given()
      .when()
      .basePath("/q/metrics")
      .get()
      .then()
      .statusCode(200)
      .extract()
      .asString();

    assertThat(body).contains("jvm_threads_live_threads");
    // ... verify the other important metrics
  }
}

@tomekl007
Copy link
Contributor Author

tomekl007 commented Mar 3, 2022

Approach looks good. But I think there are a couple more places like

/content/{key}
/trees//tree/{refName}  (getReferenceByName)
/trees//tree/{refType}/{refName}    (assign/delete, can be a "tag" as well)
/trees//tree/branch/{refName}/transplant
/trees//tree/branch/{refName}/merge

I've added additional mappings; however, I am not sure about the /content/{key}. Where is it used and created?

@snazy
Copy link
Member

snazy commented Mar 3, 2022

I've added additional mappings; however, I am not sure about the /content/{key}. Where is it used and created?

It's in org.projectnessie.api.http.HttpContentApi#getContent

@snazy
Copy link
Member

snazy commented Mar 3, 2022

Out of scope and maybe something for later, but is there a way to automatically generate these configs from an openapi spec?
Or maybe even just with some custom annotation processor (that we'd maybe have to build)?

@tomekl007
Copy link
Contributor Author

Out of scope and maybe something for later, but is there a way to automatically generate these configs from an openapi spec? Or maybe even just with some custom annotation processor (that we'd maybe have to build)?

We are hitting this: quarkusio/quarkus#21034 so once it will be resolved, the mapping could be removed. So I am not sure if we want to create a mechanism for generating that.

@snazy
Copy link
Member

snazy commented Mar 3, 2022

We are hitting this: quarkusio/quarkus#21034 so once it will be resolved, the mapping could be removed. So I am not sure if we want to create a mechanism for generating that.

Ah! GTK!

snazy
snazy previously approved these changes Mar 3, 2022
@snazy
Copy link
Member

snazy commented Mar 3, 2022

LGTM!

Can you open a revert-PR that we can then just merge once a Quarkus version with the fix gets released? The PR as a "permanent reminder".

nastra
nastra previously approved these changes Mar 3, 2022
@tomekl007 tomekl007 dismissed stale reviews from nastra and snazy via be4e348 March 4, 2022 09:51
@tomekl007 tomekl007 requested review from nastra and snazy March 4, 2022 09:57
@tomekl007
Copy link
Contributor Author

tomekl007 commented Mar 4, 2022

LGTM!

Can you open a revert-PR that we can then just merge once a Quarkus version with the fix gets released? The PR as a "permanent reminder".

@snazy How do we prevent someone from accidentally merging such a PR? Maybe it's better to create a follow-up ticket?

.asString();

// then
assertThat(body).contains("jvm_threads_live_threads");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems ok for now. We'll follow up with whatever other metrics we want to verify in a separate PR, so let's merge what we have

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just open it as a draft, nobody will/should touch it then

@tomekl007 tomekl007 merged commit 216d616 into projectnessie:main Mar 4, 2022
@ebullient
Copy link

ebullient commented Mar 7, 2022

I am having some trouble reproducing the Quarkus issue mentioned above with the provided reproducer. In your case, how are you making outbound Http requests? The current support looks for usage of the Rest Client (which declares templated URIs in a way we can look for at build time).

@tomekl007
Copy link
Contributor Author

@ebullient

I am having some trouble reproducing the Quarkus issue mentioned above with the provided reproducer. In your case, how are you making outbound HTTP requests? The current support looks for usage of the Rest Client (which declares templated URIs in a way we can look for at build time).

Before the fix in this ticket, the Metrics.globalRegistry.getMeters() was growing to > 100. So this method
(MeterFilter#251):

@Override
public MeterFilterReply accept(Meter.Id id) {
    String value = matchNameAndGetTagValue(id);
    if (value != null) {
        if (!observedTagValues.contains(value)) {
            if (observedTagValues.size() >= maximumTagValues) {
                return onMaxReached.accept(id);
            }
            observedTagValues.add(value);
        }
    }
    return MeterFilterReply.NEUTRAL;
}

the observedTagValues were containing this:

value = "/api/v1/diffs/testDiffFromRef*2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d...testDiffToRef*20fdab673d5cdde68cf50694b00bb9bcadc5c73e98ad1381ba255337f336929e"
observedTagValues = {ConcurrentHashMap$KeySetView@16334}  size = 100
 0 = "/api/v1/trees/branch/contentAndOperation_Put_ImmutableIcebergView_view_spark/commit"
 1 = "/api/v1/trees/tree/contentAndOperation_ImmutableUnchanged_iceberg_unchanged/entries"
 2 = "/api/v1/trees/branch/commitLogExtendedUnchanged/commit"
 3 = "/api/v1/trees/branch/contentAndOperation_Put_ImmutableDeltaLakeTable_delta"
 4 = "/api/v1/trees/branch/contentAndOperation_ImmutableUnchanged_iceberg_unchanged"
 5 = "/api/v1/diffs/testDiffFromRef*2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d...testDiffToRef*4bbd2ff4e9ac08ecd4480883c28d401cbd07c13646ad58bdbf8dce2c7ef1e63b"
 6 = "/api/v1/trees/tree/filterCommitLogByAuthor/log"
 7 = "/api/v1/trees/tree/contentAndOperation_ImmutableDelete_view_spark_delete/entries"
 8 = "/api/v1/trees/branch/contentAndOperation_Put_ImmutableDeltaLakeTable_delta/commit"
 9 = "/api/v1/trees/branch/commitLogExtended"
 10 = "/api/v1/trees/tree/contentAndOperation_ImmutableUnchanged_view_spark_unchanged/entries"
 11 = "/api/v1/diffs/testDiffFromRef...testDiffToRef"
 12 = "/api/v1/trees/branch/filterCommitLogByAuthor/commit"
 13 = "/api/v1/trees/tree/commitLogExtendedUnchanged/log"
 14 = "/api/v1/contents"
 15 = "/api/v1/trees/branch/foo"
 16 = "/api/v1/trees/tree/contentAndOperation_ImmutableUnchanged_view_dremio_unchanged/entries"
 17 = "/api/v1/trees/tag/testTag"
 18 = "/api/v1/trees/branch/filterCommitLogByTimeRange/commit"
 19 = "/api/v1/diffs/testDiffFromRef*2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d...testDiffToRef*7169a3f189d8492be0e55381750f9bed97c6e98b269cd1da07b7a4757287a265"
 20 = "/api/v1/trees/branch/testDiffFromRef/commit"
 21 = "/api/v1/trees/branch/filterCommitLogByTimeRange"
 22 = "/api/v1/trees/branch/commitLogPagingAndFiltering/commit"
 23 = "/api/v1/trees/tree/contentAndOperation_ImmutableDelete_view_dremio_delete/entries"
 24 = "/api/v1/trees/tree/commitLogPagingAndFiltering/log"
 25 = "/api/v1/trees/branch/contentAndOperationAll"
 26 = "/api/v1/trees/tree/contentAndOperation_Put_ImmutableIcebergView_view_dremio/entries"
 27 = "/api/v1/trees/branch/filterCommitLogOperations"
 28 = "/api/v1/trees/branch/filterCommitLogByCommitRange/commit"
 29 = "/api/v1/trees/tree/contentAndOperation_Put_ImmutableIcebergTable_iceberg/entries"
 30 = "/api/v1/trees/tree/commitLogExtended/log"
 31 = "/api/v1/trees/tree/main"
 32 = "/api/v1/diffs/testDiffFromRef...testDiffToRef*2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d"
 33 = "/api/v1/trees/branch/contentAndOperation_Put_ImmutableIcebergTable_iceberg/commit"
 34 = "/api/v1/trees/branch/testDiffToRef"
 35 = "/api/v1/trees/tree/filterCommitLogByProperties/log"
 36 = "/api/v1/diffs/testDiffFromRef*2e1cfa82b035c26cbbbdae632cea070514eb8b773f616aaeaf668e2f0be8f10d...testDiffToRef*a5264163f328c0e60aa4d42b585aec71697795f53b0630b505be36401db73d12"
 37 = "/api/v1/trees/tree/commitLogExtended"
 38 = "/api/v1/trees/branch/testDiffToRef/commit"
 39 = "/api/v1/trees/branch/contentAndOperation_Put_ImmutableIcebergView_view_spark"
 40 = "/api/v1/trees/branch/contentAndOperation_ImmutableDelete_delta_delete/commit"
....
 97 = "/api/v1/trees/tree/DETACHED/log"
 98 = "/api/v1/diffs/testDiffFromRef...DETACHED*a0ef1970f2dbc1550824385f33106ba357dbfc55f2d6722ad72e499405223567"
 99 = "/api/v1/trees/tree/testBranch"

although our interfaces that are defining URLs were using template parameters, for example:

To reproduce it in our project, you can run this test: https://github.com/projectnessie/nessie/blob/cd24dcfc77fd1495e646dbaef521acc7e94c412f/servers/quarkus-server/src/test/java/org/projectnessie/server/ITRestApiDynamo.java
(You need of course comment out config changes introduced in this PR:
https://github.com/projectnessie/nessie/pull/3511/files#diff-ca7eb8ba4ae9613ed09d291d9fa46438d30132d967606d756732efedd7535b20R173-R183)

Please let me know if you have more questions.

@ebullient
Copy link

Thank you! This is server side, not client side, right? (received requests, rather than outbound)?

@tomekl007
Copy link
Contributor Author

Thank you! This is server side, not client side, right? (received requests, rather than outbound)?

yes, server-side.

@imjuoy
Copy link

imjuoy commented Apr 25, 2022

I am also facing a similar issue of higher number of URI tags. Since I am planning to follow the same path that you guys have followed, I have a question regarding having multiple path parameters.
Everywhere I refer to for examples of configurations, it's mostly just one path param. For example:

https://quarkus.io/guides/all-config#quarkus-micrometer_quarkus.micrometer.binder.http-server.match-patterns

https://github.com/tomekl007/nessie/blob/be4e348f4b1b4e12453912cb594a4781dc73a5d6/servers/quarkus-server/src/main/resources/application.properties#L173

In our case, the application has three path parameters. Let's say:
/v1/api/param1/param2/param3

Param2 has limited set of values and will never lead to high cardinality situations.
Is it possible to just record param2 and keep param1 and param3 templatized?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate "Maximum number of URI tags" issue
5 participants