Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][broker][PIP-195] Add metrics for bucket delayed message tracker #19716

Merged
merged 11 commits into from
Mar 30, 2023

Conversation

coderzc
Copy link
Member

@coderzc coderzc commented Mar 5, 2023

PIP: #16763

Motivation

Add metrics for bucket delayed message tracker.

Modifications

The new metrics and change metrics:

Name Type Description
pulsar_delayed_message_index_bucket_total Gauge The number of delayed message index buckets (immutable buckets + LastMutableBucket )
pulsar_delayed_message_index_loaded Gauge The total number of delayed message indexes for in the memory.
pulsar_delayed_message_index_bucket_snapshot_size_bytes Gauge The total size of delayed message index bucket snapshot (in bytes).
pulsar_delayed_message_index_bucket_op_count Counter The total number of operate delayed message index bucket snapshot. The state label can be succeed,failed,all (the all means is the total number of all states) and the type label can be create,load,delete,merge.
pulsar_delayed_message_index_bucket_op_latency_ms Histogram The latency of delayed message index bucket snapshot operation with a given quantile (threshold). The labeltype label can be create,load,delete,merge
The label quantile can be:
  • quantile="50" is operation latency between (0ms, 50ms]
  • quantile="100" is operation latency between (50ms, 100ms]
  • quantile="500" is operation between (100ms, 500ms]
  • quantile="1000" is operation latency between (500ms, 1s]
  • quantile="5000" is operation latency between (1s, 5s]
  • quantile="30000" is operation latency between (5s, 30s]
  • quantile="60000" is operation latency between (30s, 60s]
  • quantile="overflow" is operation latency > 1m

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@coderzc coderzc self-assigned this Mar 5, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 5, 2023
@coderzc coderzc added type/feature The PR added a new feature or issue requested a new feature and removed doc-not-needed Your PR changes do not impact docs labels Mar 5, 2023
@coderzc coderzc added this to the 3.0.0 milestone Mar 5, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 5, 2023
@coderzc coderzc force-pushed the bucket_delayed_mertices branch from 3944bd1 to 1d0d7d7 Compare March 5, 2023 15:23
@coderzc coderzc force-pushed the bucket_delayed_mertices branch 5 times, most recently from bff1402 to a0a917e Compare March 28, 2023 15:41
@coderzc coderzc force-pushed the bucket_delayed_mertices branch from a0a917e to a776d18 Compare March 29, 2023 06:26
@coderzc coderzc force-pushed the bucket_delayed_mertices branch from e0486bd to 067cc2c Compare March 29, 2023 06:37
@coderzc coderzc force-pushed the bucket_delayed_mertices branch from f56fcd7 to 8b8ec82 Compare March 29, 2023 08:22
@coderzc coderzc force-pushed the bucket_delayed_mertices branch from 921158f to f10b352 Compare March 29, 2023 09:00
@@ -212,8 +208,8 @@ private CompletableFuture<Void> addEntry(LedgerHandle ledgerHandle, byte[] data)
});
}

CompletableFuture<Enumeration<LedgerEntry>> getLedgerEntryThenCloseLedger(LedgerHandle ledger,
long firstEntryId, long lastEntryId) {
CompletableFuture<Enumeration<LedgerEntry>> getLedgerEntry(LedgerHandle ledger,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ledger is already closed after createBucketSnapshot, so we don't need to close it again.

@coderzc coderzc force-pushed the bucket_delayed_mertices branch from 18952cf to 64967e8 Compare March 30, 2023 03:36
@coderzc coderzc merged commit 68c10ee into apache:master Mar 30, 2023
* Tells whether this DelayedDeliveryTracker contains this message index,
* if the tracker is not supported it or disabled this feature also will return false.
*/
boolean containsMessage(long ledgerId, long entryId);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we remove this method in this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry this addresses a previous comment.
See: #17677 (review)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the context!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker doc-not-needed Your PR changes do not impact docs ready-to-test type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants