[INLONG-11355][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15 #11356
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes [Feature][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15 #11355
Coauthored with yangyang-12-wq
Motivation
The purpose of this PR is to enhance observability for the
sort-connector-mongodb-cdc-v1.15
by introducing new source metrics. These metrics will provide detailed insights into the deserialization process and checkpoint management, facilitating better monitoring, troubleshooting, and optimization for users.Modifications
Basically the same way of modifying as that of #11130
Deserialization Metrics:
Added counters to track successful and failed deserialization attempts (
numDeserializeSuccess
,numDeserializeError
).Added latency gauge to measure time taken for deserialization (
deserializeTimeLag
).SnapshotState Metrics:
Added counters for the number of snapshots created (
numSnapshotCreate
) and errors encountered during snapshot operations (numSnapshotError
).NotifyComplete Metrics:
Added a counter to track completed snapshots (
numCompletedSnapshots
).Added latency gauge for the time between snapshot creation and checkpoint completion (
snapshotToCheckpointTimeLag
).Verifying this change
(Please pick either of the following options)
This change is a trivial rework/code cleanup without any test coverage.
This change is already covered by existing tests, such as:
(please describe tests)
This change added tests and can be verified as follows:
Can use the same way of verification as that in #11130, an simpler way, however, can be used in the following way.
Use an End-to-End test called
inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Mongodb2StarRocksTest.java
.Add a while loop after all the checks in
testMongodbUpdateAndDelete
to stop the testing container from being torn down.Add
in the source connection option of
inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/resources/flinkSql/pulsar_test.sql
.Run the maven test.
Wait until all the tests down(should take a while), visit
localhost:8081
, which is the url forFlink Web Dashboard
.Click the operator, and check the
metrics
column.The result should be like this:
Documentation
inlong-website
repo