Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11355][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15 #11356

Merged
merged 2 commits into from
Oct 16, 2024

Conversation

PeterZh6
Copy link
Contributor

@PeterZh6 PeterZh6 commented Oct 14, 2024

Fixes [Feature][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15 #11355

Coauthored with yangyang-12-wq

Motivation

The purpose of this PR is to enhance observability for the sort-connector-mongodb-cdc-v1.15 by introducing new source metrics. These metrics will provide detailed insights into the deserialization process and checkpoint management, facilitating better monitoring, troubleshooting, and optimization for users.

Modifications

Basically the same way of modifying as that of #11130

Deserialization Metrics:
Added counters to track successful and failed deserialization attempts (numDeserializeSuccess, numDeserializeError).
Added latency gauge to measure time taken for deserialization (deserializeTimeLag).

SnapshotState Metrics:
Added counters for the number of snapshots created (numSnapshotCreate) and errors encountered during snapshot operations (numSnapshotError).

NotifyComplete Metrics:
Added a counter to track completed snapshots (numCompletedSnapshots).
Added latency gauge for the time between snapshot creation and checkpoint completion (snapshotToCheckpointTimeLag).

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

Can use the same way of verification as that in #11130, an simpler way, however, can be used in the following way.

  • Use an End-to-End test called inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Mongodb2StarRocksTest.java.

  • Add a while loop after all the checks in testMongodbUpdateAndDelete to stop the testing container from being torn down.

  • Add

'inlong.metric.labels' = 'groupId=mongoGroup&streamId=mongoStream&nodeId=mongoNode'

in the source connection option of inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/resources/flinkSql/pulsar_test.sql.

  • Run the maven test.

  • Wait until all the tests down(should take a while), visit localhost:8081, which is the url for Flink Web Dashboard.

  • Click the operator, and check the metrics column.
    The result should be like this:

MongoDBMetrics1

Documentation

  • Does this pull request introduce a new feature? Yes
  • If yes, how is the feature documented? Docs in inlong-website repo

@aloyszhang aloyszhang merged commit 67676d8 into apache:master Oct 16, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15
3 participants