[SNOW-870373] Enable JMX metrics for Snowpipe Streaming #674
Conversation
sfc-gh-rcheng
force-pushed
the
rcheng-jmx
branch
from
6a1741b
to
3f6e739
Compare
Codecov Report
@@ Coverage Diff @@
## master #674 +/- ##
==========================================
+ Coverage 87.95% 88.02% +0.06%
==========================================
Files 50 52 +2
Lines 4144 4266 +122
Branches 451 455 +4
==========================================
+ Hits 3645 3755 +110
- Misses 331 342 +11
- Partials 168 169 +1
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
sfc-gh-rcheng
force-pushed
the
rcheng-jmx
branch
from
5b2bf3d
to
534ffb8
Compare
sfc-gh-rcheng
changed the title
sfc-gh-rcheng
requested review from
sfc-gh-japatel,
sfc-gh-tzhang,
sfc-gh-tjones and
a team
as code owners
sfc-gh-rcheng
changed the title
|
Putting reminder here to update our docs when this is available in the next release: |
| /** | ||
| * See {@link com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel} for offset | ||
| * description | ||
| */ | ||
| public static final String OFFSET_PERSISTED_IN_SNOWFLAKE = "persisted-in-snowflake-offset"; | ||
|
|
||
| public static final String LATEST_CONSUMER_OFFSET = "latest-consumer-offset"; | ||
|
|
There was a problem hiding this comment.
They're really useful for our internal debugging, do you think they can be send to Snowhouse as well?
There was a problem hiding this comment.
100% we can send it through the telemetry service, we currently don't have any telemetry coming out streaming specifically (similar to pipe_usage). planning to add it in a follow up pr if i have time. jira
| * @param metricsJmxReporter wrapper class for registering all metrics related to above connector | ||
| * and channel | ||
| */ | ||
| public void registerChannelJMXMetrics( |
There was a problem hiding this comment.
Could we add latency related metrics similar to https://docs.snowflake.com/en/user-guide/kafka-connector-monitor#category-latencies?
There was a problem hiding this comment.
yes, i'm planning to add that in a follow up pr. wanted to keep this one small to just enable JMX, and then add latencies and buffer thresholds later
Tracks the offsetPersistedInSnowflake, processedOffset and latestConsumerOffset for Snowpipe Streaming. This PR only enables the JMX metrics, further metrics may be added in the future as needed, please comment any metrics you think would be useful to track!