Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add monitorType JMX #472

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add monitorType JMX #472

wants to merge 1 commit into from

Conversation

alaturqua
Copy link
Contributor

Description

This pull request introduces a new monitorType for JMX-based monitoring in Trino Gateway. It includes documentation and configuration details for retrieving cluster information using specific JMX metrics.

Details

  • Component: JMX Monitoring in Trino Gateway

  • New Feature: Added support for monitorType: JMX to enable cluster monitoring via JMX metrics.

  • Metrics Used:

    • trino.execution:name=QueryManager
    • trino.metadata:name=DiscoveryNodeManager

  • Configuration Instructions:

    • Users can configure a username and password using backendState.
    • Credentials must be valid across all clusters with read rights on system_information.

  • Example Configuration:

    backendState:
  username: "user"
  password: "password"
clusterStatsConfiguration:
  monitorType: JMX

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

*

@cla-bot cla-bot bot added the cla-signed label Sep 17, 2024
@alaturqua
Copy link
Contributor Author

I am not very experienced with Java and Testing in Java. Please feel free to add tests.

@alaturqua alaturqua changed the title Add monitorType JMX via trino's /v1/jmx/mbean endpoint Add monitorType JMX Sep 17, 2024
ebyhr
ebyhr reviewed Sep 21, 2024
View reviewed changes
gateway-ha/src/main/java/io/trino/gateway/ha/config/ClusterStatsMonitorType.java
@@ -18,5 +18,6 @@ public enum ClusterStatsMonitorType
NOOP,
INFO_API,
UI_API,
JDBC
JDBC,
JMX
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why existing monitor types don't work well in your environment?

Copy link
Contributor Author

@alaturqua alaturqua Sep 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using oauth for web-ui on trino and QueryCountBasedRouter on Trino Gateway, which requires ClusterStats regarding running and queued queries.

  • UI_API does not work because of the Oauth
  • JDBC works but often ClusterStats queries go into queue and they time out if it takes longer than 10 secs. This makes Trino Gateway throw 500 erros if it happens on adhoc cluster
  • JMX is working flawlessly now, I implemented it. This could be the temporary solution, until Trino exposes required stats behind /v1/api

Copy link
Member

@ebyhr ebyhr Sep 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your explanation.

I don't think we want to document it if it's the temporary solution. Please revret changes in docs/installation.md.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

ebyhr
ebyhr reviewed Sep 27, 2024
View reviewed changes
Copy link
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just skimmed.

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
@@ -0,0 +1,153 @@
/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add tests even if this is a temporary solution.

gateway-ha/src/main/java/io/trino/gateway/ha/handler/QueryIdCachingProxyHandler.java
@@ -26,6 +26,7 @@ public class QueryIdCachingProxyHandler
public static final String OAUTH_PATH = "/oauth2";
public static final String AUTHORIZATION = "Authorization";
public static final String USER_HEADER = "X-Trino-User";
public static final String JMX_PATH = "/v1/jmx/mbean";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to ClusterStatsJmxMonitor class.

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
{
String jmxUrl = backend.getProxyTo() + JMX_PATH + "/" + mbeanName;
log.debug("Querying JMX at URL: %s", jmxUrl);
OkHttpClient client = new OkHttpClient.Builder().build();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use airlift HTTP client?

gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java

Call call = client.newCall(request);

try (Response res = call.execute()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid abbreviations. https://github.com/trinodb/trino/blob/master/.github/DEVELOPMENT.md#avoid-abbreviations

andythsu
andythsu reviewed Oct 14, 2024
View reviewed changes
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Call call = client.newCall(request);

try (Response res = call.execute()) {
if (fromStatusCode(res.code()) == io.airlift.http.client.HttpStatus.OK) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd be more concise if we can import io.airlift.http.client.HttpStatus

@alaturqua
Copy link
Contributor Author

Thanks for the reviews. I was on vacation and didn't have time to implement the requested changes yet. I will take care of them in the next days.

I might need some support to activate the /v1/jmx/mbean endpoint in the test environment for testing.

@willmostly
Copy link
Contributor

@alaturqua please ping me when this is ready for another look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andythsu andythsu andythsu left review comments

@ebyhr ebyhr ebyhr left review comments

@mosabua mosabua Awaiting requested review from mosabua

@willmostly willmostly Awaiting requested review from willmostly

Assignees

@willmostly willmostly

Labels
cla-signed
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@alaturqua @willmostly @ebyhr @andythsu