HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode #4452

taklwu · 2022-05-20T00:11:39Z

tested locally and now standalone with these configuration export HBASE_TRACE_OPTS="-Dotel.traces.exporter=jaegar -Dotel.metrics.exporter=none" in conf/hbase-env.sh no longer failed

before the change, it's always using the default and create Otlp metrics exporter but it cannot be done.

[opentelemetry.auto.trace 2022-05-19 17:09:13:587 -0700] [grpc-default-executor-3] WARN io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter - Failed to export metrics
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:534)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/0:0:0:0:0:0:0:1:4317
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Apache-HBase · 2022-05-20T00:22:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 52s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests
uname	Linux 01a94c808939 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	40 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-20T00:22:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 13s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		2m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests
uname	Linux 9a573bd93aab 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	33 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-20T00:23:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	spotless	1m 10s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	spotless	0m 36s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		3m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests	dupname asflicense spotless shellcheck shelldocs
uname	Linux 391b2ca6a53e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	34 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/1/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-05-20T12:50:50Z

bin/hbase

@@ -508,7 +508,7 @@ add_opentelemetry_agent() {
    fi
    agent_jar=$(tr ':' '\n' < "${f}" | grep opentelemetry-javaagent)
  fi
-  HBASE_OPTS="$HBASE_OPTS -javaagent:$agent_jar"
+  HBASE_OPTS="$HBASE_TRACE_OPTS -javaagent:$agent_jar"


Good catch!

I don't think this correct either. Shouldn't we be adding HBASE_TRACE_OPTS to HBASE_OPTS, not overriding HBASE_OPTS ?

Maybe instead we need an if-block specifically for standalone mode?

oops, you're right, I was just testing on standalone mode and forgot we may have more in HBASE_OPTS

ndimiduk · 2022-05-20T13:00:27Z

Reading back over the patch on #3762. Maybe the fix is to require the operator to assign HBASE_OPTS="${HBASE_OPTS} ${HBASE_TRACING_OPTS}" explicitly in conf/hbase-env.sh ? So, after the existing commented line that assigns a value to HBASE_TRACING_OPTS we have a commented line that assigns HBASE_OPTS="${HBASE_OPTS} ${HBASE_TRACING_OPTS}". Then we can document that if you want to have per-process tracing options, instead of this line, you use the block below.

What do you think?

ndimiduk · 2022-05-20T13:06:06Z

Something like,

modified   conf/hbase-env.sh
@@ -149,11 +149,14 @@
 # https://github.com/open-telemetry/opentelemetry-java-instrumentation for details on how to
 # configure exporters and other components through system properties.
 #
-# The presence HBASE_TRACE_OPTS indicates that tracing should be enabled, and serves as site-wide
-# settings.
+# The presence HBASE_TRACE_OPTS indicates that tracing should be enabled, adding the agent to the
+# JVM launch command.
 # export HBASE_TRACE_OPTS="-Dotel.traces.exporter=none -Dotel.metrics.exporter=none"
 #
-# Per-process configuration variables allow for fine-grained configuration control.
+# You must explicitly add HBASE_TRACE_OPTS to HBASE_OPTS. Do this site wide by uncommenting this line,
+# export HBASE_OPTS="${HBASE_OPTS} ${HBASE_TRACE_OPTS} -Dotel.resource.attributes=service.name=hbase-standalone"
+# Or, enabled tracing with per-process configuration variables, allowing for fine-grained
+# configuration control.
 # export HBASE_SHELL_OPTS="${HBASE_SHELL_OPTS} ${HBASE_TRACE_OPTS} -Dotel.resource.attributes=service.name=hbase-shell"
 # export HBASE_JSHELL_OPTS="${HBASE_JSHELL_OPTS} ${HBASE_TRACE_OPTS} -Dotel.resource.attributes=service.name=hbase-jshell"
 # export HBASE_HBCK_OPTS="${HBASE_HBCK_OPTS} ${HBASE_TRACE_OPTS} -Dotel.resource.attributes=service.name=hbase-hbck"

ndimiduk · 2022-05-20T13:08:47Z

Actually, site-wide is not sufficient for any multi-process deployment. In most cases, the operator really should use the per-process options so that they can provide proper service names in trace metadata.

ndimiduk

I think it's better to leave this up to the operator to configure explicitly in conf/hbase-env.sh. Alternatively, you'll need to re-work how we handle per-process configuration.

taklwu · 2022-05-20T17:28:54Z

I think it's better to leave this up to the operator to configure explicitly in conf/hbase-env.sh. Alternatively, you'll need to re-work how we handle per-process configuration.

I got the comment for the conf/hbase-env.sh, but I need to spend some time to think if the current per-processes has anything wrong....

…standalone mode

taklwu · 2022-05-20T20:27:55Z

I rethink about your comment on per-processor configuration, the current per-process configuration should be the best solution. I was from the standalone testing environment, and that's minor to the overall configuration.

such I changed the subject of this PR to add additional comment when using HBASE_TRACE_OPTS with standalone mode, that should solve the confusion as an user for this environment parameters.

or you can say this task is not necessary, we can just close it.

Apache-HBase · 2022-05-20T20:32:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		2m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests
uname	Linux d221c8a56f96 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	32 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-20T20:32:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 13s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		2m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests
uname	Linux 0be43945b1c2 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	40 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-20T20:33:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	spotless	0m 36s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	shellcheck	0m 0s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	spotless	0m 37s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 7s	The patch does not generate ASF License warnings.
		3m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4452
Optional Tests	dupname asflicense spotless shellcheck shelldocs
uname	Linux 690f4e5eab8a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c7eb30d`
Max. process+thread count	33 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4452/2/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk

Yep, looks right to me. Thanks for noticing this one!

…standalone mode (#4452) Signed-off-by: Nick Dimiduk <[email protected]>

…standalone mode (apache#4452) Signed-off-by: Nick Dimiduk <[email protected]>

…TS with standalone mode (apache#4452)" This reverts commit 445573e.

taklwu requested a review from ndimiduk May 20, 2022 00:11

ndimiduk reviewed May 20, 2022

View reviewed changes

ndimiduk requested review from apurtell, Apache9 and joshelser May 20, 2022 13:00

ndimiduk requested changes May 20, 2022

View reviewed changes

taklwu changed the title ~~HBASE-27055 Fix HBASE_TRACE_OPTS when adding opentelemetry agent~~ HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode May 20, 2022

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with …

2997756

…standalone mode

taklwu force-pushed the HBASE-27055 branch from 44e0fd7 to 2997756 Compare May 20, 2022 20:24

taklwu requested a review from ndimiduk May 20, 2022 20:28

ndimiduk approved these changes May 23, 2022

View reviewed changes

taklwu merged commit 194f8c2 into apache:master May 24, 2022

taklwu added a commit that referenced this pull request May 24, 2022

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with …

02c1fef

…standalone mode (#4452) Signed-off-by: Nick Dimiduk <[email protected]>

taklwu added a commit that referenced this pull request May 24, 2022

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with …

20aaa1e

…standalone mode (#4452) Signed-off-by: Nick Dimiduk <[email protected]>

wenwj0 pushed a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with …

445573e

…standalone mode (apache#4452) Signed-off-by: Nick Dimiduk <[email protected]>

wenwj0 added a commit to wenwj0/hbase that referenced this pull request Jun 14, 2022

Revert "HBASE-27055 Add additional comments when using HBASE_TRACE_OP…

118c0cf

…TS with standalone mode (apache#4452)" This reverts commit 445573e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode #4452

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode #4452

taklwu commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

ndimiduk May 20, 2022

ndimiduk May 20, 2022

taklwu May 20, 2022

ndimiduk commented May 20, 2022 •

edited

Loading

ndimiduk commented May 20, 2022 •

edited

Loading

ndimiduk commented May 20, 2022

ndimiduk left a comment

taklwu commented May 20, 2022

taklwu commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

ndimiduk left a comment

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode #4452

HBASE-27055 Add additional comments when using HBASE_TRACE_OPTS with standalone mode #4452

Conversation

taklwu commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

ndimiduk May 20, 2022

Choose a reason for hiding this comment

ndimiduk May 20, 2022

Choose a reason for hiding this comment

taklwu May 20, 2022

Choose a reason for hiding this comment

ndimiduk commented May 20, 2022 • edited Loading

ndimiduk commented May 20, 2022 • edited Loading

ndimiduk commented May 20, 2022

ndimiduk left a comment

Choose a reason for hiding this comment

taklwu commented May 20, 2022

taklwu commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

Apache-HBase commented May 20, 2022

ndimiduk left a comment

Choose a reason for hiding this comment

ndimiduk commented May 20, 2022 •

edited

Loading

ndimiduk commented May 20, 2022 •

edited

Loading