Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't set configs on docker run #19

Closed
nelsonfassis opened this issue Jun 27, 2018 · 10 comments
Closed

Can't set configs on docker run #19

nelsonfassis opened this issue Jun 27, 2018 · 10 comments

Comments

@nelsonfassis
Copy link

nelsonfassis commented Jun 27, 2018 •
edited
Loading

I am trying to set some configs like user and password on my cassandra_exporter, however, everytime I try to mount the config on the container, I have this error mentioned below.

I tried to set those configs using -e "VARKey value" but it also didn't work.

As I see there are many people using it, I just assume I'm doing it wrong, but I didn't figure out how to properly use it.

The configs are currently on /tmp/config.yml, but it was in different directories before, I was just trying to move around to check permissions.

docker run --privileged --rm -ti -v /tmp/config.yml:/etc/cassandra_exporter/config.yml --name cassandra-exporter criteord/cassandra_exporter

Starting Cassandra exporter
JVM_OPTS: 
CASSANDRA_EXPORTER_CONFIG_user 
sed: cannot rename /etc/cassandra_exporter/sedjzhwca: Device or resource busy
@erebe
Copy link
Contributor

erebe commented Jun 28, 2018 •
edited
Loading

Hello,

You need to mount a directory instead of a single file, as sed is trying to replace it and this is not possible when the file is a mounted one.
See moby/moby#6011 for more details.

So do instead a docker run --privileged --rm -ti -v /tmp/config/:/etc/cassandra_exporter/ --name cassandra-exporter criteord/cassandra_exporter

@nelsonfassis
Copy link
Author

Ahhhh, that makes sense, but I'm still hitting this issue
It is like it is not even reading the config file. I'm using the sample config from the README page. I changed listen port and config host just to see if I'd see something different but it was ignored :(
The service is running and I can telnet into it.
docker run --privileged --rm -ti -v /tmp/config/:/etc/cassandra_exporter/ --name cassandra-exporter criteord/cassandra_exporter

Starting Cassandra exporter
JVM_OPTS: 
CASSANDRA_EXPORTER_CONFIG_user 
CASSANDRA_EXPORTER_CONFIG_ssl False
CASSANDRA_EXPORTER_CONFIG_host localhost:7199
CASSANDRA_EXPORTER_CONFIG_listenPort 8080
CASSANDRA_EXPORTER_CONFIG_password 
[main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from /etc/cassandra_exporter/config.yml
[main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)]
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
	at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:148)
	at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:36)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)]
	at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
	at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:207)
	at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
	... 3 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
	at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
	at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
	... 8 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:400)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:243)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:225)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
	at java.base/java.net.Socket.connect(Socket.java:591)
	at java.base/java.net.Socket.connect(Socket.java:540)
	at java.base/java.net.Socket.<init>(Socket.java:436)
	at java.base/java.net.Socket.<init>(Socket.java:213)
	at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 13 more

@erebe
Copy link
Contributor

erebe commented Jun 29, 2018 •
edited
Loading

If you are looking at the run.sh that gets invoked during the start of the container https://github.com/criteo/cassandra_exporter/blob/master/docker/run.sh, you will see that if you don't provide ENV variable it will override them with some default value.

So, you need to override the env variable when starting the container.
docker -e CASSANDRA_EXPORTER_CONFIG_host='foo:1212'

I may need to change that, as I think about it i think it is stupid to put default value in the run.sh

@erebe
Copy link
Contributor

erebe commented Jun 29, 2018

I don't have time today, but I will remove the default values from the run.sh next week

@erebe
Copy link
Contributor

erebe commented Jul 2, 2018

If you repull the docker image (latest and 2.0.0) you should not get the default values overriding your config file.

Let me know if everything is OK,
Thanks for having reported the issue

@nelsonfassis
Copy link
Author

I still can't create the container, but I don't think the Cassandra Exporter is to blame here...

docker run -e CASSANDRA_EXPORTER_CONFIG_host='10.40.2.112:7199' -e CASSANDRA_EXPORTER_CONFIG_user='root' -e CASSANDRA_EXPORTER_CONFIG_password='' --privileged --rm -ti -v /tmp/config/:/etc/cassandra_exporter/ --name cassandra-exporter criteord/cassandra_exporter

Starting Cassandra exporter
JVM_OPTS: 
CASSANDRA_EXPORTER_CONFIG_user root
CASSANDRA_EXPORTER_CONFIG_host 10.40.2.112:7199
CASSANDRA_EXPORTER_CONFIG_password 
[main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from /etc/cassandra_exporter/config.yml
[main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.40.2.112; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)]
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
	at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:148)
	at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:36)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.40.2.112; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)]
	at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
	at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:207)
	at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
	... 3 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.40.2.112; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
	at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
	at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
	at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
	... 8 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:400)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:243)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:225)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
	at java.base/java.net.Socket.connect(Socket.java:591)
	at java.base/java.net.Socket.connect(Socket.java:540)
	at java.base/java.net.Socket.<init>(Socket.java:436)
	at java.base/java.net.Socket.<init>(Socket.java:213)
	at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 13 more

Good chance documentation is outdated and someone set a password for this instance and I can't find it anymore. Will update it later today.

@erebe
Copy link
Contributor

erebe commented Jul 3, 2018 •
edited
Loading

Either a password or your jmx on cassandra is not listening for remote connexion.
You can look like at this issue #4 in order to try it out

@erebe erebe closed this as completed Jul 6, 2018
@nelsonfassis
Copy link
Author

Hi @erebe , sorry for the late response, was working on some critical issues, had to hit pause on this one for a while.
So, seems like JMX wasn't listening remote connections.
I edited /etc/cassandra/cassandra-env.sh and added the options mentioned on #4

-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.rmi.port=7199 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

Now, running my container, it kinda works

docker run -e CASSANDRA_EXPORTER_CONFIG_host='10.40.1.10:7199' -e CASSANDRA_EXPORTER_CONFIG_user='root' -e CASSANDRA_EXPORTER_CONFIG_password='myPassword' --privileged --rm -ti -v /tmp/config/:/etc/cassandra_exporter/ --name cassandra-exporter criteord/cassandra_exporter

Starting Cassandra exporter
JVM_OPTS: 
CASSANDRA_EXPORTER_CONFIG_user root
CASSANDRA_EXPORTER_CONFIG_host 10.40.1.10:7199
CASSANDRA_EXPORTER_CONFIG_password myPassword
[main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from /etc/cassandra_exporter/config.yml
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 52978ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 17670ms for the whole run

And... got stuck. I opened a second terminal to check if any port (I expected por 8080) was open, if I could get metrics and... doesn't seem to work. Would it be an issue with my Cassandra?

@nelsonfassis
Copy link
Author

Silly me.
docker run -p 8080:8080 Fix the issue, metrics were just not being exposed in the node :)
My command now is
docker run -p 8080:8080 -e CASSANDRA_EXPORTER_CONFIG_host='10.40.1.10:7199' -e CASSANDRA_EXPORTER_CONFIG_user='root' -e CASSANDRA_EXPORTER_CONFIG_password='myPassword' --privileged --rm -ti -v /tmp/config/:/etc/cassandra_exporter/ --name cassandra-exporter criteord/cassandra_exporter

And it works!
Thank you so much for all the help @erebe

@erebe
Copy link
Contributor

erebe commented Jul 8, 2018

You welcome,
Btw, except if it is on purpose, you should not need the --privileged on your docker run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erebe @nelsonfassis