how to connect to spline server from another machine in the cluster

[Yixuan-L]

hi there, I have deployed the server/ui/arangodb on a machine A, and successfully catched the lineage by spark-submit with jars-parameter of spline-agent on machine A. However, I failed to catch the lineage from a machine B (A and B are in the same cluster).
My submit command is here:(default yarn mode)

spark2-submit --driver-memory 4g --executor-memory 10g --executor-cores 2 --conf spark.sql.adaptive.enabled=true  --class com.test.demoJob --jars /home/spark-2.4-spline-agent-bundle_2.11-0.7.12.jar --conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener --conf spark.spline.lineageDispatcher.http.producer.url=http://machineA-IP:8080/producer /home/test-assembly-2.0.jar` 
The exception I got:
`22/11/30 11:19:00 ERROR harvester.QueryExecutionEventHandlerFactory: Spline initialization failed! Spark Lineage tracking is DISABLED.
za.co.absa.spline.harvester.exception.SplineInitializationException: Spark Agent was not able to establish connection to Spline Gateway
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders$1.applyOrElse(HttpLineageDispatcher.scala:126)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders$1.applyOrElse(HttpLineageDispatcher.scala:124)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
	at scala.util.Try$.apply(Try.scala:192)
	at scala.util.Failure.recover(Try.scala:216)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$.za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders(HttpLineageDispatcher.scala:124)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$serverHeaders$lzycompute(HttpLineageDispatcher.scala:57)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$serverHeaders(HttpLineageDispatcher.scala:57)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$2.apply(HttpLineageDispatcher.scala:58)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$2.apply(HttpLineageDispatcher.scala:58)
	at scala.Option.getOrElse(Option.scala:121)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.<init>(HttpLineageDispatcher.scala:58)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.<init>(HttpLineageDispatcher.scala:46)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher.<init>(HttpLineageDispatcher.scala:52)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at za.co.absa.commons.HierarchicalObjectFactory$$anonfun$instantiate$2.applyOrElse(HierarchicalObjectFactory.scala:50)
	at za.co.absa.commons.HierarchicalObjectFactory$$anonfun$instantiate$2.applyOrElse(HierarchicalObjectFactory.scala:49)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
	at scala.util.Try$.apply(Try.scala:192)
	at scala.util.Failure.recover(Try.scala:216)
	at za.co.absa.commons.HierarchicalObjectFactory.instantiate(HierarchicalObjectFactory.scala:49)
	at za.co.absa.spline.harvester.conf.DefaultSplineConfigurer.createComponentByKey(DefaultSplineConfigurer.scala:142)
	at za.co.absa.spline.harvester.conf.DefaultSplineConfigurer.lineageDispatcher(DefaultSplineConfigurer.scala:120)
	at za.co.absa.spline.harvester.conf.DefaultSplineConfigurer.queryExecutionEventHandler(DefaultSplineConfigurer.scala:117)
	at za.co.absa.spline.harvester.QueryExecutionEventHandlerFactory.za$co$absa$spline$harvester$QueryExecutionEventHandlerFactory$$initEventHandler(QueryExecutionEventHandlerFactory.scala:64)
	at za.co.absa.spline.harvester.QueryExecutionEventHandlerFactory$$anonfun$createEventHandler$6.apply(QueryExecutionEventHandlerFactory.scala:43)
	at za.co.absa.spline.harvester.QueryExecutionEventHandlerFactory$$anonfun$createEventHandler$6.apply(QueryExecutionEventHandlerFactory.scala:42)
	at za.co.absa.spline.harvester.QueryExecutionEventHandlerFactory.withErrorHandling(QueryExecutionEventHandlerFactory.scala:55)
	at za.co.absa.spline.harvester.QueryExecutionEventHandlerFactory.createEventHandler(QueryExecutionEventHandlerFactory.scala:41)
	at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener$.za$co$absa$spline$harvester$listener$SplineQueryExecutionListener$$constructEventHandler(SplineQueryExecutionListener.scala:67)
	at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener.<init>(SplineQueryExecutionListener.scala:37)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2737)
	at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2726)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2726)
	at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$$lessinit$greater$1.apply(QueryExecutionListener.scala:83)
	at org.apache.spark.sql.util.ExecutionListenerManager$$anonfun$$lessinit$greater$1.apply(QueryExecutionListener.scala:82)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.sql.util.ExecutionListenerManager.<init>(QueryExecutionListener.scala:82)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$listenerManager$2.apply(BaseSessionStateBuilder.scala:270)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$listenerManager$2.apply(BaseSessionStateBuilder.scala:270)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.listenerManager(BaseSessionStateBuilder.scala:269)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:297)
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1104)
	at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:145)
	at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:144)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144)
	at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141)
	at org.apache.spark.sql.SparkSession.udf(SparkSession.scala:215)
	at com.garena.importJob.dim.TransifyApi$.getLanguageInfo(TransifyApi.scala:30)
	at com.garena.importJob.dim.Transify$.main(Transify.scala:13)
	at com.garena.importJob.dim.Transify.main(Transify.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:367)
	at scalaj.http.HttpRequest.exec(Http.scala:343)
	at scalaj.http.HttpRequest.asString(Http.scala:492)
	at za.co.absa.spline.harvester.dispatcher.httpdispatcher.rest.RestEndpoint.head(RestEndpoint.scala:33)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders$2.apply(HttpLineageDispatcher.scala:115)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$$anonfun$za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders$2.apply(HttpLineageDispatcher.scala:115)
	at scala.util.Try$.apply(Try.scala:192)
	at za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher$.za$co$absa$spline$harvester$dispatcher$HttpLineageDispatcher$$getServerHeaders(HttpLineageDispatcher.scala:115)
	... 73 more

Thanks

[wajda]

It all depends on your particular cluster network configuration. Spline agents is executed on the Spark driver machine, and connects to the given producer URL via HTTP protocol as usual. It you can curl it from that machine, so can do Spline agent. Check your networking.