HOW TO: Troubleshooting Spline deployment

[cerveada]

How to make sure everything is working and found issues when deploying Spline?

[cerveada]

Minimal spline deployment consists of Spline Agent, Spline Server, Arango DB and Spline UI. The lineage data flow from the agent direction, so it makes sense to start there.

Troubleshooting Spline Agent

• Make sure you use the correct Agent bundle for Spark and Scala you use.
• Use the latest version of Agent available, that is the version that has most bugs fixed
• To test Agent in isolation, you can use console or logging dispatcher (see: https://github.com/AbsaOSS/spline-spark-agent#lineage-dispatchers). Run your spark job and watch the logs to see if the Agent is generating a lineage data there.
• Remember that Agent produces the lineage only when write occurs.

Once you run your job and see that the lineage is creating by console/logging dispatcher, you can assume Agent is working fine.

Troubleshooting Arango DB

• Connect to Arango UI and see the spline db is there and inside are tables ready to be used be Spline Server

Troubleshooting Spline Server

• Accessing the server via browser you should be able to see the webpage with status ok
• if status is not ok, check the server logs to see what went wrong
• make sure the database URL configuration is correct and there is nothing in the network blocking access from the server to the db

If the server is running fine, you can try to send the lineage data from the agent to the server

• On the Agent, switch to http dispatcher and configure it with the correct producer API url
• Make sure there is nothing in the network blocking access from the Agent to the Server

Now run your job and check that the lineage data appeared in the Arango DB. In case of issues check the logs of the Agent and Server.

Troubleshooting Spline UI

If you are sure everything else is working, you can now run the UI

• make sure the consumer url is correct and acces is not blocked