Disabling low level lineage insert from Producer API

[vishalag001]

Hi @wajda,
I am using spline in my orgainisation and the data volume is huge. The lineage payloads are also sometimes huge for some spark jobs. With Spline 0.7.x the insertExecutionPlan makes all inserts in one go and it sometimes takes a lot of processing time( around 1.5 min). Because of this the lineages insert gets timed out on the rest-gateway and it is dropped. And all subsequent requests are kept waiting leading to AsyncTimeoutException error (500 Server error)
Inorder to mitigate this, is it possible to disable the low level lineage collections inserts so that the API processes the request faster and we dont miss on the high level lineages also ?

Error that we usually get

org.springframework.web.context.request.async.AsyncRequestTimeoutException: null
	at org.springframework.web.context.request.async.TimeoutDeferredResultProcessingInterceptor.handleTimeout(TimeoutDeferredResultProcessingInterceptor.java:42)
	at org.springframework.web.context.request.async.DeferredResultInterceptorChain.triggerAfterTimeout(DeferredResultInterceptorChain.java:79)
	at org.springframework.web.context.request.async.WebAsyncManager.lambda$startDeferredResultProcessing$5(WebAsyncManager.java:438)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.springframework.web.context.request.async.StandardServletAsyncWebRequest.onTimeout(StandardServletAsyncWebRequest.java:151)
	at org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:44)
	at org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:133)
	at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:137)
	at org.apache.coyote.AbstractProcessor.dispatch(AbstractProcessor.java:241)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:57)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2160)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:750)

[wajda]

Well, not right out of the box, but there are solutions. If you can control agents, doing it on the agent would be the easiest and the most efficient way. Write a custom filter and remove all unnecessary details from the execution plan. Just be careful to correctly remove references as well. For instance, if you remove attributes or expressions you also need to remove all instances of AttrOrExprRef from all the rest of the objects in the payload recursively. Likewise if you remove some operations you need to update childIds of those operations that are left, so the operation DAG structure remains correct. Please refer the following example project of how to develop and deploy the Spline agent extensions - https://github.com/AbsaOSS/spline-getting-started/tree/main/spark-agent-extension-example

On the other hand there are might be some things that you can do to improve the server response time. Of course we need to know more details about your particular deployment, but this is what comes to my mind straight away:

1. Try to build Spline server form the develop branch. It contains unreleased version 1.0.0, but it should be stable enough and is worth to try. It comes with a lot of changes on the persistent level that might make big inserts significantly faster as those aren't executed all in one go anymore. It would also be very beneficial to us to know how the version 1.0 is performing on a real big payloads in comparison to the 0.7. So if you could do some tests on this we would appreciate your feedback. The version is going to be released as soon as possible, we just need to implement a transaction recovery algorithm for a rare case of database failures, but that's not affecting the normal use-case.

2. Look at your ArangoDB deployment. Use the latest Arangodb version (we use 3.10, but there is also 3.11 available). Usually every next major update of ArangoDB comes with a number of performance and scalability improvements, so it's certainly worth trying out. If you use a cluster deployment mode, try reducing the number of replicas and better give nodes more resources (ArangoDB cluster mode uses sync replication, so the response time of the insert request of the whole cluster would be aligned with the response time of the slowest replica). Also try to increase the number of coordinators, and see if it makes any difference.

3. If it's appropriate, try ArangoDB Enterprise edition, they offer a lot of optimizations under the hood that are not available in the community edition.

4. Give more RAM to the ArangoDB nodes and use faster disks.

Please Let us know if something of that helps.

[vishalag001]

Hi @wajda Thank you for a quick reply. Can I run the develop brnach spline backend with spline-agent 0.7.x version or it requires spline-agent to be upgraded as well ?
Similarly is it possible to run spline 0.7.x version with spline-agent 1.x version ?

[wajda]

Yes, and yes. We strive for mutual backward and forward compatibility between the agent and the server. If something isn't working, let us know.