Sling terminate without error while source connection is killed

[EDORR-NEWORCH]

Hello,

I write a parquet file locally from a SQL Server data source. When I kill the SQL Source query, Sling stops processing but indicates that the task was completed successfully.

This means that we ultimately have no way of being sure that we have recovered all the data in the parquet file. Is there a way to raise an error when the source is interrupted?

[flarco]

Hi, to clarify, you are not interrupting the sling process (with Ctrl+C for example), but killing the SQL query from inside SQL Server, correct?

[sdonati-orch]

Hi, this is correct.
We discoverd this during a network outage => stream stopped reading and as soon as the network went back data has been transferred without error (and number of rows transferred was below what we expected).
Killing the query was the way to repoduce the issue.

[EDORR-NEWORCH]

Hi @flarco ,

Do you plan to add error handling to resolve this issue ? If yes, what will be the deadline ?

Thanks in advance!!

[flarco]

Hey, I am not sure if it's possible. Sling uses the Microsoft official driver, and it should error by itself (like postgres driver, which errors if query is cancelled by server). I have another idea, but needs to be thought out a bit more.

[sdonati-orch]

Hi @flarco ,
His there any way you can help on this point?

[flarco]

hey @sdonati-orch I need to reproduce this on my side.
Can you give me some more context? Is the query being killed intentionally? Or is the network connection between the machine running sling and the database server unstable?
Also, can run with env var DEBUG=TRACE and give me the output? Please ensure to remove any sensitive logs. You could email the log output to [email protected] if that works better for you.

[sdonati-orch]

Hey @flarco ,

Here is the difference between sqserver and mariadb executions with netwwork issue
=> in both case it ends with successful status even if we did not retrieve all the expected data (but with mariadb we can see network interruption)

[flarco]

thanks @sdonati-orch @EDORR-NEWORCH, this was indeed a bug. I pushed a fix. Can you try with the latest dev build?

[sdonati-orch]

Hi @flarco,

Looks good 😊

Thanks a lot !!!!

[EDORR-NEWORCH]

I confirm, it's perfect!!!
Thanks a lot @flarco !!