vdk-impala: Add support for lineage in vdk-impala

[VladimirPetkov1]

Add support for calculating data lineage for jobs that execute SQL queries
against impala.

Lineage is calculated by examining the query plan of the executed query. The
query plan is accessible from the cursor that executed the query, hence no
additional load is present on the target query engine. As this is not very
official way to obtain the query plan there is a risk that this implentation
may break in future impala releases.However the only option was to execute
additional explain statements that would practically double the amount of
executed queries, which could be problematic for loaded environments. Lineage
is calculated only for successfully executed queries that have actually read
or written data from/to the underlying storage.

Implemented functional tests that verify the behaviour with all templates and
simple jobs. As I hit problems starting impala with pytest, I have manually
run the docker compose file and run the tests against that instance.

New feature (non-breaking change which adds functionality)

Signed-off-by: Vladimir Petkov [email protected]

[antoniivanov]

Thanks for making the change.

I made some more high level questions and suggestions which we need to clear up first. I will take a more detailed look at the code later.