Move integration test into examples

[mrchtr]

• moved integration test into examples folder
• adjust sample implementation to new interface and how a user would do it
• added script to execute the pipeline using fondant cli
• add execution to ci/cd pipeline

For now I've used a bash script for the execution. If we want to add more integration test we should think about a different approach.

Fixes #727

[RobbeSneyders]

Thanks @mrchtr!

[RobbeSneyders]

As discussed, we need to check each step in the propagation chain of the exit codes:

• Docker container to docker compose: Check current behavior and --abort-on-container-exit flag
• Docker compose to Fondant: We should at least use check_call when running docker compose here
• Fondant to test script: I think we need to reraise the exit code as mentioned in my comment above

[mrchtr]

I've updated the code.

Docker container to docker compose: Check current behavior and --abort-on-container-exit flag
I've checked it and it is not needed. When we use check_call instead of call in the local runners code, the exception is the same.

I've updated the test script. The sample pipeline is located in examples/sample_pipeline. I've adjusted the code of the pipeline that is behaves as a user would expect it. You can go into the folder and run fondant run local pipeline.py for quickly test the pipeline.

Beside that in the folder is a run.sh now. Which is used to execute the sample pipeline within the ci/cd pipeline. It exits with the exit code of the pipeline execution.

I've adopted the ci/cd script. Now it scans the example folders and looks for run.sh scripts, execute each of them and tracks the exit codes. When one of the scripts failed, it will print the failed ones and exit with 1. Otherwise exit 0 when every test passed.

This allows us to add more integration test samples to the example folder in the future.

[RobbeSneyders]

• Docker container to docker compose: Check current behavior and --abort-on-container-exit flag

I think we are still missing this

[mrchtr]

• Docker container to docker compose: Check current behavior and --abort-on-container-exit flag

I think we are still missing this

A bit of back and forth. In the meanwhile--abort-on-container-exit made it to main. I've merge latest main into this branch. Should be up to date now.

[RobbeSneyders]

Yeah I missed your comment about it not being needed before, my mistake. Looks good now.