[data_pipelines][rne] Save flux json file even if it's not completed

[HAEKADI]

Given the unreliable nature of the RNE API, it's crucial to enhance the robustness of our workflow to ensure we capture as much data as possible. Specifically, the goal is to persist all data received from the API, even if it's incomplete for a particular day.

In response to this, this PR introduces a mechanism to save JSON files, even if they are unfinished, into MinIO in the event of an exception during the API request. In the subsequent execution of the workflow, the system will retrieve the latest saved file from MinIO. It will then extract the last siren in the file, allowing the workflow to resume from that point onwards.

This approach serves as a resilient strategy, safeguarding against potential data loss caused by API disruptions. By storing partial data in MinIO, we ensure that any information obtained before encountering an exception is preserved. This way, the workflow can continue seamlessly by picking up where it left off, focusing on the latest available data.

In summary, this PR not only addresses the fickle nature of the RNE API by saving incomplete data but also establishes a reliable mechanism to resume workflow execution from the last successfully obtained siren, contributing to the overall robustness of our data retrieval process.

[XavierJp]

I dont really understand how you use the first_execlogic to determine that previous workflow failed. Plus I think ti needs a better name

[XavierJp]

ti is not very self explanatory

[HAEKADI]

As discussed, ti stands for task instance and represents a specific run of a task within the context of a DAG.
The name ti is integral to Airflow's design and is used throughout the codebase.

[HAEKADI]

@XavierJp The current implementation of the first_exec var logic is obviously not the most optimal, but the fastest in my opinion. The goal is to ensure that the most recent file from MinIO is retrieved once (first iteration in loop) during the workflow. However, if this logic is removed, during each iteration (meaning for every day of the execution), the workflow will attempt to retrieve the latest file saved to MinIO. This leads to unnecessary calls to both MinIO and the RNE API, resulting in inefficiencies.