🐛Correctly handle file type of on-demand clusters

[sanderegg]

What do these changes do?

This PR fixes the issue where using on-demand clusters would not properly use S3 file types but instead were locked on presigned links.
This had the consequence of limiting uploads to max 5Gib, but is also the sad fact that we still do not properly fixed it by using the public API from the dask-sidecars.

Related issue/s

How to test

Use a python runner and run the following script:

import os


def generate_large_file(file_path, size_gb):
    with open(file_path, "wb") as f:
        # Generate random data
        chunk_size = 1024 * 1024  # 1 MB chunks
        for _ in range(size_gb * 1024):
            data = os.urandom(chunk_size)
            f.write(data)


def create_file():
    # Example usage:
    desired_file_path = "/outputs/large_file.txt"
    desired_size_gb = 6  # Adjust this to your desired size in gigabytes

    generate_large_file(desired_file_path, desired_size_gb)

    print(f"Large file created: {desired_file_path}")


if __name__ == "__main__":
    create_file()

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

• hotfix to sim4life.io

[codecov]

Codecov Report

Merging #5101 (a6a9511) into master (00c54cd) will decrease coverage by 0.6%.
Report is 1 commits behind head on master.
The diff coverage is 100.0%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #5101     +/-   ##
========================================
- Coverage    70.6%   70.0%   -0.6%     
========================================
  Files         559     563      +4     
  Lines       27394   28596   +1202     
  Branches      207     198      -9     
========================================
+ Hits        19344   20029    +685     
- Misses       8000    8517    +517     
  Partials       50      50             

Flag	Coverage Δ
integrationtests	64.8% <100.0%> (-0.1%)	⬇️
unittests	86.3% <100.0%> (+20.9%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...e_service_director_v2/modules/dask_clients_pool.py	93.4% <100.0%> (+5.2%)	⬆️
.../storage/src/simcore_service_storage/exceptions.py	100.0% <100.0%> (ø)
...rage/src/simcore_service_storage/handlers_files.py	99.1% <100.0%> (ø)
...rage/src/simcore_service_storage/simcore_s3_dsm.py	94.3% <100.0%> (ø)

... and 207 files with indirect coverage changes

[bisgaard-itis]

Should this one be referenced here?

• https://github.com/orgs/ITISFoundation/projects/13?pane=issue&itemId=27816895

[bisgaard-itis]

Looks good. Thanks!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit a6a9511 and detected 0 issues on this pull request.

View more on Code Climate.