maintenance/scaling of the platform

[esraneufeld]

the scrum master wrote:
We are having quite a hard time maintaining the platformS, so that they pass our E2E tests [...]

in this sprint:

• [...] I would say we could say having a stable E2E with 20 users for this sprint. [...]

in the following ones:

• [...] then it should be recurring because we always have things to be done in maintenance. [...]

[sanderegg]

Update on sprint Red Panda

E2E

• added Jupyters e2e test running in parallel to simulate 20 users on master deploy, 10 users on staging AWS and production AWS deploys
• A number of issues were detected and fixed to improve the success rate of the e2e/p2p testing (see platform stability #1426 case)

Maintenance

• 3 data loss problems were detected:
  1. Jupyter lab state data was lost due to (un)archiving issue. Issue was fixed through #2192. Also DevOps change to ensure automated backups of S3 storage is done and also versioning which should prevent loosing data. Currently tested in on-premise master deploy, and will be moved to staging/production (on-premise and AWS) next sprint.
  2. A user opened a shared study and could not see the sharer user data in a Jupyter lab. After closing the study the empty data overwrote the original data, thus loosing the original Jupyter lab data. Issue was fixed through #2041.
  3. User A with a shared study opened got disconnected from the platform for a long time. User B worked on the shared study and closed it properly. User A eventually re-connected, this overwrote the data of user B. Issue was fixed through #2225
• All maintenance cases are listed in platform stability #1426. in the future will be connected to this issue.

NOTE: these fixes are currently all deployed in master/staging/staging_AWS. They will be deployed to production ASAP.

[sanderegg]

Update on sprint Schwarznasenschaf

Improvements

• Studies/template pagination #2268, #2273, #2292, #2298, to improve platform reactivity
• Optimize listing of services from backend #2313
• Allow cancelling request to list services #2302

Bugfixes

• Creating node link after selecting/moving node works again #2317
• Storage project copy failing #2301, #2295
• Notebook service cannot start #117

Testing facilities

• Integration of locust tool for load testing #2314, #2301

E2E

• Bugfixes in E2E scripts #2311, #2299, #2294
• Added studies cleaner script to remove studies left over after failed E2Es 2312

Maintenance

• Security updates #2307
• Upgrade testing and tooling requirements #2291
  NOTE: these fixes are currently all deployed in master/staging/staging_AWS. They will be deployed to production ASAP.

[sanderegg]

Update on sprint Chinchilla

Improvements

• Added static-webserver backend service, improves static website responsivity #2342

Bugfixes

• Revision of fastapi-based backend services startup #2356
• Fix slow blocking calls at startup #2346
• Passing variable to computation services non global #2330, #2316
• Faster access to S3 storage #2329
• Webserver background task auto-restarts when database connection is invalidated #2246

Testing facilities

• Improvements of performance testing scripts #2347), #2339, #2338, #2337, #2335, #2365

E2E

• Added test for guest study dispatcher #2362

Maintenance

• Error improvements, cleanup: #2355, #2304
• Simplify Github CI scripts: #2353
• Improved logging of blocking calls in webserver: 2341
• Libraries upgrades #2343, #2345, #2340, #2319, 2327, #2326
• improve maintenance scripts #2328

[sanderegg]

Update on sprint Marmoset

Improvements

• Improved service configuration approach
  • New settings-library #2395
  • Updates on settings in storage #2369
• Re-enabled DAT-Core - Pennsieve access since name change #2391

Bug fixes

• Fixes session cookies with latest aiohttp library #2386
• Cannot link files from File Picker to some cardiac models [cannot link files from File Picker to some cardiac model nodes #442]
• User with full access to study renders as viewer [#2272]
• Project locked issue #2381, [A user cannot re-open his/her study when he/she closes a tab inadvertently #431]
• fix blocking call to celery backend #2372
• fix blocking calls in webserver #2397

Changed

• Upgrade to python 3.8.10 in all osparc-simcore services #2079
• Libraries updates #2394, #2367
• Cap unarchiving workers to 2 #2384
• Improved error handling in frontend #2283

Testing facilities

• Upgrades of e2e libraries #2352
• Upgrades of tests linked to python 3.8 #2387

Open issue / ongoing

• webserver responsiveness issue (503s)

[sanderegg]

Update on sprint Wombat

Improvements

Bug fixes

• Fixes running computational task not aborted #2449
• Fixes dynamic-sidecar settings in director-v2 #2431
• Fixes API upload timeout too short #2433
• Fixes timeout for synchronisation of storage metadata #2420

Changed

• Webserver responsiveness issues: disabling unused metrics with high cardinality #2452
• Improve log messages #2430
• Director-v2 uses settings-library #2427
• Replaced auto-generated internal storage REST API client #2578

Testing facilities

• Improvements of Github-CI workflow #2430, #2450, #2457, #2460, #2462

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• test to check metrics endpoints #2417
• improving CI workflow speed #2466
• storage service refactoring #2396
• webserver service refactoring #2008

[sanderegg]

Update on sprint Chevrotain

Improvements

Bug fixes

Changed

• Refactor of service library (separation of concerns) #2516
• Upgrade of library, services dependencies #2485, #2524

Testing facilities

• Added data consistency scripts #2531, #2533, #2535
• Improve Github CI workflow #2510
• Upgrade testing and tooling dependencies #2475

3rd party services

• Fix dependencies in Mattward viewer #137

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• test to check metrics endpoints #2417
• improving CI workflow speed #2466
• storage service refactoring #2396
• webserver service refactoring #2008

[sanderegg]

Update on sprint Capra delle nevi

Bug fixes

• Frontend: improve logging when failure to retrieve data in service occurs #2574
• Frontend: Copy thumbnail only if it exists #2548
• webserver/storage: Fail to copy studies with a lot of data #2542
• webserver: Handle error when webserver tries to disconnect an already disconnected websocket #2551
• Frontend: Shared guided mode settings in template #2561

Changed

• Refactoring webserver module "director-v2" and remove cycling dependencies #2567
• Updated settings practices in api-server #2563
• Refactoring servicelib #2550
• Upgrade of testing&tooling dependencies #2547
• Maintenance of dependencies #2545

Testing facilities

• New scripts to validate project database tables #2550

Open issue / ongoing

• Adding tracing in fastapi-based services #2558
• Adding tracing in aiohttp-based services #2559
• Adding more logs from pending services #2566
• Refactoring webserver settings #2376
• webserver responsiveness issue (503s) under observation
• improving CI workflow speed #2466, #2525
• storage service refactoring #2396
• webserver service refactoring #2008
• service-integration backlog 2409

[sanderegg]

Update on sprint Anti-PER

Bug fixes

• Improve CI reliability/stability #2626, #2623, #2624, #2616, #2609, #2600
• Use director-v0 expected signature #2619
• Fixes unhandled exception when parsing invalid string #2608
• Divers fixes (codestyle, openapi, dockerignore) #2593

E2E testing

• Fix 3D e2e after design changes #2601

Changed

• Ensure director-v2 auto-restart only triggers when src code folder changes #2598, #2596
• Adding tracing in aiohttp-based services #2559
• Adding more logs from pending services #2566

Open issue / ongoing

• Adding tracing in fastapi-based services #2558
• Refactoring webserver settings #2376
• webserver responsiveness issue (503s) under observation
• improving CI workflow speed #2466, #2525
• storage service refactoring #2396
• webserver service refactoring #2008
• service-integration backlog 2409

[sanderegg]

Update on sprint Meerkat

New

• Script for listing repo contents #2639
• New Handling of staging hotfixes #2649
• Add config for github automatic changelog generation upon release #2662

Bug fixes

• Properly handle timeouts when copying a project #2655
• Fixes makefile recipe for hotfix-releases #2650

Deployment

• Fix Storage lazy update of files #2636

Changed

• Adding tracing in fastapi-based services #2558
• New interfaces for webserver.director_v2 plugin #2647
• Maintenance of libraries dependencies and repo tooling #2660, #2663, #2679, #2669, #2676, #2664
• Remove overload of pytest loop fixture: #2674
• Github issue template reworked: #2673

Open issue / ongoing

• Refactoring webserver settings #2376
• webserver responsiveness issue (503s) under observation
• improving CI workflow speed #2466, #2525
• storage service refactoring #2396
• webserver service refactoring #2008
• service-integration backlog 2409

[sanderegg]

Update on sprint Rudolph

New

Bug fixes

• Fixes metrics high cardinality in webserver, storage and director-v0 services ♻️ Improve metrics generation osparc-simcore#2726

Changed

• Refactoring webserver settings ♻️ Enhances BaseCustomSettings in settings_library osparc-simcore#2750:

  • ♻️ Maintenance/new settings (2nd round) osparc-simcore#2739
  • ♻️ Maintenance/new settings (1st round) osparc-simcore#2736

• Libraries upgrades

  • ⬆️ upgrades pydantic==1.9.0 repo-wide osparc-simcore#2752
  • ⬆️ uvicorn upgrade osparc-simcore#2748
  • ⬆️ upgrades dependencies osparc-simcore#2733
  • 📌 Maintenance/webserver requirements osparc-simcore#2741
  • ♻️ Completely remove celery osparc-simcore#2730
  • Bump celery from 5.2.1 to 5.2.2 in /ci/helpers osparc-simcore#2727

• Improved reverse proxy rules to only take care of the minimal necessary instead of defaulting to webserver

  • ♻️ Traefik rules: final Path needed for viewing files osparc-simcore#2737
  • ♻️ set traefik rules to the minimal that allows entrypoints to go through osparc-simcore#2734,
  • ✨ Performance testing: allow for auth calls osparc-simcore#2732
  • ✨ Improve Traefik rules (⚠️ devops) osparc-simcore#2710

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• Garbage collector seems to have an impact on webserver responsiveness osparc-simcore#2757
• services with uvicorn do not auto reload in development mode osparc-simcore#2765
• webserver shows write is none when restarted while connecting to RabbitMQ osparc-simcore#2758
• improving CI workflow speed:
  • ♻️ Maintenance/reduce num unittests osparc-simcore#2525
• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• service integration (or service onboarding) backlog osparc-simcore#2409

[mguidon]

@sanderegg @pcrespov These bullet lists are priceless for the quarterly reports! Thanks a lot for updating them.

[sanderegg]

Update on sprint R. Schumann

New

• Iframe lazy loading osparc-simcore#2790
• Show "please wait" message when Nature services are starting osparc-simcore#2801

Bug fixes

• 🐛 adds exception rules for legacy services (⚠️ devops) osparc-simcore#2813
• 🐛 Fixes wrong signature and adds tests osparc-simcore#2812
• ⚗️ Reject saving project while pipeline is running osparc-simcore#2797
• Fix: Search filters osparc-simcore#2804
• Fix: service catalog search filter osparc-simcore#2821
• Lock links when pipeline is running: frontend osparc-simcore#2795
• 🐛 #2758 exclusive queue cannot be reacquired osparc-simcore#2780
• services with uvicorn do not auto reload in development mode osparc-simcore#2765

Changed

• Frontend:

  • Few frontend optimizations osparc-simcore#2823
  • Enable/disable Export and Duplicate osparc-simcore#2820
  • Show "please wait" message when Nature services are starting osparc-simcore#2801

• Computational backend improvements:

  • Logs are missing in frontend when dask-scheduler is restarted osparc-simcore#2787
  • ♻️✨ Improve logging output in case of error when scheduling tasks osparc-simcore#2842

• Refactoring webserver settings ♻️ Enhances BaseCustomSettings in settings_library osparc-simcore#2750:

  • ♻️✨ ⚠️ [DEVOPS] maintenance new settings (final round) osparc-simcore#2836
  • 🐛 Fix/setting exclude unset option osparc-simcore#2846
  • ♻️ gc as a service (preparation) osparc-simcore#2828
  • ✨ Is2805/can disable exporter with envs (⚠️ devops)  osparc-simcore#2814

• Libraries upgrades

  • WIP: ⬆️ vulnerability upgrade ujson -> fastapi osparc-simcore#2762
  • ⬆️ Fixes test&tooling requirements in devel mode osparc-simcore#2832
  • ⬆️ Maintenance/week 05: test and tooling libs osparc-simcore#2809

• Services upgrade:

  • ⬆️ upgrade redis to 6.2.6, added persistency ⚠️ devops osparc-simcore#2793

• E2E:

  • e2e fix: Wait for iframe after opening node osparc-simcore#2802
  • e2e: log 503s osparc-simcore#2775

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• service integration (or service onboarding) backlog osparc-simcore#2409
• 🐛♻️ Deleting a project now removes its files osparc-simcore#2777
• 🐛♻️ Bugfix/handle ever running tasks - refactor director-v2 workflow scheduler osparc-simcore#2798
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• 🔒️ Adds codeql-analysis cron job osparc-simcore#2826
• ⬆️ Upgrades api-server requirements osparc-simcore#2827
• 🐛 dynamic-sidecar permission fixing osparc-simcore#2847

[sanderegg]

Update on sprint E. Schackleton

View of this issue on Zehnhub March1 - Apr 3

New

• Security: 🔒️ Adds codeql-analysis cron job osparc-simcore#2826
• ✨ Cache list service request in catalog osparc-simcore#2874

Bug fixes

• 🐛 fixes concurrency issue with directory-watcher osparc-simcore#2932
• 🐛 Fix email sending issue missing "From" field osparc-simcore#2926
• 🐛 Avoids locking the dynamic-sidecar in an error state osparc-simcore#2928
• 🐛 graceful stop for dynamic-sidecar started services osparc-simcore#2879
• webserver blocked by unidentified reason osparc-simcore#2399
• 🐛Fix save state for non-responsive services osparc-simcore#2883
• 🐛 Fixed email sending osparc-simcore#2875
• computational service stays in "Running" state osparc-simcore#2786
• 🐛 #2923 - sharing not possible osparc-simcore#2935
• 🐛 dynamic-sidecar permission fixing osparc-simcore#2847
• Fix wrong git sha osparc-deployment-agent#41

Changed

• Webserver refactoring
  • ♻️ Refactors webserver's healthcheck osparc-simcore#2910
  • Garbage collector can be run as a separate service: Garbage collector seems to have an impact on webserver responsiveness osparc-simcore#2757, 🚑️ GC hotfix: fixes dependency with storage plugin and new healthcheck osparc-simcore#2906, 🚑️ GC hotfix: fixes bad setup osparc-simcore#2905, 🚑️ GC needs dependency to login storage osparc-simcore#2904, 🚑️ hotfix GC: adds missing plugin dependency osparc-simcore#2901, 🚑️ hotfix GC first deploy osparc-simcore#2899, ♻️ Is2757/gc as a service implementation (⚠️⚠️ devops) osparc-simcore#2834
• ♻️ fully deprecates models_library.settings and removes most of its submodules osparc-simcore#2895
• ♻️ Replace aiopg in catalog service osparc-simcore#2869
• 🗑️ Maintenance/ cleanup unsued files osparc-simcore#2855
• Libraries upgrades
  • ⬆️ Maintenance/week 12 test, tooling, libs/packages osparc-simcore#2916
  • 🔨 Maintenance/tools and non-critical fixes osparc-simcore#2902
  • Bump log4js from 6.3.0 to 6.4.0 in /tests/e2e osparc-simcore#2893
  • ⬆️ Maintenance/upgrade pip osparc-simcore#2887
  • ⬆️ Maintenance/upgrade dask libraries osparc-simcore#2890
  • ⬆️ Maintenance upgrade sqlalchemy osparc-simcore#2864
  • ⬆️ Upgrades api-server requirements osparc-simcore#2827
  • Bump pylint from 2.12.2 to 2.13.2 osparc-deployment-agent#42
  • Bump faker from 13.3.2 to 13.3.3 osparc-deployment-agent#43
• Deployment
  • ♻️ Enhance regex validation director v2 - Allow registries by IP and Port osparc-simcore#2892
  • release Github workflow process fail when there are branches left in upstream osparc-simcore#2888
• CI
  • Bump actions/cache from 2 to 3 osparc-simcore#2907
  • ♻️ Maintenance: reduce number of webserver CI jobs osparc-simcore#2900
  • Bump actions/checkout from 2 to 3 osparc-simcore#2873
  • Bump pip-tools from 6.5.0 to 6.5.1 osparc-deployment-agent#27
  • Bump actions/cache from 2 to 3 osparc-deployment-agent#40
  • Bump pytest from 7.1.0 to 7.1.1 osparc-deployment-agent#39
  • Bump actions/setup-python from 2 to 3 osparc-deployment-agent#35
  • Bump actions/checkout from 2 to 3 osparc-deployment-agent#34
  • Bump actions/cache from 2 to 3 osparc-python-runner#41
  • Bump pytest from 7.1.0 to 7.1.1 osparc-python-runner#40
  • Bump actions/setup-python from 2 to 3 osparc-python-runner#36
  • Bump pytest from 7.0.1 to 7.1.0 osparc-python-runner#39
  • Bump actions/checkout from 2 to 3 osparc-dask-gateway#27
  • Bump actions/cache from 2 to 3 osparc-dask-gateway#28

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• 🐛♻️ Deleting a project now removes its files osparc-simcore#2777
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• ♻️ Maintenance: refactors guest user and save-state logic to improve flaky tests osparc-simcore#2897
• ⬆️ Maintenance/update aioredis osparc-simcore#2898
• If there is a space in the folder where the requirements.txt file is located the service fail osparc-python-runner#34

[sanderegg]

Update on sprint Macarons

View of this issue on Zenhub March1 - Apr 3

Bug fixes

• 🐛 Cluster credentials passed as ***** osparc-simcore#2937
• opening a running project shows an error message in the workbench osparc-simcore#2947
• 🐛 persist dynamic-sidecar state to docker service labels osparc-simcore#2961
• ♻️🐛 docker engine cleans up volumes when removing dynamic-sidecars osparc-simcore#2915
• 🐛fix CSP frame-ancestors for dynamic-sidecar services osparc-simcore#2977
• 🐛dynamic-sidecar fixes osparc-simcore#2990

Changed

• Repository wide upgrade of Python to 3.9.12: ⬆️ Maintenance: upgrade to python 3.9.12 osparc-simcore#2973
• Libraries upgrades
  • ⬆️ upgrade pyyaml due to vulnerability osparc-simcore#2960
  • ⬆️ upgrades urllib3==1.26.5 to fix CVE-2021-33503 osparc-simcore#2963
  • ⬆️ udpates datcore-adapter service dependencies osparc-simcore#2964
  • ⬆️ Maintenance: update dask dependencies osparc-simcore#2968
• Storage refactoring
  • ♻️ Maintenance/remove pytest docker storage osparc-simcore#3011

Open issue / ongoing

• webserver responsiveness issue (503s) under observation
• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• Common functionality for aiohttp and httpx clients osparc-simcore#2156
• Use the same port for both aiohttp and fastapi apps osparc-simcore#2759
• Remove aiozipkin library and go for OpenTelemetry osparc-simcore#2715
• review Project entrypoint in webserver osparc-simcore#2980
• Catalog KeyError: Caching issues? - portal links fail osparc-simcore#3005
• Director-v2 fails to parse dask-job (IndexError: list index out of range) osparc-simcore#3000
• Dynamic Services are shut down in an open study when second tab with a second study is closed osparc-simcore#2994
• Gunicorn needs --timeout=90 to properly start webserver osparc-simcore#2991
• 5Gb limit #617

[sanderegg]

Update on sprint Croissant

Done

• 5Gib limit:

  • ✨Allow computational sidecar to understand S3 links for download/upload osparc-simcore#2997
  • ✨Storage can return a S3 link instead of a presigned link on demand osparc-simcore#3004
  • ✨ Use S3 links in internal computational backend cluster, prepare for temporary tokens (⚠️ devops) osparc-simcore#3006
  • ✨♻️ (⚠️ devops) Remove 5gb limit when uploading data via nodeports osparc-simcore#2993

• Libraries updates:

  • ⬆️ Maintenance/test and tools [week 19] osparc-simcore#3031
  • ⬆️ Maintenance: upgrade dependencies of webserver service osparc-simcore#3038
  • ⬆️ maintenance/ updates pip-tools repo-wide osparc-simcore#3075

• Others:

  • 🐛 CI: properly get original git branch name osparc-simcore#2970
  • ♻️ Maintenance/remove pytest docker storage osparc-simcore#3011
  • ♻️ maintenance: fixes scicrunch tests and set as integration tests osparc-simcore#3018
  • (⚠️ devops) 🐛 Adding missing custom constraints osparc-simcore#3017
  • Maintenance: test with scicrunch AddexBIo fails osparc-simcore#3043
    - Maintenance: old services are using outdated Docker REST API, or wrong osparc-simcore#3044
  • 🐛🔨 Maintenance: fixes log-level of server in boot and bumpversion commit failure osparc-simcore#3070
  • 🐛 Started dynamic services do not take the correct Reservation osparc-simcore#3047
  • Catalog KeyError: Caching issues? - portal links fail osparc-simcore#3005
  • 🐛 Fix file-picker downstream service notification issues osparc-simcore#3058

• E2E:

  • Use jupyterlabs sim4life in the e2e osparc-simcore#3068
  • e2e: test sim4life-dy osparc-simcore#3060

• Dynamic-sidecar:

  • 🐛 Fixes issues with dy-sidecar removal by director-v2 osparc-simcore#3029
  • 🐛 Fix issues with parsing S3Settings osparc-simcore#3042
  • 🚑️ tuning down scheduler for dynamic sidecars osparc-simcore#3025
  • 🐛 Fix storage not starting with regular https addresses osparc-simcore#3023
  • 🐛 fixed SimcoreServiceSettingLabelEntry parsing osparc-simcore#3052
  • ♻️ Maintenance/rclone flaky test osparc-simcore#3050

Ongoing

• ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021
• webserver responsiveness issue (503s) under observation
• ✨add image prepulling to all swarm nodes osparc-simcore#3073
• WIP: ♻️ Is560/clone projects osparc-simcore#2974

Open

• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• Common functionality for aiohttp and httpx clients osparc-simcore#2156
• Use the same port for both aiohttp and fastapi apps osparc-simcore#2759
• Remove aiozipkin library and go for OpenTelemetry osparc-simcore#2715
• review Project entrypoint in webserver osparc-simcore#2980
• Catalog KeyError: Caching issues? - portal links fail osparc-simcore#3005
• Director-v2 fails to parse dask-job (IndexError: list index out of range) osparc-simcore#3000
• Dynamic Services are shut down in an open study when second tab with a second study is closed osparc-simcore#2994
• Gunicorn needs --timeout=90 to properly start webserver osparc-simcore#2991
• 5Gb limit #617
• remove the MPI flag for computational services osparc-simcore#3078
• Define folders in Storage osparc-simcore#3033

[sanderegg]

Update on sprint Diolkos

Done

• Storage refactoring/5Gib limit:
  ♻️ Maintenance: remove node ports v1 osparc-simcore#3098
  Enhancement/storage refactoring step2 osparc-simcore#3100
  ✨♻️ Storage refactoring step3 osparc-simcore#3144

• Libraries updates:
  ⬆️ Maintenance: fixes downstream upgrades and reduces version dispersion osparc-simcore#3092
  ⬆️🔒️ Maintenance/vulnerability upgrade for ujson, upgrade fastapi+starlette osparc-simcore#3112
  ⬆️ Maintenance: upgrade dask libraries osparc-simcore#3135

• CI:
  ♻️ Maintenance: Removed deprecation warnings and marks flaky tests osparc-simcore#3085
  🐛 Fixes failing test in master osparc-simcore#3094
  🚑️ CI: cache in setup-python failing osparc-simcore#3097
  ♻️ Maintenance: renamed travis tests to heavy load osparc-simcore#3099
  ♻️ Maintenance: Re-enable caching in github CI workflow osparc-simcore#3124

• Others:
  🐛 Dy-Sidecar fix use wait_for instead of async_timeout for running subtasks osparc-simcore#3129
  🐛 ♻️ Is3125/gc review osparc-simcore#3130
  🔨 Maintenance/pyupgrade code base targeting py3.9 osparc-simcore#3122
  🐛 Fixes FileLink validation legacy format osparc-simcore#3114

Ongoing

• Dy-sidecar case: improving dynamic-sidecar design #638
• 5Gib case: ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021
• ✨add image prepulling to all swarm nodes osparc-simcore#3073
• ⬆️🔒️ Maintenance/vulnerability upgrade for ujson, upgrade fastapi+starlette osparc-simcore#3112

Open

• computational services are always restarting osparc-simcore#3137
• investigate if lock stays locked for long times osparc-simcore#3145
• SetuptoolsDeprecationWarning: Upgrade setup to PEP517 osparc-simcore#3151
• storage refactoring backlog osparc-simcore#2396)
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• Common functionality for aiohttp and httpx clients osparc-simcore#2156
• Use the same port for both aiohttp and fastapi apps osparc-simcore#2759
• Remove aiozipkin library and go for OpenTelemetry osparc-simcore#2715
• review Project entrypoint in webserver osparc-simcore#2980
• Catalog KeyError: Caching issues? - portal links fail osparc-simcore#3005
• Director-v2 fails to parse dask-job (IndexError: list index out of range) osparc-simcore#3000
• Dynamic Services are shut down in an open study when second tab with a second study is closed osparc-simcore#2994
• Gunicorn needs --timeout=90 to properly start webserver osparc-simcore#2991
• 5Gb limit #617
• remove the MPI flag for computational services osparc-simcore#3078
• Define folders in Storage osparc-simcore#3033

[sanderegg]

Update on sprint Diolkos

Done

• Storage refactoring/5Gib limit:
  • ✨ Storage: allow creation of multipart presigned upload links osparc-simcore#3021
• Libraries updates:
• CI:
  • Bump actions/setup-node from 3.4.0 to 3.4.1 osparc-simcore#3196
  • Bump actions/setup-node from 3.3.0 to 3.4.0 osparc-simcore#3180
• Others:
  • computational services are always restarting osparc-simcore#3137
  • 🔨 Maintenance: fixes requirement listings and scripts osparc-simcore#3184
  • ♻️ Is638/webserver's project and directorv2 plugins refactoring osparc-simcore#3189
  • 🐛 Dask dashboard not showing osparc-simcore#3214
  • 🐛Bugfix: Director v2 called from GC errors because of wrong use of yarl library osparc-simcore#3207
  • ✨ Is3200/log user during login/logout osparc-simcore#3205
  • 🐛 (⚠️ devops) Fix validation of DIRECTOR_V2_SERVICES_CUSTOM_CONSTRAINTS: Make it conform to docker specs osparc-simcore#3190
  • ✨adding way to start/track/resolve background tasks for FastAPI based services osparc-simcore#3174

Ongoing

• Dy-sidecar case: improving dynamic-sidecar design #638
• 5Gb limit #617
  • api-server: uploaded files are temporarily located in the server osparc-simcore#3223

Open

• ✨add image prepulling to all swarm nodes osparc-simcore#3073
• ⬆️🔒️ Maintenance/vulnerability upgrade for ujson, upgrade fastapi+starlette osparc-simcore#3112
• investigate if lock stays locked for long times osparc-simcore#3145
• SetuptoolsDeprecationWarning: Upgrade setup to PEP517 osparc-simcore#3151
• storage refactoring backlog osparc-simcore#2396
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• Common functionality for aiohttp and httpx clients osparc-simcore#2156
• Use the same port for both aiohttp and fastapi apps osparc-simcore#2759
• Remove aiozipkin library and go for OpenTelemetry osparc-simcore#2715
• review Project entrypoint in webserver osparc-simcore#2980
• Director-v2 fails to parse dask-job (IndexError: list index out of range) osparc-simcore#3000
• Dynamic Services are shut down in an open study when second tab with a second study is closed osparc-simcore#2994
• Gunicorn needs --timeout=90 to properly start webserver osparc-simcore#2991
• remove the MPI flag for computational services osparc-simcore#3078
• Define folders in Storage osparc-simcore#3033

[sanderegg]

Update on sprint Brutalism

Done

• Libraries updates:
  • ⬆️ maintenance/updates pennsieve library osparc-simcore#3246
  • ⬆️ Maintenance/week 32: upgrades test and tool libraries osparc-simcore#3250
• CI:
  • 🐛 fixed flaky director-v2 test osparc-simcore#3252
• Others:
  • ✨ Long running tasks: allow retrieving tasks by user osparc-simcore#3268
• data-integrity: ✨ Maintenance/upgrade maintenance script osparc-simcore#3227

Ongoing

• Fix duplicate feature for large studies #558
• improving dynamic-sidecar design #638
• 5Gb limit #617

Open

• ✨add image prepulling to all swarm nodes osparc-simcore#3073
• ⬆️🔒️ Maintenance/vulnerability upgrade for ujson, upgrade fastapi+starlette osparc-simcore#3112
• investigate if lock stays locked for long times osparc-simcore#3145
• SetuptoolsDeprecationWarning: Upgrade setup to PEP517 osparc-simcore#3151
• ⚗️ web-server refactoring backlog osparc-simcore#2008
• ✨Adds timestamps to sidecars (dynamic and computational) osparc-simcore#2817
• Common functionality for aiohttp and httpx clients osparc-simcore#2156
• Use the same port for both aiohttp and fastapi apps osparc-simcore#2759
• Remove aiozipkin library and go for OpenTelemetry osparc-simcore#2715
• review Project entrypoint in webserver osparc-simcore#2980
• Director-v2 fails to parse dask-job (IndexError: list index out of range) osparc-simcore#3000
• Dynamic Services are shut down in an open study when second tab with a second study is closed osparc-simcore#2994
• Gunicorn needs --timeout=90 to properly start webserver osparc-simcore#2991
• remove the MPI flag for computational services osparc-simcore#3078
• Define folders in Storage osparc-simcore#3033
• maintenance: long running tasks for FastAPI-based app needs upgrade osparc-simcore#3265

[elisabettai]

Closing this one, since we have a new one for y6 (#675)