🚀 Release v1.65.0

[YuryHrytsuk]

Release version

1.65.0

Commit SHA

TBD

Previous pre-release

https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_ThisIsSparta6

Did the commit CI suceeded?

• The commit CI succeeded.

Motivation

• Regular sprint prod release

Changes

• 🎨Improving computational clusters cli monitoring tool #5204 by @sanderegg
• 🐛Clusters-keeper: calling >100 times for the same new cluster now returns cached information #5201 by @sanderegg
• 🔒️ Maintenance: Fixes issues detected by code scanning #5207 by @pcrespov
• 🎨 reroute dynamic service stop via dynamic-scheduler #5192 by @GitHK
• 🔒️⬆️ Maintenance: GHSA-v8gr-m533-ghj9 #5211 by @pcrespov
• 🐛 FrontEnd: Fixed misleading cursor for deleting study #5213 by @jsaq007
• 🐛 Fixes registration in multiple products via invitations #5055 by @pcrespov
• ✨E2E: add playwright sleepers test in CI #5177 by @sanderegg
• ✨Computational backend: add helpful tracing log #5218 by @sanderegg
• 🐛 Bugfix/e2e sim4life test #5219 by @matusdrobuliak66
• 🔨 Ensures all active users have at least a product's group (🗃️) #5222 by @pcrespov
• ✨🐛♻️ Billing center UI/UX #5198 by @ignapas
• ♻️ Adds start_exclusive_periodic_task derived from resource usage tracker service #5228 by @GitHK
• ✨ add metrics endpoints to fastapi apps for promethius scraping #5217 by @bisgaard-itis
• 🐛 Fixed unintentional interval swapping for start_exclusive_periodic_task #5235 by @GitHK
• 🎨 Update app view for better alignment to updated design #5221 by @jsaq007
• ♻️ Preparations for product-based user sessions🔒️⚠️🚨 #4947 by @pcrespov
• 🎨 Extends user table with username and first/second name for a user 🗃️ #5229 by @pcrespov
• 🐛🎨Computational autoscaling: allow multi-machining/processing #5203 by @sanderegg
• 🐛 fixes expiration date field #5243 by @pcrespov
• 🐛 Bumping timeouts #5232 by @GitHK
• ✨ Is922/add order by/filtering to billing endpoint #5223 by @matusdrobuliak66
• 🐛 Handle nullables for firstName lastName in me response #5244 #5247 by @jsaq007
• ♻️ docker compose reads compose spec file from stdin #5231 by @GitHK
• 🐛 webserver now replies with unknown service state #5230 by @GitHK
• ✨ added CLUSTERS_KEEPER_EC2_INSTANCES_PREFIX env to facilitate testing (⚠️ devops) #5251 by @sanderegg
• 🐛 datcore-adapter (Pennsieve API): filter out deleted files #5254 by @elisabettai
• 🐛Fixes TypeError: issubclass() arg 1 must be a class while building BaseCustomSettings classes #5253 by @pcrespov
• 🐛Fix infinite Waiting for cluster when dask-scheduler is restarted #5252 by @sanderegg
• ♻️ Cleanup of models-centric packages in preparation for upgrade #5256 by @pcrespov
• 🎨⚗️Comp backend/bugfix/work stealing (⚠️ devops) #5261 by @sanderegg
• ✨Dask backend: Enable /metrics endpoint for scraping with prometheus #5264 by @sanderegg
• 🐛 ✨ Check #credits >= 0.0 before returning solver job outputs #5249 by @bisgaard-itis
• ♻️/✨use placement constraints instead of generic resources ⚠️ #5255 by @GitHK
• 🔨 Enhances release monitor script #5242 by @pcrespov
• ✨ Is922/add csv export functionality (⚠️ devops) #5258 by @matusdrobuliak66
• 🐛 Is922/wrong-null-substitution #5269 by @matusdrobuliak66
• ✨ subscribe to socketio room based on the user_id #5270 by @GitHK
• ✨ New product-bound login: user must have access to target product 🚨 #5262 by @pcrespov
• 🎨 Reduces response time of catalog/services listing entrypoint #5273 by @pcrespov
• 🐛 Fix pull progress issues #5274 by @GitHK
• ✨Adds new Redis database #5278 by @GitHK
• ⬆️ pyinstrument upgrade, removed pin from webserver #5271 by @GitHK
• ✨ Adding ordering to list projects api #5276 by @matusdrobuliak66
• 🎨 Is922/adding filtering fields to usage endpoint #5277 by @matusdrobuliak66
• Bump dorny/paths-filter from 2 to 3 #5280 by @dependabot
• ✨ dynamic-sidecar actively monitors disk usage ⚠️ #5248 by @GitHK
• Secure Dask backend using self-signed certificates (⚠️ devops) #5265 by @sanderegg
• ✨ Ask user to upgrade patch versions for services #5257 by @jsaq007
• ♻️ fixes flaky test_time_overhead_on_handlers_of_auth_decorators #5279 by @pcrespov
• 🎨 Emit proper product name on socketio connection #5285 by @matusdrobuliak66
• 🎨 Clusters-keeper: No need to install AWS CLI, it is already in AMI #5287 by @sanderegg
• 🎨Maintenance: add dask security to clusters maintenance script #5288 by @sanderegg
• ✨ rabbit_client.subscribe no longer retries forever if message_handler always raises unexpected error ⚠️ #5282 by @GitHK
• 🐛Fixes internal server error in port matching 🚨 #5292 by @pcrespov
• ✨ added notes with actions to take if rabbitmq queues are no longer created #5295 by @GitHK
• 🎨 Is922/adding email field to usage endpoint #5291 by @matusdrobuliak66
• 🎨 UI: Changed title of error page  #5296 by @pcrespov
• 🎨 reduce noisy logs in director-v2 #5300 by @GitHK
• Renaming dynamic-scheduler in docker-compose files #5297 by @GitHK
• 🎨 Making service start progress UI user readable #5298 by @GitHK
• 🎨 Fixes flaky test_studies_dispatcher_studies_access #5304 by @pcrespov
• 🐛 Fixing broken CI builds #5305 by @GitHK
• ✨ dynamic-scheduler healthcheck now restart the service #5303 by @GitHK
• ✨ e2e/adding basic jupyterlab playwright test #5308 by @matusdrobuliak66
• 🐛 fish push step of CI #5311 by @GitHK
• ⚗️ Add DEBUG log messages in the callback the API-server passes to the RabbitMQ client #5312 by @bisgaard-itis
• ✨🐛♻️ Billing center #5272 by @ignapas

Before release ⚠️ DEVOPS

• Fill correct value to the RESOURCE_USAGE_TRACKER_S3 in osparc.io [MD]
  • @matusdrobuliak66 : It can stay as NULL value in production (CSV export functionality is currently only in billable products which are not there yet)
• Manually delete RabbitMQ queues if present (io.simcore.service.tracking, io.simcore.service.wallets, io.simcore.service.wallets-credit-limit-reached, simcore.services.instrumentation)
• Update docker compose plugin version (https://git.speag.com/oSparc/osparc-infra/-/merge_requests/202)
• This should be merged if appmotion gateway gets in: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/316
  • Create appmotion docker network
• this should be merged before releasing https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/338
• Force restart traefik (Fix appmotion certs osparc-ops-environments#557)
• Upload traefik certs
  • https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/jobs/4473483
  • https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/jobs/4473467

Dev Checks

• ⚠️ once released to osparc.io @GitHK and @mguidon need to check that the internet configuration works as expected (since this feature was refactored to use common code) (✨ dynamic-sidecar actively monitors disk usage ⚠️ #5248)

Devops check 👷

• Check independent maintenance page on NIH prod
• Check independent maintenance page on DALCO prod
• Check independent maintenance page on TIP prod
• on tip.itis.swiss dynamic-scheduler fails to start. That is why it was manually fixed to dynamic-schdlr. Be aware

After release ⚠️ DEVOPS

• Manually remove stake dynamic-scheduler service

Tests assessment: e2e testing check 🧪

No response

Test assessment: targeted-testing 🔍️

No response

Test assessment: user-testing 🧐

No response

Summary 📝

• Prepare release link

make release-prod version=<M.m.0>  git_sha=<commit_sha>

• Draft release changelog
• Announce maintenance ( ** ANNOUNCE AT LEAST 24 HOURS BEFORE ** )
• redis {"start": "2023-03-06T13:00:00.000Z", "end": "2023-03-06T15:00:00.000Z", "reason": "Release <vX.X.0>"}
  • aws
  • dalco
  • tip
• status page (https://manage.statuspage.io/)
  • osparc
  • s4l
• mattermost channels
  • maintenance
  • power users

Releasing 🚀

• Maintenance page up.

cd /deployment/production/osparc-ops-environments
make up-maintenance
make down-maintenance

• Release by publishing draft
• Check release CI
• Check hanging sidecars. Helper command to run in director-v2 CLI simcore-service-director-v2 close-and-save-service <uuid>
• Check deployed
  • aws deploy
  • dalco deploy
  • tip deploy
• Check testing endpoint ex. https://testing.osparc.speag.com/
• Delete announcement
• Check e2e runs
• Announce

:tada: https://github.com/ITISFoundation/osparc-simcore/releases/tag/v<M.m.0>

[GitHK]

Issues:

• On osparc.io had to reboot the GC in order for legacy services to be properly removed

[YuryHrytsuk]

ToDo

• Issue: dalco force restart traefik led to 1 replica. Needs to be investigated
• Update route 53 user creds in repo.config (nih prod)
• Stop internal maintenance page doesn't work (from CI)

PRs on-the-fly

• Fix s4l full domains Add s4l long domain to nih prod osparc-ops-environments#560
• Update adminer urls https://git.speag.com/oSparc/e2e-ops/-/merge_requests/207
• Fix nih prod certificates https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/380

Ind Maintenance Page report

• Dalco Ind Maintenance Page works 🆗
• TIP Ind Maintenance Page does not work 🛑
• NIH PROD Ind Maint Page requires clean TF state and cannot be tested atm. I also believe that TF for ind maintenance page needs to be updated 🛑

Infra

• TF changes were applied on nih prod. TF state is clean there

[YuryHrytsuk]

ToDo is moved to ITISFoundation/osparc-ops-environments#566