Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ adding background check for running services (⚠️ OPS) 🗃️ #4925

Merged
merged 17 commits into from
Nov 6, 2023
Merged

✨ adding background check for running services (⚠️ OPS) 🗃️ #4925

merged 17 commits into from
Nov 6, 2023

Conversation

matusdrobuliak66
Copy link
Contributor

@matusdrobuliak66 matusdrobuliak66 commented Oct 25, 2023 •
edited
Loading

What do these changes do?

  • ✨ adding background check for running services

Resource tracker will be checking every 5 minutes all running services. If there is a service for which RUT didn't receive a heartbeat in the last 5 minutes it will increase the missed heartbeat counter. If the counter reaches 6 it will consider the service as unhealthy.

  • 🗃️ Adding missed heartbeat counter column in the resource_tracker_service_runs table
  • BONUS: 🐛 fixes correctly filling project_name column (before it was product_name)

Related issue/s

How to test

cd service/resource-usage-tracker
make install-dev
pytest tests/unit/with_dbs/test_background_task.py

DevOps Checklist

New Resource Usage Tracker ENV vars ⚠️:

  • RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_CHECK_ENABLED=1
  • RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_INTERVAL_SEC=300
  • RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_COUNTER_FAIL=6

@matusdrobuliak66 matusdrobuliak66 self-assigned this Oct 25, 2023
@matusdrobuliak66 matusdrobuliak66 added this to the Microhistory milestone Oct 25, 2023
@codecov
Copy link

codecov bot commented Oct 25, 2023 •
edited
Loading

Codecov Report

Merging #4925 (d8c9687) into master (1f92ca7) will increase coverage by 0.6%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #4925     +/-   ##
========================================
+ Coverage    86.2%   86.8%   +0.6%     
========================================
  Files        1237    1207     -30     
  Lines       50897   50028    -869     
  Branches     1078    1078             
========================================
- Hits        43886   43466    -420     
+ Misses       6774    6325    -449     
  Partials      237     237
Flag Coverage Δ
integrationtests 65.1% <0.0%> (+4.1%) ⬆️
unittests 84.5% <100.0%> (-0.1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...s_database/models/resource_tracker_service_runs.py 100.0% <ø> (ø)
...vice_dynamic_sidecar/modules/long_running_tasks.py 95.2% <100.0%> (ø)

... and 56 files with indirect coverage changes

@matusdrobuliak66 matusdrobuliak66 marked this pull request as ready for review October 29, 2023 15:18
@matusdrobuliak66 matusdrobuliak66 changed the title WIP: ✨ adding background check for running services ✨ adding background check for running services (⚠️ OPS) 🗃️ Oct 29, 2023
pcrespov
pcrespov reviewed Oct 29, 2023
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! thx! Left a few comments and suggestions I would like you to check.

...ages/postgres-database/src/simcore_postgres_database/models/resource_tracker_service_runs.py Show resolved Hide resolved
services/resource-usage-tracker/src/simcore_service_resource_usage_tracker/core/settings.py Show resolved Hide resolved
services/resource-usage-tracker/tests/unit/with_dbs/test_background_task.py Outdated Show resolved Hide resolved
...acker/src/simcore_service_resource_usage_tracker/modules/db/repositories/resource_tracker.py Outdated Show resolved Hide resolved
services/resource-usage-tracker/src/simcore_service_resource_usage_tracker/resource_tracker.py Outdated Show resolved Hide resolved
services/resource-usage-tracker/tests/unit/with_dbs/test_background_task.py Show resolved Hide resolved
services/docker-compose.yml Show resolved Hide resolved
.env-devel Outdated Show resolved Hide resolved
sanderegg
sanderegg approved these changes Oct 29, 2023
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! some comments

..._postgres_database/migration/versions/22404057a50c_resource_tracker_service_runs_helpers_.py Outdated Show resolved Hide resolved
..._postgres_database/migration/versions/22404057a50c_resource_tracker_service_runs_helpers_.py Show resolved Hide resolved
...ages/postgres-database/src/simcore_postgres_database/models/resource_tracker_service_runs.py Show resolved Hide resolved
services/resource-usage-tracker/src/simcore_service_resource_usage_tracker/core/settings.py Show resolved Hide resolved
...e-tracker/src/simcore_service_resource_usage_tracker/models/resource_tracker_service_runs.py Outdated Show resolved Hide resolved
...e-tracker/src/simcore_service_resource_usage_tracker/models/resource_tracker_service_runs.py Outdated Show resolved Hide resolved
...usage-tracker/src/simcore_service_resource_usage_tracker/resource_tracker_background_task.py Outdated Show resolved Hide resolved
...usage-tracker/src/simcore_service_resource_usage_tracker/resource_tracker_background_task.py Show resolved Hide resolved
...sage-tracker/src/simcore_service_resource_usage_tracker/resource_tracker_process_messages.py Outdated Show resolved Hide resolved
GitHK
GitHK reviewed Oct 30, 2023
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions let's have a look together

...acker/src/simcore_service_resource_usage_tracker/modules/db/repositories/resource_tracker.py Outdated Show resolved Hide resolved
...sage-tracker/src/simcore_service_resource_usage_tracker/resource_tracker_process_messages.py Outdated Show resolved Hide resolved
pcrespov
pcrespov approved these changes Oct 30, 2023
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx!

@matusdrobuliak66 matusdrobuliak66 enabled auto-merge (squash) October 30, 2023 19:36
@matusdrobuliak66 matusdrobuliak66 enabled auto-merge (squash) November 5, 2023 16:58
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 6, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
18.2% 18.2% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@codeclimate CodeClimate
Copy link

codeclimate bot commented Nov 6, 2023

Code Climate has analyzed commit d8c9687 and detected 0 issues on this pull request.

View more on Code Climate.

@pcrespov pcrespov merged commit 73c5414 into ITISFoundation:master Nov 6, 2023
54 of 55 checks passed
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Nov 14, 2023
Closed
17 tasks
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Nov 23, 2023
Closed
29 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GitHK GitHK GitHK left review comments

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@sanderegg sanderegg sanderegg approved these changes

@pcrespov pcrespov pcrespov approved these changes

@mrnicegyu11 mrnicegyu11 Awaiting requested review from mrnicegyu11 mrnicegyu11 is a code owner

Assignees

@matusdrobuliak66 matusdrobuliak66

Labels
None yet
Projects
None yet
Milestone
Microhistory
Development

Successfully merging this pull request may close these issues.
5 participants
@matusdrobuliak66 @GitHK @pcrespov @sanderegg @YuryHrytsuk