Add a metric to collect the number of background routines #963
Conversation
WalkthroughThe recent changes enhance the tracking of background goroutines by modifying key methods to accept metrics and task type parameters. This allows for improved monitoring of background tasks via Prometheus, facilitating better management and observability of worker counts. Such updates ensure that metrics accurately reflect the state of goroutines, thus supporting proactive performance management. Changes
Assessment against linked issues
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Outside diff range, codebase verification and nitpick comments (4)
server/backend/background/background.go (2)
28-28: Consider Import Alias for Prometheus Package.To improve code readability and avoid potential conflicts, consider using an alias for the
prometheuspackage, especially if it is used frequently throughout the codebase.import prom "github.com/yorkie-team/yorkie/server/profiling/prometheus"
64-68: Update Calls to AttachGoroutine with New ParametersThe
AttachGoroutinemethod has been updated to includemetricsandtaskTypeas parameters. However, the call to this method inserver/packs/packs.godoes not pass these new parameters. Please update this call to include the necessary arguments.
- File:
server/packs/packs.go
- Line:
AttachGoroutinecallAnalysis chain
Ensure Proper Usage of New Parameters in AttachGoroutine.
The
AttachGoroutinemethod now acceptsmetricsandtaskType. Verify that all calls to this method throughout the codebase are updated to pass these parameters correctly.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all calls to `AttachGoroutine` are updated with new parameters. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type go -A 3 $'AttachGoroutine('Length of output: 121
Script:
#!/bin/bash # Description: Verify all calls to `AttachGoroutine` are updated with new parameters. # Test: Search for the function usage with corrected regex pattern. Expect: Only occurrences of the new signature. rg --type go -A 3 'AttachGoroutine\('Length of output: 567
server/profiling/prometheus/metrics.go (2)
39-39: Add Documentation fortaskTypeLabel.Consider adding a comment to document the purpose of
taskTypeLabelfor future maintainability.// taskTypeLabel is used to categorize metrics by task type.
65-66: Ensure Consistency in Metrics Naming.The naming of
backgroundGoroutinesTotalshould be consistent with other metrics. Consider adding a prefix or suffix if needed.// Consider renaming for consistency, e.g., "background_goroutines_total"
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- server/backend/background/background.go (3 hunks)
- server/packs/packs.go (1 hunks)
- server/profiling/prometheus/metrics.go (4 hunks)
Additional comments not posted (3)
server/backend/background/background.go (1)
83-86: Ensure Metrics are Updated on Goroutine Exit.The deferred function ensures that metrics are decremented when a goroutine exits. This is a good practice to maintain accurate metrics.
server/packs/packs.go (1)
207-207: Verify Metrics and Task Type Parameters in AttachGoroutine Call.Ensure that the
metricsand"pushpull"parameters passed toAttachGoroutineare correct and consistent with the intended monitoring logic.Verification successful
Parameters in AttachGoroutine Call Verified
The parameters
be.Metricsand"pushpull"are correctly used in theAttachGoroutinecall, consistent with the intended monitoring logic. No issues were found.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the correctness of metrics and task type parameters in `AttachGoroutine` calls. # Test: Search for the specific usage in the file. Expect: Correct parameters. rg --type go -A 3 $'AttachGoroutine(func(ctx context.Context) {'Length of output: 215
Script:
#!/bin/bash # Description: Verify the correctness of metrics and task type parameters in `AttachGoroutine` calls. # Test: Search for the specific usage in the file. Expect: Correct parameters. rg --type go -A 5 'AttachGoroutine\(func\(ctx context.Context\) \{' server/packs/packs.goLength of output: 268
Script:
#!/bin/bash # Description: Verify the correctness of metrics and task type parameters in `AttachGoroutine` calls. # Test: Search for the specific usage in the file with extended context lines. Expect: Correct parameters. rg --type go -A 10 'AttachGoroutine\(func\(ctx context.Context\) \{' server/packs/packs.goLength of output: 342
Script:
#!/bin/bash # Description: Verify the correctness of metrics and task type parameters in `AttachGoroutine` calls. # Test: Search for the specific usage in the file with more context lines. Expect: Correct parameters. rg --type go -A 15 -B 5 'AttachGoroutine\(func\(ctx context.Context\) \{' server/packs/packs.goLength of output: 636
Script:
#!/bin/bash # Description: Search for the specific parameters `be.Metrics` and `"pushpull"` in the file to locate their usage in `AttachGoroutine`. rg --type go 'be.Metrics' server/packs/packs.go rg --type go '"pushpull"' server/packs/packs.goLength of output: 595
server/profiling/prometheus/metrics.go (1)
140-145: Review Initialization ofbackgroundGoroutinesTotal.Ensure that the initialization of
backgroundGoroutinesTotalis correct and aligns with the intended metric collection strategy.Verification successful
Initialization of
backgroundGoroutinesTotalis correct.The
backgroundGoroutinesTotalis properly initialized usingpromauto.With(reg).NewGaugeVecwith the necessary options and labels, aligning with the intended metric collection strategy. No issues found.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the initialization of `backgroundGoroutinesTotal`. # Test: Check for correct initialization. Expect: Proper namespace and labels. rg --type go -A 5 $'backgroundGoroutinesTotal: promauto.With(reg).NewGaugeVec'Length of output: 79
Script:
#!/bin/bash # Description: Search for the initialization and usage of `backgroundGoroutinesTotal` in the specified file. # Locate the file `server/profiling/prometheus/metrics.go` fd 'metrics.go' --exec rg 'backgroundGoroutinesTotal' {}Length of output: 292
There was a problem hiding this comment.
Actionable comments posted: 0
Outside diff range, codebase verification and nitpick comments (1)
test/integration/retention_test.go (1)
48-53: Prometheus Metrics Integration Issue inAttachGoroutineThe
AttachGoroutinemethod inserver/packs/packs.godoes not currently pass Prometheus metrics, which are expected as per the updated method signature. This indicates a potential issue with the integration of metrics in the production code.
- File:
server/packs/packs.go- Issue: Prometheus metrics are not passed to
AttachGoroutine.Please review and ensure that the metrics are correctly integrated.
Analysis chain
Update patching function signature.
The patching function now includes Prometheus metrics and a string parameter, aligning with the updated method signature.
Ensure that the metrics are correctly integrated and used in the production code.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify integration of Prometheus metrics with the AttachGoroutine method. # Test: Search for the AttachGoroutine method usage. Expect: Correct integration with Prometheus metrics. rg --type go -A 5 $'AttachGoroutine'Length of output: 2038
Script:
#!/bin/bash # Description: Check the complete usage of AttachGoroutine in server/packs/packs.go to verify Prometheus metrics integration. # Search for the function definition and its usage in server/packs/packs.go rg --type go -A 10 'AttachGoroutine' server/packs/packs.goLength of output: 308
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- test/integration/retention_test.go (2 hunks)
- test/integration/snapshot_test.go (2 hunks)
Additional comments not posted (3)
test/integration/snapshot_test.go (2)
35-35: Import Prometheus metrics.The import of Prometheus metrics indicates integration with Prometheus for monitoring.
44-49: Update patching function signature.The patching function now includes Prometheus metrics and a string parameter, aligning with the updated method signature.
Ensure that the metrics are correctly integrated and used in the production code.
test/integration/retention_test.go (1)
39-39: Import Prometheus metrics.The import of Prometheus metrics indicates integration with Prometheus for monitoring.
There was a problem hiding this comment.
Thank you for your contribution.
I have left some comments below :)
PS. After we merge this PR, let's add new panel in Yorkie Grafana Dashbaord using the PromQL you mentioned in your description.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- server/backend/backend.go (1 hunks)
- server/backend/background/background.go (3 hunks)
- server/packs/packs.go (1 hunks)
Files skipped from review as they are similar to previous changes (2)
- server/backend/background/background.go
- server/packs/packs.go
Additional comments not posted (1)
server/backend/backend.go (1)
91-91: LGTM! Verify the usage of themetricsparameter.The introduction of the
metricsparameter in theNewfunction aligns with the PR objectives and enhances functionality.Ensure that the
metricsparameter is correctly utilized throughout the codebase and that all necessary parts of the system are updated to accommodate this change.Verification successful
The
metricsparameter is consistently utilized across the codebase.The
metricsparameter is integrated into various components, including server initialization, background tasks, and tests, ensuring that the system accommodates this change effectively.
- Files with
metricsusage:
server/backend/backend.goserver/profiling/server.goserver/profiling/prometheus/metrics.gotest/integration/housekeeping_test.gotest/bench/push_pull_bench_test.goThis confirms that the
metricsparameter is properly utilized throughout the codebase.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the usage of the `metrics` parameter in the codebase. # Test: Search for the usage of `metrics`. Expect: Proper usage and handling throughout the codebase. rg --type go 'metrics'Length of output: 2265
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- test/integration/retention_test.go (1 hunks)
- test/integration/snapshot_test.go (1 hunks)
Files skipped from review as they are similar to previous changes (2)
- test/integration/retention_test.go
- test/integration/snapshot_test.go
This PR adds a metric to collect the number of background routines. Prometheus provides its own thread-safe Gauge metric, which can be used to accurately collect the number of concurrent goroutines. Also, although the only task running in the background so far is PushPull, this commit implements the AttachGoroutine() method to take a string called taskType as a parameter, since it seems to be focused on reusability.
hackerwins
changed the title
|
@kokodak We have deployed this feature with Grafana panel added. 
|
|
@krapie This was happening to me locally, but when I forcibly increased the lifetime of the goroutine, I was able to observe it normally. How about changing the metric from Gauge to Counter, and changing it to observe the gradient of the number of goroutines increasing in time? |
|
@kokodak Good point! Looks like the lifespan of the goroutine is too short to collect.
cc. @hackerwins |
…m#963) This PR adds a metric to collect the number of background routines. Prometheus provides its own thread-safe Gauge metric, which can be used to accurately collect the number of concurrent goroutines. Also, although the only task running in the background so far is PushPull, this commit implements the AttachGoroutine() method to take a string called taskType as a parameter, since it seems to be focused on reusability.
This PR adds a metric to collect the number of background routines. Prometheus provides its own thread-safe Gauge metric, which can be used to accurately collect the number of concurrent goroutines. Also, although the only task running in the background so far is PushPull, this commit implements the AttachGoroutine() method to take a string called taskType as a parameter, since it seems to be focused on reusability.
What this PR does / why we need it:
This PR adds a metric to collect the number of goroutines attached to a specific Background task.
Prometheus provides its own thread-safe Gauge metric, which can be used to accurately collect the number of concurrent goroutines.
Also, although the only task running in the background so far is PushPull, I implemented the
AttachGoroutine()method to take a string calledtaskTypeas a parameter, since it seems to be focused on reusability.In my local Grafana, I observed the metrics with a simple promQL query like the one below.
The expected effect is that it will be easy to identify which background tasks are bottlenecking and when. For example, during the logic of saving snapshots, the number of observed goroutines may temporarily increase when the DB bottleneck causes each goroutine to have a longer lifecycle. This kind of analysis would help us prevent such issues.
Which issue(s) this PR fixes:
Fixes #386
Special notes for your reviewer:
If you think there's a better metric or promQL to get more meaningful data, please feel free to comment.
Does this PR introduce a user-facing change?:
Additional documentation:
Checklist:
Summary by CodeRabbit
Summary by CodeRabbit
New Features
Improvements
AttachGoroutine,PushPull, and other related functions.These changes significantly enhance the ability to manage and monitor background tasks within the application.