Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: manage pool directly in the plugin #135

Merged
merged 9 commits into from
Aug 30, 2024
Merged

Conversation

rustatian
Copy link
Member

@rustatian rustatian commented Aug 30, 2024

Reason for This PR

ref: roadrunner-server/roadrunner#1986

Description of Changes

  • Since endure v2 we don't need to have a central place to shutdown pools of workers. So every plugin can independently maintain its own pool.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

  • All commits in this PR are signed (git commit -s).
  • The reason for this PR is clearly provided (issue no. or explanation).
  • The description of changes is clear and encompassing.
  • Any required documentation changes (code and docs) are included in this PR.
  • Any user-facing changes are mentioned in CHANGELOG.md.
  • All added/changed functionality is tested.

Summary by CodeRabbit

  • New Features

    • Enhanced server shutdown process with a graceful termination method, improving resource management.
  • Dependency Updates

    • Upgraded various dependencies to newer versions for improved performance, stability, and security.
    • Updated Go version to 1.23, aligning with the latest features and improvements.
  • Bug Fixes

    • Improved detection of loop variable handling issues through updated linting strategy.
    • Streamlined configuration settings for gRPC service, potentially enhancing job handling.

@rustatian rustatian added the enhancement New feature or request label Aug 30, 2024
@rustatian rustatian requested a review from wolfy-j August 30, 2024 13:27
@rustatian rustatian self-assigned this Aug 30, 2024
Copy link

coderabbitai bot commented Aug 30, 2024

Walkthrough

The changes involve updates to configuration files and dependencies in a Go project. The .golangci.yml file has replaced the exportloopref linter with copyloopvar. The go.mod files across various modules have been updated to newer versions of dependencies and Go language specifications. Additionally, the Stop method in plugin.go has been modified to implement a more graceful server shutdown process, enhancing resource management.

Changes

File(s) Change Summary
.golangci.yml Replaced exportloopref linter with copyloopvar and removed gochecknoinits.
go.mod, protoc_plugins/go.mod, tests/go.mod Updated Go version and several dependencies to newer versions, including google.golang.org/grpc and github.com/prometheus/common.
plugin.go Modified Stop method to use GracefulStop() for server shutdown and added resource cleanup for gPool.
codec/codec.go, codec/codec_test.go Changed Base field type from encoding.Codec to encoding.CodecV2 and updated Marshal and Unmarshal methods for better error handling and memory management.
tests/grpc_plugin_gzip_test.go, tests/grpc_plugin_test.go Removed Prefix field from config.Plugin struct and ensured proper resource management by closing connections in test functions.
server.go Minor comment modification in serverOptions function for clarity.
tests/configs/.rr-grpc-rq-multiple.yaml Updated listening socket port from 9001 to 9003 and removed max_jobs parameter.

Sequence Diagram(s)

sequenceDiagram
    participant Plugin
    participant Server
    participant gPool

    Plugin->>Server: Stop()
    Server->>Server: GracefulStop()
    alt gPool is not nil
        Plugin->>gPool: Destroy(ctx)
    end
Loading

Poem

🐰 In the meadow, changes bloom,
With linters fresh, they chase the gloom.
Dependencies dance, all shiny and new,
A graceful stop, as the server bids adieu.
Hops of joy, let's celebrate,
For every change, we elevate! 🌼


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 1c866a0 and f49b6e3.

Files ignored due to path filters (3)
  • go.sum is excluded by !**/*.sum
  • go.work.sum is excluded by !**/*.sum
  • tests/go.sum is excluded by !**/*.sum
Files selected for processing (5)
  • .golangci.yml (1 hunks)
  • go.mod (2 hunks)
  • plugin.go (1 hunks)
  • protoc_plugins/go.mod (1 hunks)
  • tests/go.mod (6 hunks)
Files skipped from review due to trivial changes (1)
  • .golangci.yml
Additional comments not posted (12)
protoc_plugins/go.mod (2)

3-3: Approved: Go version update.

Updating the Go version to 1.23 aligns with best practices to utilize new features and improvements.


5-5: Approved: Toolchain version specified.

Specifying the toolchain version as go1.23.0 ensures consistency in the build environment, which is a good practice.

go.mod (3)

25-25: Approved: gRPC library update.

Updating google.golang.org/grpc to v1.66.0 is expected to bring improvements and bug fixes. Ensure compatibility with existing gRPC services.


40-40: Approved: Prometheus common library update.

Updating github.com/prometheus/common to v0.57.0 likely includes new features or improvements. Verify that this update integrates well with existing monitoring configurations.


24-24: Approved: Dependency update for RPC library.

Updating the google.golang.org/genproto/googleapis/rpc to a newer commit is generally beneficial. However, verify that this update does not adversely affect existing RPC implementations.

tests/go.mod (7)

3-5: Approved: Go version and toolchain update.

Updating the Go version to 1.23 and adding the toolchain go1.23.0 directive are good practices to ensure consistency and reliability in builds.


17-17: Approved: RoadRunner server dependency update.

Updating github.com/roadrunner-server/server/v5 to v5.1.0 is expected to bring new features or improvements. Ensure that this update does not introduce any regressions in server behavior.


21-21: Approved: gRPC library update in tests.

Updating google.golang.org/grpc to v1.66.0 in the tests module is expected to bring improvements and bug fixes. Ensure compatibility with existing gRPC tests.


111-111: Approved: API library update.

Updating google.golang.org/genproto/googleapis/api to a newer commit is generally beneficial. However, verify that this update does not adversely affect existing API implementations.


101-101: Approved: Temporal API dependency update.

Updating go.temporal.io/api to v1.39.0 is expected to bring improvements and bug fixes. Ensure compatibility with existing Temporal API integrations.

Verification successful

No direct usage of Temporal API found.

The go.temporal.io/api package is not directly referenced in the codebase or go.sum, indicating that the update to v1.39.0 should not impact the project directly. Manual verification is recommended if there are concerns about indirect dependencies.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for any compatibility issues with the Temporal API update.

# Test: Search for Temporal API related code to ensure compatibility. Expect: No compatibility issues.
rg --type go "go.temporal.io/api"

Length of output: 33


Script:

#!/bin/bash
# Description: Check for any references to the Temporal API in go.sum and indirect usage in the codebase.

# Search for references to the Temporal API in go.sum
rg "go.temporal.io/api" go.sum

# Search for any indirect usage in the codebase
rg "temporal" --type go

Length of output: 54


64-64: Approved: Prometheus common library update in tests.

Updating github.com/prometheus/common to v0.57.0 in the tests module likely includes new features or improvements. Verify that this update integrates well with existing monitoring configurations in tests.


112-112: Approved: RPC library update in tests.

Updating google.golang.org/genproto/googleapis/rpc to a newer commit in the tests module is generally beneficial. However, verify that this update does not adversely affect existing RPC implementations in tests.

plugin.go Show resolved Hide resolved
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 1c866a0 and f49b6e3.

Files ignored due to path filters (3)
  • go.sum is excluded by !**/*.sum
  • go.work.sum is excluded by !**/*.sum
  • tests/go.sum is excluded by !**/*.sum
Files selected for processing (5)
  • .golangci.yml (1 hunks)
  • go.mod (2 hunks)
  • plugin.go (1 hunks)
  • protoc_plugins/go.mod (1 hunks)
  • tests/go.mod (6 hunks)
Files skipped from review due to trivial changes (1)
  • .golangci.yml
Additional comments not posted (14)
protoc_plugins/go.mod (2)

3-3: Go version updated to 1.23.

The update to Go 1.23 is approved. Ensure to verify compatibility with existing code and dependencies.


5-5: Toolchain directive added for Go 1.23.0.

The addition of the toolchain go1.23.0 directive is approved as it promotes reproducible builds and consistency across development environments.

go.mod (3)

24-24: Updated google.golang.org/genproto/googleapis/rpc to a newer version.

The update to google.golang.org/genproto/googleapis/rpc is approved. Verify that this update does not affect any RPC interfaces or behaviors adversely.


25-25: Updated google.golang.org/grpc to v1.66.0.

The update to google.golang.org/grpc v1.66.0 is approved. Ensure thorough testing to verify no regressions in gRPC functionalities.


40-40: Updated github.com/prometheus/common to v0.57.0.

The update to github.com/prometheus/common v0.57.0 is approved. Verify that this update does not introduce any issues in metrics reporting or configuration.

tests/go.mod (7)

3-3: Simplified Go version specification to 1.23.

The simplification of the Go version specification is approved. Verify that this change does not affect the build or test environments adversely.


5-5: Added toolchain go1.23.0 directive.

The addition of the toolchain go1.23.0 directive is approved. Verify that this directive is correctly configured and does not introduce any issues in the test environment.


17-17: Updated github.com/roadrunner-server/server/v5 to v5.1.0.

The update to github.com/roadrunner-server/server/v5 v5.1.0 is approved. Ensure thorough testing to verify no regressions in server functionalities.


21-21: Updated google.golang.org/grpc to v1.66.0.

The update to google.golang.org/grpc v1.66.0 is approved. Ensure thorough testing to verify no regressions in gRPC functionalities.


64-64: Updated github.com/prometheus/common to v0.57.0.

The update to github.com/prometheus/common v0.57.0 is approved. Verify that this update does not introduce any issues in metrics reporting or configuration.


101-101: Updated go.temporal.io/api to v1.39.0.

The update to go.temporal.io/api v1.39.0 is approved. Ensure thorough testing to verify no regressions in workflow management functionalities.


111-112: Updated google.golang.org/genproto/googleapis/api and google.golang.org/genproto/googleapis/rpc to newer versions.

The updates to google.golang.org/genproto/googleapis/api and google.golang.org/genproto/googleapis/rpc are approved. Verify that these updates do not affect any API interfaces or behaviors adversely.

plugin.go (2)

195-195: Graceful server shutdown implemented.

The replacement of p.server.Stop() with p.server.GracefulStop() is a positive change, ensuring that ongoing processes can complete before the server is fully shut down.

The code changes are approved.


200-202: Proper resource cleanup for worker pools.

The addition of a conditional block to check and destroy p.gPool if it is not nil is a crucial improvement. This ensures that the worker pools are properly cleaned up, which can help prevent memory leaks or other resource management issues.

The code changes are approved.

Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between f49b6e3 and 8c81808.

Files selected for processing (7)
  • codec/codec.go (3 hunks)
  • codec/codec_test.go (1 hunks)
  • go.mod (3 hunks)
  • plugin.go (2 hunks)
  • server.go (1 hunks)
  • tests/grpc_plugin_gzip_test.go (11 hunks)
  • tests/grpc_plugin_test.go (12 hunks)
Files skipped from review due to trivial changes (2)
  • go.mod
  • server.go
Files skipped from review as they are similar to previous changes (1)
  • plugin.go
Additional comments not posted (19)
codec/codec.go (3)

31-36: Enhanced error handling and memory management in Marshal.

The addition of error handling and the use of data.Materialize() are significant improvements.

The code changes are approved.

Run the following script to verify the impact of these changes on performance:


41-54: Improved type safety and memory management in Unmarshal.

The use of a type switch and mem.BufferSlice are commendable for enhancing type safety and memory efficiency.

The code changes are approved.

Run the following script to verify the impact of these changes on memory usage:


22-22: Approved change to Base field type.

The change from encoding.Codec to encoding.CodecV2 is approved as it likely enhances the codec's capabilities.

The code changes are approved.

Run the following script to verify the impact of this change on the rest of the codebase:

tests/grpc_plugin_test.go (10)

115-115: Good practice: Explicitly closing the connection.

The addition of _ = conn.Close() ensures that resources are properly released after the test, preventing potential resource leaks.


187-187: Good practice: Proper resource management.

Closing the connection with _ = conn.Close() is crucial for cleaning up resources after the test, aligning with best practices.


Line range hint 302-302: Configuration change: Removal of the Prefix field.

The removal of the Prefix field from the configuration may indicate a change in how configurations are handled. Ensure that this change is reflected in the documentation and that it does not affect other parts of the application.

Please verify that the removal of the Prefix field does not impact other functionalities or configurations.


440-440: Consistent resource management.

The addition of _ = conn.Close() in TestGrpcInitMultiple is consistent with other tests, ensuring that resources are properly managed and released.


518-518: Effective resource management.

Including _ = conn.Close() in TestGrpcRqRs ensures that the gRPC connection is properly closed, preventing resource leaks.


623-623: Consistent and effective resource management.

The inclusion of _ = conn.Close() in TestGrpcRqRsException aligns with best practices for managing resources in test environments.


717-717: Good practice: Proper resource management.

Closing the connection with _ = conn.Close() in TestGrpcRqRsMultiple ensures that resources are properly managed and released after the test.


802-802: Effective resource management.

Including _ = conn.Close() in TestGrpcRqRsTLS ensures that the gRPC connection is properly closed, preventing resource leaks.


887-887: Consistent resource management.

The addition of _ = conn.Close() in TestGrpcRqRsTLSRootCA is consistent with other tests, ensuring that resources are properly managed and released.


979-979: Good practice: Explicitly closing the connection.

The addition of _ = conn.Close() in TestGrpcRqRsTLS_WithReset ensures that resources are properly released after the test, preventing potential resource leaks.

codec/codec_test.go (1)

24-26: LGTM on Unmarshal method changes.

The update to accept mem.BufferSlice and the use of Materialize to convert it back to a byte slice are well-implemented and align with the goals of improved memory management.

tests/grpc_plugin_gzip_test.go (5)

102-102: Good practice: Ensure connection closure.

Adding _ = conn.Close() is a good practice to ensure that resources are properly managed and potential leaks are prevented.


196-196: Good practice: Ensure connection closure in multiple requests test.

The inclusion of _ = conn.Close() in the test for multiple gRPC requests with Gzip compression is a good practice for managing resources effectively.


281-281: Good practice: Proper resource management in TLS tests.

The addition of _ = conn.Close() in the TLS test ensures that resources are properly managed, which is crucial in secure connection scenarios.


366-366: Consistent resource management across tests.

The consistent use of _ = conn.Close() across different test scenarios, including this one with Root CA, ensures effective resource management.


459-459: Good practice: Connection closure in reset scenario.

Including _ = conn.Close() in the test involving TLS and reset scenarios is a good practice for managing resources effectively.

codec/codec_test.go Show resolved Hide resolved
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8c81808 and b77b943.

Files selected for processing (6)
  • .golangci.yml (1 hunks)
  • codec/codec.go (3 hunks)
  • plugin.go (2 hunks)
  • tests/configs/.rr-grpc-rq-multiple.yaml (2 hunks)
  • tests/grpc_plugin_gzip_test.go (12 hunks)
  • tests/grpc_plugin_test.go (13 hunks)
Files skipped from review as they are similar to previous changes (4)
  • .golangci.yml
  • plugin.go
  • tests/grpc_plugin_gzip_test.go
  • tests/grpc_plugin_test.go
Additional comments not posted (5)
codec/codec.go (3)

22-22: Verify compatibility with CodecV2.

The change from encoding.Codec to encoding.CodecV2 in the Base field of the Codec struct suggests enhancements in codec capabilities. Please ensure that all methods using Base are compatible with the new type and that no deprecated methods are being called.


31-36: Approve error handling and query data.Materialize() usage.

The addition of error handling in the Marshal method is a positive change, enhancing robustness. The use of data.Materialize() is intriguing and suggests improvements in memory management or performance. Please verify the impact of this method on memory usage and performance to ensure it aligns with expected benefits.


41-61: Approve enhanced type handling in Unmarshal.

The addition of a type switch in the Unmarshal method enhances its flexibility and robustness, allowing for specific handling based on the input type. This is a significant improvement, especially for handling proto.Message more efficiently. Consider adding documentation to explain the handling logic for each case to aid future maintainability.

tests/configs/.rr-grpc-rq-multiple.yaml (2)

18-18: Approve change in listening socket and verify documentation.

The update of the listening socket from tcp://127.0.0.1:9001 to tcp://127.0.0.1:9003 is approved. Please ensure that all related documentation, scripts, and configurations are updated to reflect this change.


18-18: Query removal of max_jobs parameter and verify impact.

The removal of the max_jobs parameter from the workers pool configuration could significantly impact how jobs are allocated and managed. Please provide the rationale behind this change and verify its impact on the system's behavior, especially in peak load scenarios.

@rustatian rustatian merged commit 0d6dced into master Aug 30, 2024
6 checks passed
@rustatian rustatian deleted the chore/pool-shutdown branch August 30, 2024 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant