Backport #2559

[kpango]

Description

SSIA

Related Issue

Versions

• Vald Version: v1.7.12
• Go Version: v1.22.5
• Rust Version: v1.80.0
• Docker Version: v27.1.1
• Kubernetes Version: v1.30.3
• Helm Version: v3.15.3
• NGT Version: v2.2.4
• Faiss Version: v1.8.0

Checklist

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer

Summary by CodeRabbit

• New Features

  • Introduced a new workflow for automating updates to GitHub Actions dependencies.
  • Added foundational functionalities for insert, remove, search, update, and upsert operations in the agent system.
  • Enhanced the handling of asynchronous operations with new methods for streaming and batch processing.

• Bug Fixes

  • Removed redundant package installations to streamline Docker builds.

• Documentation

  • Updated API documentation for improved clarity and consistency.

• Chores

  • Updated dependency versions for Go and Rust projects to ensure compatibility and access to the latest features.
  • Incremented version numbers for various components to reflect minor updates.

[cloudflare-workers-and-pages]

Deploying vald with    Cloudflare Pages

Latest commit:	847281a
Status:	✅  Deploy successful!
Preview URL:	https://3d3660c5.vald.pages.dev
Branch Preview URL:	https://backport-2559.vald.pages.dev

View logs

[coderabbitai]

Walkthrough

Walkthrough

The changes primarily involve updates to configuration files, Dockerfiles, and Rust project files, including version increments and the introduction of new dependencies. Notable adjustments include refinements to GitHub Actions workflows, simplifications in Dockerfile dependencies, and enhancements to Rust modules for improved functionality and organization. The overall focus is on optimizing build processes and ensuring the application supports new features, particularly in asynchronous operations and dependency management.

Changes

Files	Change Summary
.github/workflows/*.yml	Modifications to path filters for pull requests and pushes, refining build triggers and improving workflow efficiency.
Makefile	Set shell to bash and removed -lquadmath from linking flags, clarifying environment settings and streamlining dependencies.
dockers/...Dockerfile	Removed libquadmath0 from installation lists and repositioned environment variable declarations to enhance clarity and reduce dependencies.
go.mod, Cargo.toml	Updated dependencies and versions for Go and Rust projects to maintain compatibility and leverage new features.
rust/bin/agent/src/*.rs	Introduced new handler files implementing gRPC methods for insertion, removal, searching, updating, and upserting vectors, enhancing the agent's functionality.
rust/libs/proto/Cargo.toml, rust/libs/proto/src/lib.rs	Updated versions of tonic and restructured modules for better organization, removing tests but adding a core module and its version updates.
versions/*	Incremented version numbers for stacks and actions, indicating minor updates and enhancements.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Agent
    participant Database

    Client->>Agent: Send insert request
    Agent->>Database: Insert vector
    Database-->>Agent: Confirm insertion
    Agent-->>Client: Return location of inserted vector

    Client->>Agent: Send multi-insert request
    Agent->>Database: Batch insert vectors
    Database-->>Agent: Confirm batch insertion
    Agent-->>Client: Return locations of inserted vectors

Loading

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[vdaas-ci]

[WARNING:CONFLICT] You may require to fix the conflict. Please check.

[github-actions]

🚫 [documents testlint] <eslint.rules.prh> _{reported by reviewdog 🐶}
indices => indexes (prh)

[github-actions]

🚫 [documents testlint] <eslint.rules.prh> _{reported by reviewdog 🐶}
indices => indexes (prh)

[coderabbitai]

Actionable comments posted: 25

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 5023899 and 847281a.

Files ignored due to path filters (2)

• go.sum is excluded by !**/*.sum
• rust/Cargo.lock is excluded by !**/*.lock

Files selected for processing (26)

• .gitfiles (8 hunks)
• .github/workflows/dockers-agent-faiss-image.yml (1 hunks)
• .github/workflows/dockers-agent-image.yml (1 hunks)
• .github/workflows/dockers-agent-ngt-image.yml (1 hunks)
• .github/workflows/update-actions.yaml (1 hunks)
• Makefile (2 hunks)
• dockers/agent/core/agent/Dockerfile (2 hunks)
• dockers/agent/core/faiss/Dockerfile (1 hunks)
• dockers/ci/base/Dockerfile (2 hunks)
• dockers/dev/Dockerfile (1 hunks)
• docs/api/flush.md (2 hunks)
• go.mod (5 hunks)
• hack/docker/gen/main.go (1 hunks)
• rust/bin/agent/Cargo.toml (1 hunks)
• rust/bin/agent/src/handler.rs (1 hunks)
• rust/bin/agent/src/handler/common.rs (1 hunks)
• rust/bin/agent/src/handler/insert.rs (1 hunks)
• rust/bin/agent/src/handler/remove.rs (1 hunks)
• rust/bin/agent/src/handler/search.rs (1 hunks)
• rust/bin/agent/src/handler/update.rs (1 hunks)
• rust/bin/agent/src/handler/upsert.rs (1 hunks)
• rust/bin/agent/src/main.rs (1 hunks)
• rust/libs/proto/Cargo.toml (1 hunks)
• rust/libs/proto/src/lib.rs (1 hunks)
• versions/PROMETHEUS_STACK_VERSION (1 hunks)
• versions/actions/PETER_EVANS_CREATE_PULL_REQUEST (1 hunks)

Files skipped from review due to trivial changes (6)

• .gitfiles
• dockers/agent/core/faiss/Dockerfile
• dockers/dev/Dockerfile
• hack/docker/gen/main.go
• versions/PROMETHEUS_STACK_VERSION
• versions/actions/PETER_EVANS_CREATE_PULL_REQUEST

Additional context used

LanguageTool

docs/api/flush.md

[uncategorized] ~56-~56: The verb “is” doesn’t seem to fit in this context, “are” is probably more formally correct.
Context: ...te indicating whether vald-agent pods is present in the saving. | ### Status ...

(AI_HYDRA_LEO_CPT_IS_ARE)

Additional comments not posted (45)

rust/bin/agent/src/handler.rs (3)

1-15: File header looks good.

The file header contains the appropriate licensing information.

---

16-22: Module declarations look good.

The module declarations are clear and well-organized.

---

24-27: Empty struct definition.

The Agent struct is defined but currently empty. Ensure that fields and methods are added as needed in the future.

Is this struct intended to be expanded upon later? If so, consider adding a TODO comment to indicate future plans.

rust/libs/proto/Cargo.toml (1)

26-27: Updated dependencies to newer versions.

The tonic and tonic-types dependencies have been updated to version 0.11.0. Ensure compatibility with the rest of the codebase.

Please verify that the rest of the codebase is compatible with these updated dependencies and that any breaking changes have been addressed.

rust/bin/agent/src/handler/common.rs (2)

1-15: File header looks good.

The file header contains the appropriate licensing information.

---

16-21: Macro definition looks good.

The stream_type macro is well-defined and simplifies the creation of a specific type of stream.

rust/libs/proto/src/lib.rs (2)

34-37: LGTM! The addition of the core module enhances modularity.

The new core module with a nested v1 module includes generated code from core.v1.tonic.rs, which is a positive change for organizing core functionalities.

---

34-37: Ensure test coverage.

The removal of the tests module may impact the ability to run unit tests for this code. Ensure that tests are provided elsewhere or reintroduce the necessary tests.

rust/bin/agent/src/main.rs (1)

19-29: LGTM! The changes enhance functionality with asynchronous operations.

The main function now supports asynchronous operations and initializes a server using tokio and tonic. Ensure proper error handling and logging for robust server operations.

rust/bin/agent/Cargo.toml (5)

25-25: LGTM! The addition of prost is appropriate.

The prost library version 0.12.4 is suitable for handling Protocol Buffers.

---

26-26: LGTM! The addition of proto is appropriate.

The proto library version 0.1.0 with a specified path is suitable for managing protocol-related functionalities.

---

27-27: LGTM! The addition of tokio is appropriate.

The tokio library version 1.37.0 with full features is suitable for supporting asynchronous operations.

---

28-28: LGTM! The addition of tokio-stream is appropriate.

The tokio-stream library version 0.1.15 with full features is suitable for handling asynchronous streams.

---

29-29: LGTM! The addition of tonic is appropriate.

The tonic library version 0.11.0 is suitable for supporting gRPC functionality.

docs/api/flush.md (1)

51-52: LGTM!

The reformatting improves readability and consistency.

rust/bin/agent/src/handler/insert.rs (1)

1-15: LGTM!

The license header and imports are appropriate.

rust/bin/agent/src/handler/upsert.rs (1)

1-15: LGTM!

The license header and imports are appropriate.

.github/workflows/dockers-agent-image.yml (4)

33-33: Approve the addition of Makefile to workflow paths.

Including the Makefile ensures that changes to the build configuration will trigger the appropriate workflows.

Also applies to: 49-49

---

34-34: Approve the addition of dockers/agent/core/agent/Dockerfile to workflow paths.

Including the Dockerfile ensures that changes to the Docker configuration will trigger the appropriate workflows.

Also applies to: 50-50

---

41-41: Approve the addition of versions/FAISS_VERSION to workflow paths.

Including the version file ensures that changes to the FAISS version will trigger the appropriate workflows.

Also applies to: 57-57

---

43-43: Approve the addition of versions/RUST_VERSION to workflow paths.

Including the version file ensures that changes to the Rust version will trigger the appropriate workflows.

Also applies to: 59-59

.github/workflows/dockers-agent-ngt-image.yml (6)

30-30: Approve the exclusion of internal/**/*_test.go from workflow paths.

Excluding test files helps to streamline the workflow process by preventing unnecessary builds.

Also applies to: 48-48

---

31-31: Approve the exclusion of internal/db/** from workflow paths.

Excluding database-related files helps to streamline the workflow process by preventing unnecessary builds.

Also applies to: 49-49

---

32-32: Approve the exclusion of internal/k8s/** from workflow paths.

Excluding Kubernetes-related files helps to streamline the workflow process by preventing unnecessary builds.

Also applies to: 50-50

---

36-36: Approve the addition of Makefile to workflow paths.

Including the Makefile ensures that changes to the build configuration will trigger the appropriate workflows.

Also applies to: 54-54

---

37-37: Approve the addition of apis/grpc/** to workflow paths.

Including the apis/grpc directory ensures that changes to the gRPC API will trigger the appropriate workflows.

Also applies to: 55-55

---

38-38: Approve the addition of cmd/agent/core/ngt/** to workflow paths.

Including the cmd/agent/core/ngt directory ensures that changes to the NGT agent core will trigger the appropriate workflows.

Also applies to: 56-56

.github/workflows/dockers-agent-faiss-image.yml (2)

30-45: LGTM! Path filters updated for pull_request trigger.

The changes refine the paths that trigger the workflow, potentially optimizing build processes.

---

48-63: LGTM! Path filters updated for pull_request_target trigger.

The changes align with the updates made to the pull_request trigger, ensuring consistency.

rust/bin/agent/src/handler/remove.rs (2)

1-19: LGTM! License header and imports are correct.

The license header and imports are appropriate for the file's functionality.

---

38-40: LGTM! Type definition for StreamRemoveStream is appropriate.

The type definition using the macro is suitable for server streaming responses.

.github/workflows/update-actions.yaml (4)

1-16: LGTM! License header and workflow name are correct.

The license header and workflow name are appropriate for the file.

---

17-21: LGTM! Workflow triggers are appropriate.

The workflow_dispatch and daily cron schedule are suitable triggers for this workflow.

---

22-27: LGTM! Job steps for dump-contexts-to-log are appropriate.

The steps for checking out the repository and dumping context are suitable for the job.

---

28-64: LGTM! Job steps for update-version are comprehensive and appropriate.

The steps cover all necessary actions for updating Actions dependencies and creating a PR.

dockers/agent/core/agent/Dockerfile (1)

47-47: LGTM! Verify if there are any dependencies or build processes affected.

The RUSTUP_HOME environment variable declaration has been moved from a commented-out state to an active declaration. This change ensures that the Rustup toolchain is properly configured within the environment.

However, verify if this change affects any dependencies or build processes.

dockers/ci/base/Dockerfile (2)

52-52: LGTM!

The CARGO_HOME environment variable declaration has been repositioned to precede the RUSTUP_HOME declaration. This change enhances clarity regarding the order of environment variable definitions.

---

52-52: Verify if the removal of libquadmath0 affects any dependencies or build processes.

The libquadmath0 package has been removed from the installation list. This optimization may affect any dependencies that require libquadmath0.

Verification successful

---

The removal of libquadmath0 does not affect any dependencies or build processes.

• No references to libquadmath0 were found in the entire codebase, including common dependency files.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify if the removal of `libquadmath0` affects any dependencies or build processes.

# Test: Check for any references to `libquadmath0` in the codebase.
rg --type dockerfile --type sh 'libquadmath0'

Length of output: 80

---

Script:

#!/bin/bash
# Description: Verify if the removal of `libquadmath0` affects any dependencies or build processes.

# Test: Check for any references to `libquadmath0` in the entire codebase.
rg 'libquadmath0'

Length of output: 17

---

Script:

#!/bin/bash
# Description: Verify if the removal of `libquadmath0` affects any dependencies or build processes by checking common dependency files.

# Test: Check for any references to `libquadmath0` in common dependency files.
rg 'libquadmath0' --glob '*Cargo.toml' --glob '*package.json' --glob '*requirements.txt' --glob '*Pipfile' --glob '*pyproject.toml'

Length of output: 131

Makefile (2)

17-17: LGTM! Ensures consistent shell environment.

Setting the SHELL variable to bash ensures that the Makefile uses bash for executing commands, which might be necessary for specific bash features.

---

148-148: Verify the impact of removing the -lquadmath flag.

The -lquadmath flag has been removed from FAISS_LDFLAGS. Ensure that the quad math library is no longer needed and that this change does not affect any functionality.

go.mod (5)

317-317: LGTM! Version update for google.golang.org/api.

The version for google.golang.org/api has been incremented from v0.189.0 to v0.190.0. This update might include bug fixes, performance improvements, or new features.

---

317-317: LGTM! Version update for google.golang.org/genproto.

The version for google.golang.org/genproto has been updated to v0.0.0-20240730163845-b1a4ccb954bf. This update might include bug fixes, performance improvements, or new features.

---

409-409: LGTM! Version update for google.golang.org/genproto/googleapis/rpc.

The version for google.golang.org/genproto/googleapis/rpc has been updated to v0.0.0-20240730163845-b1a4ccb954bf. This update might include bug fixes, performance improvements, or new features.

---

425-425: LGTM! Version update for cloud.google.com/go/auth.

The version for cloud.google.com/go/auth has been incremented from v0.7.2 to v0.7.3. This update might include bug fixes, performance improvements, or new features.

---

465-465: LGTM! Version update for github.com/google/s2a-go.

The version for github.com/google/s2a-go has been incremented from v0.1.7 to v0.1.8. This update might include bug fixes, performance improvements, or new features.