implement ngt-rs and its unit test and add GitHub Actions workflows

[kmrmt]

Description

Implement ngt wrapper for Rust and its unit tests.
And add GitHub Actions workflows.
This PR includes interfaces for implementing Rust agent ngt.

Related Issue

Versions

• Go Version: 1.22.2
• Docker Version: 20.10.8
• Kubernetes Version: v1.29.3
• NGT Version: 2.2

Checklist

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer

Summary by CodeRabbit

• New Features

  • Introduced a Docker image for the "ngt" Rust-based application, enhancing the deployment process.
  • Added a new GitHub Actions workflow to automate the building of Docker images for the agent component.
  • Enhanced the agent component with new capabilities for handling nearest neighbor searches and index management.

• Documentation

  • Updated documentation to detail the setup and capabilities of the vald-agent Docker image.

• Refactor

  • Integrated ngt-rs as a member in the Cargo workspace to streamline dependency management.

• Chores

  • Configured Makefile to support new Docker image targets and Rust build processes.

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[cloudflare-workers-and-pages]

Deploying vald with    Cloudflare Pages

Latest commit:	e24bf3a
Status:	✅  Deploy successful!
Preview URL:	https://b23af568.vald.pages.dev
Branch Preview URL:	https://feature-agent-implement-ngt.vald.pages.dev

View logs

[github-actions]

⚠️ [hadolint] <DL3008> _{reported by reviewdog 🐶}
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

[kpango]

LGTM

[coderabbitai]

Walkthrough

Walkthrough

The project enhancements involve integrating a Rust-based NGT (Neighborhood Graph and Tree) library for efficient nearest neighbor searches, with Docker support for deployment. Changes include Dockerfile configurations, Makefile updates for image building, and new Rust code for managing NGT indices. This setup aims to streamline development and deployment processes for the agent component in various environments.

Changes

Files	Change Summary
.github/workflows/dockers-agent-image.yml	Introduces a GitHub Actions workflow for building Docker images specific to the agent component.
Makefile, Makefile.d/build.mk, Makefile.d/docker.mk	Updates and adds new targets for Docker image handling and building Rust targets for the agent.
dockers/agent/core/agent/Dockerfile, dockers/agent/core/agent/README.md	Sets up a Docker environment for building and running a Rust-based application, and details its usage within a Vald cluster or standalone setups.
rust/Cargo.toml, rust/bin/agent/Cargo.toml, rust/libs/.../Cargo.toml	Adds new dependencies and workspace configurations for handling the NGT library in Rust.
rust/libs/ngt-rs/, including build.rs, src/...	Introduces new Rust and C++ code for managing NGT indices, including FFI bindings and utility functions for nearest neighbor operations.

Poem

🐰🌟
A hop, a skip, in code we dip,
Rust and Docker, our new ship.
With indices built, and vectors set,
Our agent's ready, on this you can bet!
🚀🎉

---

Recent Review Details

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 87b4d9a and e24bf3a.

Files ignored due to path filters (1)

• rust/Cargo.lock is excluded by !**/*.lock

Files selected for processing (14)

• .github/workflows/dockers-agent-image.yml (1 hunks)
• Makefile (1 hunks)
• Makefile.d/build.mk (1 hunks)
• Makefile.d/docker.mk (1 hunks)
• dockers/agent/core/agent/Dockerfile (1 hunks)
• dockers/agent/core/agent/README.md (1 hunks)
• rust/Cargo.toml (1 hunks)
• rust/bin/agent/Cargo.toml (1 hunks)
• rust/libs/ngt-rs/Cargo.toml (1 hunks)
• rust/libs/ngt-rs/build.rs (1 hunks)
• rust/libs/ngt-rs/src/input.cpp (1 hunks)
• rust/libs/ngt-rs/src/input.h (1 hunks)
• rust/libs/ngt-rs/src/lib.rs (1 hunks)
• rust/libs/ngt/Cargo.toml (1 hunks)

Files skipped from review due to trivial changes (2)

• rust/Cargo.toml
• rust/libs/ngt-rs/Cargo.toml

Additional Context Used

LanguageTool (56)

dockers/agent/core/agent/README.md (56)

Near line 1: Possible spelling mistake found.
Context: # Vald Agent ...

---

Near line 7: This sentence does not start with an uppercase letter.
Context: ...ing and runs incorrectly. vald-agent is the docker image for the vald-agent com...

---

Near line 7: Possible spelling mistake found.
Context: ...vald-agent` is the docker image for the vald-agent component. This image is responsible f...

---

Near line 15: Possible spelling mistake found.
Context: ... use the external storage, it requires [vald-agent-sidecar](https://hub.docker.com/r/vdaas/vald-ag...

---

Near line 24: Possible spelling mistake found.
Context: ...s://opensource.org/licenses/Apache-2.0) [![latest ver.](https://img.shields.io/github/release...

---

Near line 32: Possible spelling mistake found.
Context: ...ribe here with ⚠️ emoji --> ### linux/amd64 - CPU instruction: requires `AVX...

---

Near line 36: Possible spelling mistake found.
Context: ...ction: requires AVX2 or AVX512 ### linux/arm64 ⚠ This image does NOT support ru...

---

Near line 45: Possible spelling mistake found.
Context: ...apter Agent Standalone --> You can use vald-agent in 3 ways. 1. One of the components o...

---

Near line 47: Possible spelling mistake found.
Context: ... ways. 1. One of the components of the Vald cluster - Refer to [Get Started](htt...

---

Near line 48: Only proper nouns start with an uppercase character (there are exceptions for headlines).
Context: ...the components of the Vald cluster - Refer to [Get Started](https://vald.vdaas.org...

---

Near line 50: Unpaired symbol: ‘]’ seems to be missing
Context: ...on the Kubernetes cluster - Refer to [Vald Agent Standalone on Kubernetes](htt...

---

Near line 50: Possible spelling mistake found.
Context: ...n the Kubernetes cluster - Refer to [Vald Agent Standalone on Kubernetes](https:/...

---

Near line 52: Unpaired symbol: ‘]’ seems to be missing
Context: ...) 1. Standalone on Docker - Refer to [Vald Agent Standalone on Docker](https:/...

---

Near line 52: Possible spelling mistake found.
Context: ... 1. Standalone on Docker - Refer to [Vald Agent Standalone on Docker](https://val...

---

Near line 56: Possible typo: you repeated a whitespace
Context: ...andalone-on-docker/) ## Versions | tag | linux/amd64 | linux/arm64 | descriptio...

---

Near line 56: Possible spelling mistake found.
Context: ...e-on-docker/) ## Versions | tag | linux/amd64 | linux/arm64 | description ...

---

Near line 56: Possible spelling mistake found.
Context: ... ## Versions | tag | linux/amd64 | linux/arm64 | description ...

---

Near line 56: Possible typo: you repeated a whitespace
Context: ... linux/amd64 | linux/arm64 | description | | :------ | :---------: | :---------: ...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...----------------------------- | | latest | ✅ | ✅ | the latest i...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...-------------------------- | | latest | ✅ | ✅ | the latest image i...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...-------------------- | | latest | ✅ | ✅ | the latest image is the s...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...------------- | | latest | ✅ | ✅ | the latest image is the same as...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...------- | | latest | ✅ | ✅ | the latest image is the same as the la...

---

Near line 58: Possible spelling mistake found.
Context: ...e is the same as the latest version of vdaas/vald re...

---

Near line 58: Possible spelling mistake found.
Context: ...he same as the latest version of vdaas/vald reposit...

---

Near line 58: Possible typo: you repeated a whitespace
Context: ...hub.com/vdaas/vald) repository version. | | nightly | ✅ | ✅ | ...

---

Near line 59: Possible typo: you repeated a whitespace
Context: ...ository version. | | nightly | ✅ | ✅ | the nightly applie...

---

Near line 59: Possible typo: you repeated a whitespace
Context: ...y version. | | nightly | ✅ | ✅ | the nightly applies the m...

---

Near line 59: Possible typo: you repeated a whitespace
Context: ...on. | | nightly | ✅ | ✅ | the nightly applies the main br...

---

Near line 59: Possible typo: you repeated a whitespace
Context: ... | | nightly | ✅ | ✅ | the nightly applies the main branch's ...

---

Near line 59: This adverb was used twice in the sentence. Consider removing one of them or replacing them with a synonym.
Context: ...nightly | ✅ | ✅ | the nightly applies the main branch's source code o...

---

Near line 59: Possible spelling mistake found.
Context: ...s the main branch's source code of the vdaas/vald re...

---

Near line 59: Possible spelling mistake found.
Context: ...main branch's source code of the vdaas/vald reposit...

---

Near line 59: Possible typo: you repeated a whitespace
Context: ...ps://github.com/vdaas/vald) repository. | | vX.Y.Z | ✅ | ✅ | ...

---

Near line 60: Possible spelling mistake found.
Context: ...as/vald) repository. | | vX.Y.Z | ✅ | ✅ | the vX...

---

Near line 60: Add a space between sentences.
Context: ...vald) repository. | | vX.Y.Z | ✅ | ✅ | the vX.Y...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ...d) repository. | | vX.Y.Z | ✅ | ✅ | the vX.Y.Z i...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ...repository. | | vX.Y.Z | ✅ | ✅ | the vX.Y.Z image a...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ...tory. | | vX.Y.Z | ✅ | ✅ | the vX.Y.Z image applies ...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ... | | vX.Y.Z | ✅ | ✅ | the vX.Y.Z image applies the so...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ... | | vX.Y.Z | ✅ | ✅ | the vX.Y.Z image applies the source co...

---

Near line 60: Possible spelling mistake found.
Context: ...vX.Y.Z | ✅ | ✅ | the vX.Y.Z image applies the source code of th...

---

Near line 60: Add a space between sentences.
Context: ...Y.Z | ✅ | ✅ | the vX.Y.Z image applies the source code of the ...

---

Near line 60: Possible spelling mistake found.
Context: ...Z image applies the source code of the vdaas/vald re...

---

Near line 60: Possible spelling mistake found.
Context: ...e applies the source code of the vdaas/vald reposit...

---

Near line 60: Possible typo: you repeated a whitespace
Context: ...ps://github.com/vdaas/vald) repository. | | pr-XXX | ✅ | ❌ | ...

---

Near line 61: Possible typo: you repeated a whitespace
Context: ...tory. | | pr-XXX | ✅ | ❌ | the pr-XXX i...

---

Near line 61: Possible typo: you repeated a whitespace
Context: ...y. | | pr-XXX | ✅ | ❌ | the pr-XXX image a...

---

Near line 61: Possible typo: you repeated a whitespace
Context: ... | | pr-XXX | ✅ | ❌ | the pr-XXX image applies ...

---

Near line 61: Possible typo: you repeated a whitespace
Context: ... | | pr-XXX | ✅ | ❌ | the pr-XXX image applies the so...

---

Near line 61: Possible typo: you repeated a whitespace
Context: ... | | pr-XXX | ✅ | ❌ | the pr-XXX image applies the source co...

---

Near line 61: Possible spelling mistake found.
Context: ...ce code of the pull request XXX of the vdaas/vald re...

---

Near line 61: Possible spelling mistake found.
Context: ...e of the pull request XXX of the vdaas/vald reposit...

---

Near line 69: Possible spelling mistake found.
Context: ...agent/core/agent/Dockerfile). ## About Vald Project <!...

---

Near line 74: Possible spelling mistake found.
Context: ...s static --> The information about the Vald project, please refer to the following:...

---

Near line 81: The phrase ‘feel free to’ is used quite frequently. Consider using a less frequent alternative to set your writing apart from others and make it sound more professional.
Context: ...## Contacts We're love to support you! Please feel free to contact us anytime with your questions ...

AST-based Instructions (1)

rust/libs/ngt-rs/src/lib.rs (1)

[warning] 119-129: Found use of unsafe code. Unsafe code should be avoided whenever possible. Instead, prefer safe code and use unsafe code only when necessary.
Context: unsafe {
index.pin_mut().search(
gen_random_vector(DIMENSION).as_slice(),
K as i32,
0.05,
-1.0,
i32::MIN,
&mut ids[0] as *mut i32,
&mut distances[0] as *mut f32,
)?
}
Note: [CWE-242]: Use of Inherently Dangerous Function [REFERENCES]
- https://doc.rust-lang.org/std/keyword.unsafe.html

Additional comments not posted (19)

rust/libs/ngt/Cargo.toml (1)

24-24: Dependency on ngt-rs version 0.1.0 added correctly.

rust/bin/agent/Cargo.toml (1)

24-24: Dependency on ngt version 0.1.0 added correctly.

rust/libs/ngt-rs/build.rs (1)

16-33: Build script setup correctly for integrating C++ with Rust.

rust/libs/ngt-rs/src/input.h (1)

16-55: Header file correctly defines classes interfacing with NGT library.

.github/workflows/dockers-agent-image.yml (1)

16-74: GitHub Actions workflow correctly set up for building Docker images.

dockers/agent/core/agent/Dockerfile (1)

18-95: Dockerfile correctly sets up the environment for the agent using best practices.

dockers/agent/core/agent/README.md (1)

1-89: README.md provides clear and informative details about the vald-agent Docker image.

rust/libs/ngt-rs/src/input.cpp (1)

16-137: Source file correctly implements NGT operations and interfaces with Rust.

rust/libs/ngt-rs/src/lib.rs (6)

18-24: LGTM! The enum ObjectType is well-defined and covers various data types.

---

26-42: LGTM! The enum DistanceType is comprehensive and covers a wide range of distance metrics.

---

48-48: Ensure proper documentation for new_property function to clarify its usage and the type of property it initializes.

---

49-49: Ensure set_dimension checks for non-negative dimensions to prevent runtime errors.

---

57-66: Consider replacing the unsafe block with safe Rust code if possible, to enhance safety and maintainability.

---

102-150: The unit test test_ngt appears comprehensive. However, ensure that edge cases and error conditions are also tested to improve test coverage.

Makefile.d/build.mk (2)

98-99: Ensure that the directory change with pushd is paired with popd to avoid unexpected directory changes affecting subsequent commands.

---

101-102: Similar to the release target, verify that directory changes do not affect other parts of the build process.

Makefile.d/docker.mk (2)

112-113: LGTM! The target docker/name/agent correctly outputs the Docker image name for the agent.

---

116-120: Ensure that the Docker build process for the agent is optimized and that all necessary build arguments are provided.

Makefile (1)

27-27: Ensure that the AGENT_IMAGE variable is consistently used across all relevant Docker and build targets to maintain consistency.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.