Backport PR #2375 to release/v1.7 for Add `index-operator` template implementation #2424

vdaas-ci · 2024-02-27T08:07:17Z

Description

This PR adds below for index-operator.

Template implementation of Pod and Job reconcilers
Docker image
Helm template files

Each reconciler logic should be implemented in podOnReconcile and jobOnReconcile functions.

Related Issue

Versions

Go Version: 1.22.0
Docker Version: 20.10.8
Kubernetes Version: v1.29.2
NGT Version: 2.1.6

Checklist

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer

* improve Backport Pull Request Title & Description Quality (#2373) Signed-off-by: kpango <[email protected]> Co-authored-by: Kosuke Morimoto <[email protected]> * [bugfix] fix backport github value bypass failure (#2378) Signed-off-by: kpango <[email protected]> * [bugfix] fix backport github value bypass failure (#2381) Signed-off-by: kpango <[email protected]> * Add index operator boilerplate * Add k8s controller * Add pod and job controllers * Add annotations in the pod reconciler * Update auto indexing service configurations * Update WithOnReconcileFunc signature to include context * Refactor debugging log * Refactor reconcile funcs * add nolint * remove unused parameters * lint * Add dockerfile and build ci * remove params * Add index-operator image build command to Makefile * Add index operator k8s templates * Update helm schema * Update readme * Add internal/k8s as a trigger for images that depends on controller-runtime * remove comment * Update pkg/index/operator/service/operator.go Co-authored-by: Yusuke Kato <[email protected]> Signed-off-by: Yusuke Kadowaki <[email protected]> * Update pkg/index/operator/service/operator.go Co-authored-by: Yusuke Kato <[email protected]> Signed-off-by: Yusuke Kadowaki <[email protected]> * Refactor * Add build commands * Refactor Co-authored-by: Yusuke Kato <[email protected]> --------- Signed-off-by: kpango <[email protected]> Signed-off-by: Yusuke Kadowaki <[email protected]> Co-authored-by: Yusuke Kato <[email protected]> Co-authored-by: Kosuke Morimoto <[email protected]>

cloudflare-workers-and-pages · 2024-02-27T08:07:19Z

Deploying with Cloudflare Pages

Latest commit:	`2b18488`
Status:	✅ Deploy successful!
Preview URL:	https://a49dd279.vald.pages.dev
Branch Preview URL:	https://backport-release-v1-7-featur-gmb9.vald.pages.dev

View logs

github-actions · 2024-02-27T08:14:45Z

dockers/index/operator/Dockerfile

+
+
+# skipcq: DOK-DL3008
+RUN apt-get update && apt-get install -y --no-install-recommends \


⚠️ [hadolint] <DL3008> _{reported by reviewdog 🐶}
Pin versions in apt get install. Instead of apt-get install <package> use apt-get install <package>=<version>

vdaas-ci · 2024-02-27T08:14:48Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

vdaas-ci · 2024-02-27T08:14:53Z

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

…operator

github-actions · 2024-02-27T10:19:43Z

pkg/index/operator/config/config.go

@@ -0,0 +1,71 @@
+// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>


🚫 [golangci] _{reported by reviewdog 🐶}
ST1000: package comment should be of the form "Package config ..." (stylecheck)

github-actions · 2024-02-27T10:19:43Z

pkg/index/operator/service/operator.go

@@ -0,0 +1,141 @@
+// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>


🚫 [golangci] _{reported by reviewdog 🐶}
ST1000: package comment should be of the form "Package service ..." (stylecheck)

github-actions · 2024-02-27T10:19:43Z

pkg/index/operator/service/options.go

@@ -0,0 +1,32 @@
+// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>


🚫 [golangci] _{reported by reviewdog 🐶}
ST1000: package comment should be of the form "Package service ..." (stylecheck)

github-actions · 2024-02-27T10:19:43Z

pkg/index/operator/usecase/operator.go

@@ -0,0 +1,152 @@
+// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>


🚫 [golangci] _{reported by reviewdog 🐶}
ST1000: package comment should be of the form "Package usecase ..." (stylecheck)

codecov · 2024-02-27T10:22:57Z

Codecov Report

Attention: Patch coverage is 0% with 217 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (release/v1.7@483699f). Click here to learn what that means.

Files	Patch %	Lines
pkg/index/operator/usecase/operator.go	0.00%	81 Missing ⚠️
pkg/index/operator/service/operator.go	0.00%	65 Missing ⚠️
pkg/index/operator/config/config.go	0.00%	27 Missing ⚠️
cmd/index/operator/main.go	0.00%	22 Missing ⚠️
internal/k8s/pod/pod.go	0.00%	10 Missing ⚠️
pkg/index/operator/service/options.go	0.00%	6 Missing ⚠️
internal/config/index_operator.go	0.00%	4 Missing ⚠️
internal/k8s/pod/option.go	0.00%	1 Missing ⚠️
pkg/discoverer/k8s/service/discover.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##             release/v1.7    #2424   +/-   ##
===============================================
  Coverage                ?   18.90%           
===============================================
  Files                   ?      555           
  Lines                   ?    74079           
  Branches                ?        0           
===============================================
  Hits                    ?    14004           
  Misses                  ?    59312           
  Partials                ?      763

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fix: git add chart directory for release (#2356) (#2357) [patch] release v1.7.11 (#2358) :bookmark: :robot: Release v1.7.11 (#2360) Change docker scan timeout longer (#2363) (#2364) refactor code using golangci-lint (#2362) (#2365) Create SECURITY.md (#2367) (#2368) add commit hash build image (#2359) (#2371) update docker build target platform selection rules (#2370) (#2374) Make agent export index metrics to Pod k8s resource (#2319) (#2372) backport ci deps others (#2386) Update workflow to release readreplica chart (#2383) (#2387) :green_heart: :recycle: Add Con-Bench helm chart to the Vald charts (#2388) (#2389) Delete unnecessary code for mirror (#2366) (#2391) change JP logo to EN logo (#2369) (#2392) Add rotate-all option to rotator (#2305) (#2393) fix: build error of internal kvs test (#2396) (#2398) Resolve kvs already closed before last saving (#2390) (#2394) :robot: Update license headers / Format Go codes and YAML files (#2397) (#2400) create continous benchmark doc (#2352) (#2395) fix: disable protobuf dispatch for client (#2401) (#2403) update deps (#2404) (#2405) [patch] release v1.7.12 (#2406) :bookmark: :robot: Release v1.7.12 (#2408) :pencil: Fix typo of file name (#2413) (#2415) Fix agent-faiss build failed (#2418) (#2419) Add tests for index information export (#2412) (#2414) Fix the logic to determine docker image (#2410) (#2420) Update build rule for nightly image (#2421) (#2422) Fix output settings to determine-docker-image-tag action and release branch build tag name (#2423) (#2425) Add `index-operator` template implementation (#2375) (#2424) fix: typo of execution rule (#2426) (#2427) Backport Flush API (#2434) update deps & add validation for Flush API when agent is Read Only (#2433) (#2436) docs: add hrichiksite as a contributor for doc (#2441) (#2442) fix: bugfix version update for docker build (#2445) (#2446) Fix index job logic to pass DNS A record (#2438) (#2448) Added snapshot timestamp annotations to read replica agent (#2428) (#2443) Fix operator-sdk version (#2447) (#2449) add file name lint (#2417) (#2450) fix: add extra option for ci-container build (#2451) (#2452) Add base of benchmark operator dashboard (#2430) (#2453) Implement index operator logic for read replica rotation (#2444) (#2456) add inner product distance type for ngt (#2454) (#2458) Fix e2e for read replica and add e2e for index operator (#2455) (#2459) Add unit tests for index operator (#2460) (#2461) Bugfix recreate benchmark job when operator reboot (#2463) (#2464) Refactor k8s types (#2462) (#2465) :robot: Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2457) (#2469) Fix workflow trigger for backport pr creation (#2471) (#2472) Automatically add backport main label for release-pr (#2473) (#2475) update deps (#2468) (#2476) Implement client metrics interceptor for continuous benchmark job (#2477) (#2480) :chart_with_upwards_trend: Add client metrics panels for continuous benchmark job (#2481) (#2483) Update continuous benchmark docs (#2485) (#2486) Signed-off-by: kpango <[email protected]>

fix: git add chart directory for release (#2356) (#2357) [patch] release v1.7.11 (#2358) :bookmark: :robot: Release v1.7.11 (#2360) Change docker scan timeout longer (#2363) (#2364) refactor code using golangci-lint (#2362) (#2365) Create SECURITY.md (#2367) (#2368) add commit hash build image (#2359) (#2371) update docker build target platform selection rules (#2370) (#2374) Make agent export index metrics to Pod k8s resource (#2319) (#2372) backport ci deps others (#2386) Update workflow to release readreplica chart (#2383) (#2387) :green_heart: :recycle: Add Con-Bench helm chart to the Vald charts (#2388) (#2389) Delete unnecessary code for mirror (#2366) (#2391) change JP logo to EN logo (#2369) (#2392) Add rotate-all option to rotator (#2305) (#2393) fix: build error of internal kvs test (#2396) (#2398) Resolve kvs already closed before last saving (#2390) (#2394) :robot: Update license headers / Format Go codes and YAML files (#2397) (#2400) create continous benchmark doc (#2352) (#2395) fix: disable protobuf dispatch for client (#2401) (#2403) update deps (#2404) (#2405) [patch] release v1.7.12 (#2406) :bookmark: :robot: Release v1.7.12 (#2408) :pencil: Fix typo of file name (#2413) (#2415) Fix agent-faiss build failed (#2418) (#2419) Add tests for index information export (#2412) (#2414) Fix the logic to determine docker image (#2410) (#2420) Update build rule for nightly image (#2421) (#2422) Fix output settings to determine-docker-image-tag action and release branch build tag name (#2423) (#2425) Add `index-operator` template implementation (#2375) (#2424) fix: typo of execution rule (#2426) (#2427) Backport Flush API (#2434) update deps & add validation for Flush API when agent is Read Only (#2433) (#2436) docs: add hrichiksite as a contributor for doc (#2441) (#2442) fix: bugfix version update for docker build (#2445) (#2446) Fix index job logic to pass DNS A record (#2438) (#2448) Added snapshot timestamp annotations to read replica agent (#2428) (#2443) Fix operator-sdk version (#2447) (#2449) add file name lint (#2417) (#2450) fix: add extra option for ci-container build (#2451) (#2452) Add base of benchmark operator dashboard (#2430) (#2453) Implement index operator logic for read replica rotation (#2444) (#2456) add inner product distance type for ngt (#2454) (#2458) Fix e2e for read replica and add e2e for index operator (#2455) (#2459) Add unit tests for index operator (#2460) (#2461) Bugfix recreate benchmark job when operator reboot (#2463) (#2464) Refactor k8s types (#2462) (#2465) :robot: Automatically update PULL_REQUEST_TEMPLATE and ISSUE_TEMPLATE (#2457) (#2469) Fix workflow trigger for backport pr creation (#2471) (#2472) Automatically add backport main label for release-pr (#2473) (#2475) update deps (#2468) (#2476) Implement client metrics interceptor for continuous benchmark job (#2477) (#2480) :chart_with_upwards_trend: Add client metrics panels for continuous benchmark job (#2481) (#2483) Update continuous benchmark docs (#2485) (#2486) Sync release/v1.7 to main (#2495) add read replica and rotator docs (#2497) (#2499) add reviewer guideline (#2507) (#2508) update large top-K ratio handling logic (#2509) (#2511) Change default image tag from latest to nightly (#2516) (#2518) Bugfix that caused an error when argument has 3 or more nil arguments (#2517) (#2520) add faiss in values.yaml & valdrelease.yaml (#2514) (#2519) capitalize faq (#2512) (#2522) Backport docs updates to release/v1.7 (#2521) [CI] Add workflow to synchronize ubuntu base image (#2526) (#2527) fix: update schedule (#2528) (#2530) refactor index manager service add index service API to expose index informations (#2525) (#2532) fix conflict bug (#2537) fix: make format (#2534) (#2540) Backport PR #2542, #2538 to release/v1.7 (#2543) fix: add checkout option (#2545) (#2546) Implement ngt Statistics API (#2539) (#2547) Add workflow to check git conflict for backport PR (#2548) (#2550) [create-pull-request] automated change (#2552) (#2556) Update dependencies, C++ standard, and improve Dockerfiles for better build systems and localization (#2549) (#2557) Backport #2559 (#2560) [BUGFIX] index correction process (#2565) (#2566) change external docker image reference to ghcr.io registry (#2567) (#2568) [patch] Release v1.7.13 (#2569) :bookmark: :robot: Release v1.7.13 (#2570) add HTTP2 support for http.Client and Vald HTTP Server (#2572) (#2575) Signed-off-by: kpango <[email protected]>

github-actions bot reviewed Feb 27, 2024

View reviewed changes

github-actions bot added area/discoverer area/internal type/ci area/helm area/makefile size/XXXL priority/low labels Feb 27, 2024

kpango approved these changes Feb 27, 2024

View reviewed changes

Merge branch 'release/v1.7' into backport/release/v1.7/feature/index/…

2b18488

…operator

github-actions bot reviewed Feb 27, 2024

View reviewed changes

kpango merged commit 012a3d3 into release/v1.7 Feb 27, 2024
159 checks passed

kpango deleted the backport/release/v1.7/feature/index/operator branch February 27, 2024 10:54

This was referenced Mar 1, 2024

Monthly review time metrics report: 2024-02-01..2024-02-29 ykadowak/actions-test#11

Closed

Monthly review time metrics report: 2024-02-01..2024-02-29 #2429

Closed

This was referenced Aug 1, 2024

[patch] Release v1.7.3 #2558

Closed

[patch] Release v1.7.13 #2561

Closed

[patch] Release v1.7.13 #2562

Closed

[patch] Release v1.7.13 #2569

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport PR #2375 to release/v1.7 for Add `index-operator` template implementation #2424

Backport PR #2375 to release/v1.7 for Add `index-operator` template implementation #2424

vdaas-ci commented Feb 27, 2024

cloudflare-workers-and-pages bot commented Feb 27, 2024 •

edited

Loading

github-actions bot Feb 27, 2024

vdaas-ci commented Feb 27, 2024

vdaas-ci commented Feb 27, 2024

github-actions bot Feb 27, 2024

github-actions bot Feb 27, 2024

github-actions bot Feb 27, 2024

github-actions bot Feb 27, 2024

codecov bot commented Feb 27, 2024



		# skipcq: DOK-DL3008
		RUN apt-get update && apt-get install -y --no-install-recommends \

		@@ -0,0 +1,71 @@
		// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>

		@@ -0,0 +1,141 @@
		// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>

		@@ -0,0 +1,32 @@
		// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>

		@@ -0,0 +1,152 @@
		// Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>

Backport PR #2375 to release/v1.7 for Add index-operator template implementation #2424

Backport PR #2375 to release/v1.7 for Add index-operator template implementation #2424

Conversation

vdaas-ci commented Feb 27, 2024

Description

Related Issue

Versions

Checklist

Special notes for your reviewer

cloudflare-workers-and-pages bot commented Feb 27, 2024 • edited Loading

Deploying with Cloudflare Pages

github-actions bot Feb 27, 2024

Choose a reason for hiding this comment

vdaas-ci commented Feb 27, 2024

vdaas-ci commented Feb 27, 2024

github-actions bot Feb 27, 2024

Choose a reason for hiding this comment

github-actions bot Feb 27, 2024

Choose a reason for hiding this comment

github-actions bot Feb 27, 2024

Choose a reason for hiding this comment

github-actions bot Feb 27, 2024

Choose a reason for hiding this comment

codecov bot commented Feb 27, 2024

Codecov Report

Backport PR #2375 to release/v1.7 for Add `index-operator` template implementation #2424

Backport PR #2375 to release/v1.7 for Add `index-operator` template implementation #2424

cloudflare-workers-and-pages bot commented Feb 27, 2024 •

edited

Loading