Refactor helm template and operator logic

[vankichi]

Description:

This PR includes refactoring benchmark operator logic and adding some new benchmark specs.

Change benchmark operator logic

• Enable delete job pods when updating ValdBenchmarkJobResource
  • Due to the operator creating job resources using controller-runtime, the job pod is set to blockOwnerDeletion=true in owner reference. To enable deleting job pods when job resources are deleted, I have added the DeletePropagationBackground to the deleting option.

Adding setting parameters
To be a more customizable continuous benchmark tool, I have added the new parameters as followings:

• enable_linear_search: user can select whether to use a linear search process to calculate search recall when executing the search operation. linear search operation may require much time for execution, so it should be set as false when a user wants only to track search operation latency.
• ttl_seconds_after_finished: user can set ttl_seconds_after_finished for each benchmark job resource. Completed job pods will be deleted when this value is set. On the other hand, pods will remain when it is not set.
• dataset.url: When the dataset URL is set and dataset.name is original, user can execute benchmark jobs with a user-defined HDF5 dataset, which satisfies the defined dataset group.
• concurrencyLimit: It controls the goroutine limit for executing the benchmark job. When the number of requests is large, it takes much time and requires CPU/Memory resources to complete all goroutines. I added this setting to save OOMKill or some failure based on resource failure.
• search_config.aggregation_algorithm: aggregation_algorithm is the new feature enabled from v1.7.6.

Enable user-defined benchmark job image for each operator
Allowing user-defined benchmark job image for each benchmark operator by setting job_image at the vald-benchmark-operator/values.yaml

Related Issue:

Versions:

• Go Version: 1.20.3
• Docker Version: 20.10.8
• Kubernetes Version: 1.22.0
• NGT Version: 2.0.11

Checklist:

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 💌 /changelog - replace the PR body by changelog details
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• ⏪ /rebase - rebase main
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[vdaas-ci]

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

[codecov]

Codecov Report

Patch coverage: 18.42% and project coverage change: +0.10 🎉

Comparison is base (2190111) 29.66% compared to head (fce3307) 29.76%.

❗ Current head fce3307 differs from pull request most recent head 34d2101. Consider uploading reports for the commit 34d2101 to get more accurate results

Additional details and impacted files

@@                       Coverage Diff                        @@
##           feature/continuous-benchmark    #2043      +/-   ##
================================================================
+ Coverage                         29.66%   29.76%   +0.10%     
================================================================
  Files                               374      374              
  Lines                             35430    35446      +16     
================================================================
+ Hits                              10511    10552      +41     
+ Misses                            24429    24407      -22     
+ Partials                            490      487       -3     

Impacted Files	Coverage Δ
internal/config/benchmark.go	0.00% <0.00%> (ø)
pkg/tools/benchmark/job/config/config.go	8.47% <0.00%> (-0.30%)	⬇️
internal/test/data/hdf5/hdf5.go	9.03% <12.00%> (+3.29%)	⬆️
internal/test/data/hdf5/option.go	70.83% <100.00%> (+60.83%)	⬆️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	34d2101
Status:	✅  Deploy successful!
Preview URL:	https://5af0c576.vald.pages.dev
Branch Preview URL:	https://refactor-internal-charts-pkg.vald.pages.dev

View logs

[sonatype-lift]

typecheck: undefined: atomic.Pointer

❗❗ 2 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path	Line Number
pkg/tools/benchmark/operator/service/operator.go	60
pkg/tools/benchmark/operator/service/operator.go	61

Visit the Lift Web Console to find more details in your report.

---

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command	Usage
@sonatype-lift ignore	Leave out the above finding from this PR
@sonatype-lift ignoreall	Leave out all the existing findings from this PR
@sonatype-lift exclude <file|issue|path|tool>	Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

[github-actions]

🚫 [golangci] _{reported by reviewdog 🐶}
ContainerImage is a global variable (gochecknoglobals)

[github-actions]

🚫 [golangci] _{reported by reviewdog 🐶}
ImagePullPolicy is a global variable (gochecknoglobals)

[github-actions]

🚫 [golangci] _{reported by reviewdog 🐶}
ST1020: comment on exported function WithJobImage should be of the form "WithJobImage ..." (stylecheck)