Implement continuous benchmark tool

[vankichi]

This is a re-created PR instead of #1951

Description:

I have implemented the continuous benchmark tool for verifying the Vald cluster.
This PR includes two new components, vald-benchmark-operator and vald-benchmark-job.
The continuous benchmark tool is designed for the vald-benchmark-operator, which operates benchmark job tasks, running on vald-benchmark-job, using user-defined Kubernetes Custom Resouce (CR).
There are two CRs for running continuous benchmarks, ValdBenchmarkScenario and ValdBenchmarkJob.
ValdBenchmarkJob defines the one benchmark job, and ValdBenchmarkScenario defines the scenario with multiple benchmark jobs.
User can use it properly in their situation.

The diagram below shows the sequence diagram of continuous benchmark tools.

sequenceDiagram
	participant user
	participant operator
	participant k8s
	user->>k8s: update scenario.yaml
  operator->>k8s: reconcile scenario.yaml
  operator->>k8s: delete current working benchmark job resources
	Note right of k8s: delete current benchmark job resource<br>jobs are also deleted automatically.
  operator->>k8s: apply new benchmark job resources
	Note right of k8s: operator should create Flush job at first
  operator->>k8s: reconcile benchmark job resources
	operator->>k8s: apply job resource
  loop monitoring
		operator->>k8s: reconcile resources
	end
	k8s->>k8s: Executing job
	operator->>operator: check state is safe
	Note right of operator: Show error if error is not nil

Loading

The diagram below shows the operator's update of each CR status diagram.

sequenceDiagram
	autonumber
	participant user
	participant operator
	participant benchmarkScenario
	participant benchmarkJob
	participant job
	user->>benchmarkScenario: apply scenario 
	benchmarkScenario->>operator: reconcile scenario
	operator-->operator: create benchmarkJob resource
	operator->>benchmarkJob: apply benchmarkJob resources
	benchmarkJob->>operator: reconcile benchmark
	operator->>+job: create job
	Note right of job: execute job
	loop monitoring
		operator->>benchmarkScenario: reconcile resource 
		operator->>benchmarkJob: reconcile resource
		operator->>job: reconcile job
	end
	job->>-operator: reconcile job with status is `COMPLETED`
	Note right of job: end job
	operator->>benchmarkJob: update status with `BenchmarkJobCompleted`
	benchmarkJob->>operator: reconcile benchmarkJob
	operator-->operator: check each resource state
	Note over operator,benchmarkScenario: If all status of benchmarkJob in benchmarkScenario is COMPLETED
	operator->>benchmarkScenario: updater status with `BenchmarkScenarioCompleted`
	benchmarkScenario->>operator: reconcile scenario

Loading

This PR has many added files, but you DO NOT have to check all of them.
Please check the following files intensively:

• internal/config/*.go: config files
• pkg/tools/benchmark/operator/service/operator.go: operating logic with CRs.
• pkg/tools/benchmark/job/service/job.go: main logic for running each benchmark job.

Related Issue:

Versions:

• Go Version: 1.20
• Docker Version: 20.10.8
• Kubernetes Version: 1.22.0
• NGT Version: 2.0.9

Checklist:

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

[vdaas-ci]

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

golangci

pkg/tools/benchmark/job/service/search.go|34 col 10| payload.Search_Config is missing fields RequestId, IngressFilters, EgressFilters (exhaustruct)
pkg/tools/benchmark/job/service/update.go|35 col 10| payload.Update_Config is missing fields Filters, Timestamp (exhaustruct)
pkg/tools/benchmark/job/service/update.go|67 col 14| payload.Object_Vector is missing field Timestamp (exhaustruct)
pkg/tools/benchmark/job/service/upsert.go|35 col 10| payload.Upsert_Config is missing fields Filters, Timestamp (exhaustruct)
pkg/tools/benchmark/job/service/upsert.go|67 col 14| payload.Object_Vector is missing field Timestamp (exhaustruct)
internal/k8s/job/job.go|56 col 8| job.reconciler is missing fields mgr, name, namespaces, onError, onReconcile, listOpts (exhaustruct)
internal/k8s/vald/benchmark/job/job.go|42 col 19| scheme.Builder is missing field SchemeBuilder (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|82 col 3| v1.Container is missing fields Command, Args, WorkingDir, EnvFrom, Resources, ResizePolicy, VolumeMounts, VolumeDevices, ReadinessProbe, Lifecycle, TerminationMessagePath, TerminationMessagePolicy, SecurityContext, Stdin, StdinOnce, TTY (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|86 col 20| v1.Probe is missing fields SuccessThreshold, FailureThreshold, TerminationGracePeriodSeconds (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|90 col 19| v1.ProbeHandler is missing fields HTTPGet, TCPSocket, GRPC (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|99 col 19| v1.Probe is missing fields InitialDelaySeconds, SuccessThreshold, TerminationGracePeriodSeconds (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|103 col 19| v1.ProbeHandler is missing fields HTTPGet, TCPSocket, GRPC (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|113 col 5| v1.ContainerPort is missing fields HostPort, HostIP (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|118 col 5| v1.ContainerPort is missing fields HostPort, HostIP (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|125 col 5| v1.EnvVar is missing field Value (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|127 col 18| v1.EnvVarSource is missing fields ResourceFieldRef, ConfigMapKeyRef, SecretKeyRef (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|128 col 18| v1.ObjectFieldSelector is missing field APIVersion (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|133 col 5| v1.EnvVar is missing field Value (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|135 col 18| v1.EnvVarSource is missing fields ResourceFieldRef, ConfigMapKeyRef, SecretKeyRef (exhaustruct)
internal/k8s/vald/benchmark/job/job_template.go|136 col 18| v1.ObjectFieldSelector is missing field APIVersion (exhaustruct)
internal/test/mock/controller_runtime.go|37 col 23| mock.MockSubResourceWriter is missing field SubResourceWriter (exhaustruct)
internal/test/mock/controller_runtime.go|64 col 12| mock.MockClient is missing field Client (exhaustruct)
cmd/tools/benchmark/job/main.go|53 col 24| type assertion must be checked (forcetypeassert)
internal/k8s/job/job.go|103 col 2| type assertion must be checked (forcetypeassert)
cmd/tools/benchmark/operator/main.go|52 col 24| type assertion must be checked (forcetypeassert)
internal/k8s/vald/benchmark/api/v1/info.go|26 col 2| GroupVersion is a global variable (gochecknoglobals)
internal/k8s/vald/benchmark/api/v1/info.go|30 col 2| AddToScheme is a global variable (gochecknoglobals)
pkg/tools/benchmark/job/service/option.go|34 col 5| defaultOpts is a global variable (gochecknoglobals)
pkg/tools/benchmark/job/handler/grpc/option.go|22 col 5| defaultOpts is a global variable (gochecknoglobals)
pkg/tools/benchmark/job/handler/rest/option.go|22 col 5| defaultOpts is a global variable (gochecknoglobals)
internal/k8s/vald/benchmark/job/job.go|40 col 2| GroupVersion is a global variable (gochecknoglobals)
internal/k8s/vald/benchmark/job/job.go|44 col 2| AddToScheme is a global variable (gochecknoglobals)
internal/k8s/vald/benchmark/job/job_template_option.go|27 col 5| defaultBenchmarkJobTplOpts is a global variable (gochecknoglobals)
internal/k8s/vald/benchmark/job/job_template_option.go|67 col 5| defaultBenchmarkJobOpts is a global variable (gochecknoglobals)
pkg/tools/benchmark/operator/usecase/benchmarkd.go|52 col 5| JOB_NAMESPACE is a global variable (gochecknoglobals)
internal/timeutil/rate/rate.go|38 col 30| mnd: Magic number: 32, in detected (gomnd)
pkg/tools/benchmark/job/service/job.go|253 col 26| mnd: Magic number: 3, in detected (gomnd)
pkg/tools/benchmark/job/service/option.go|36 col 16| mnd: Magic number: 748, in detected (gomnd)
pkg/tools/benchmark/job/service/option.go|38 col 10| mnd: Magic number: 1000, in detected (gomnd)
pkg/tools/benchmark/job/service/option.go|39 col 22| mnd: Magic number: 200, in detected (gomnd)
pkg/tools/benchmark/job/usecase/benchmarkd.go|194 col 26| mnd: Magic number: 3, in detected (gomnd)
internal/k8s/job/job.go|91 col 37| mnd: Magic number: 100, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|87 col 32| mnd: Magic number: 60, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|88 col 32| mnd: Magic number: 10, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|89 col 32| mnd: Magic number: 300, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|100 col 29| mnd: Magic number: 30, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|101 col 29| mnd: Magic number: 10, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|102 col 29| mnd: Magic number: 300, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|116 col 27| mnd: Magic number: 3000, in detected (gomnd)
internal/k8s/vald/benchmark/job/job_template.go|121 col 27| mnd: Magic number: 3001, in detected (gomnd)
internal/k8s/vald/benchmark/job/job.go|90 col 37| mnd: Magic number: 100, in detected (gomnd)
internal/k8s/vald/benchmark/scenario/scenario.go|79 col 37| mnd: Magic number: 100, in detected (gomnd)
pkg/tools/benchmark/operator/usecase/benchmarkd.go|153 col 26| mnd: Magic number: 3, in detected (gomnd)
internal/config/config_test.go|1435 col 1| Function name: TestMerge, Cyclomatic Complexity: 8, Halstead Volume: 6956.07, Maintainability Index: 16 (maintidx)
internal/config/config_test.go|1436 col 7| config should be annotated with the json tag as it is passed to json.Marshal at internal/config/config_test.go:1461:13 (musttag)
internal/k8s/job/job.go|100 col 3| naked return in func Reconcile with 47 lines of code (nakedret)
internal/k8s/job/job.go|127 col 2| naked return in func Reconcile with 47 lines of code (nakedret)
internal/k8s/vald/benchmark/job/job.go|99 col 3| naked return in func Reconcile with 37 lines of code (nakedret)
internal/k8s/vald/benchmark/job/job.go|111 col 2| naked return in func Reconcile with 37 lines of code (nakedret)
internal/k8s/vald/benchmark/scenario/scenario.go|88 col 3| naked return in func Reconcile with 37 lines of code (nakedret)
internal/test/data/hdf5/hdf5.go|59 col 6| ST1003: type DatasetUrl should be DatasetURL (stylecheck)
internal/test/data/hdf5/hdf5.go|62 col 2| ST1003: const FashionMNIST784EuclideanUrl should be FashionMNIST784EuclideanURL (stylecheck)

[codecov]

Codecov Report

Attention: 566 lines in your changes are missing coverage. Please review.

Comparison is base (ac50bf3) 29.96% compared to head (ac1c82e) 30.31%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
pkg/tools/benchmark/operator/service/operator.go	64.48%	160 Missing and 19 partials ⚠️
internal/test/data/hdf5/hdf5.go	9.03%	155 Missing and 6 partials ⚠️
internal/config/benchmark.go	0.00%	82 Missing ⚠️
pkg/tools/benchmark/job/config/config.go	8.33%	55 Missing ⚠️
internal/config/config.go	57.27%	34 Missing and 13 partials ⚠️
internal/errors/config.go	0.00%	12 Missing ⚠️
hack/license/gen/main.go	21.42%	10 Missing and 1 partial ⚠️
internal/errors/benchmark.go	0.00%	9 Missing ⚠️
internal/test/data/hdf5/option.go	70.83%	6 Missing and 1 partial ⚠️
internal/k8s/reconciler.go	0.00%	2 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2216      +/-   ##
==========================================
+ Coverage   29.96%   30.31%   +0.34%     
==========================================
  Files         371      378       +7     
  Lines       36165    37197    +1032     
==========================================
+ Hits        10838    11277     +439     
- Misses      24814    25363     +549     
- Partials      513      557      +44     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	ac1c82e
Status:	✅  Deploy successful!
Preview URL:	https://18e8d2da.vald.pages.dev
Branch Preview URL:	https://feature-continuous-benchmark.vald.pages.dev

View logs

[ykadowak]

Please resolve conflicts. For internal/k8s/client/**, you can just use the main versinon since I didn't change your part. Sorry for the inconvenience.

[hlts2]

LGTM!