Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <[email protected]> * ✨ add load hdf5 functions Signed-off-by: vankichi <[email protected]> * ♻️ fix format Signed-off-by: vankichi <[email protected]> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <[email protected]> * ♻️ fix Signed-off-by: vankichi <[email protected]> * ♻️ refactor benchmark job Signed-off-by: vankichi <[email protected]> * ♻️ fix Signed-off-by: vankichi <[email protected]> * ♻️ fix proto Signed-off-by: vankichi <[email protected]> * 💚 add benchmark job image build ci Signed-off-by: vankichi <[email protected]> * 💚 invest Signed-off-by: vankichi <[email protected]> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <[email protected]> * ♻️ apply code review Signed-off-by: vankichi <[email protected]> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <[email protected]> * ✨ apply from feedback Signed-off-by: vankichi <[email protected]> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <[email protected]> * ♻️ change directory path Signed-off-by: vankichi <[email protected]> Signed-off-by: vankichi <[email protected]> Co-authored-by: Yusuke Kato <[email protected]> Co-authored-by: Hiroto Funakoshi <[email protected]> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <[email protected]> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <[email protected]> * ✨ rename package names and add doc.go Signed-off-by: vankichi <[email protected]> * ✨ create runtime object Signed-off-by: vankichi <[email protected]> * Apply suggestions from code review Co-authored-by: Yusuke Kato <[email protected]> * ♻️ apply feedback Signed-off-by: vankichi <[email protected]> Signed-off-by: vankichi <[email protected]> Co-authored-by: Yusuke Kato <[email protected]> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <[email protected]> * ♻️ fix format & rename file Signed-off-by: vankichi <[email protected]> Signed-off-by: vankichi <[email protected]> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <[email protected]> * ✨ create benchmark operator framework Signed-off-by: vankichi <[email protected]> * ♻️ remove unness changes Signed-off-by: vankichi <[email protected]> Signed-off-by: vankichi <[email protected]> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <[email protected]> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <[email protected]> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <[email protected]> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <[email protected]> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <[email protected]> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <[email protected]> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <[email protected]> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <[email protected]> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <[email protected]> --------- Signed-off-by: vankichi <[email protected]> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <[email protected]> Co-authored-by: Yusuke Kato <[email protected]> Co-authored-by: Hiroto Funakoshi <[email protected]> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <[email protected]> * ♻️ apply feedback Signed-off-by: vankichi <[email protected]> --------- Signed-off-by: vankichi <[email protected]> Co-authored-by: Yusuke Kato <[email protected]> Co-authored-by: Hiroto Funakoshi <[email protected]> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
vdaas · Dec 1, 2023 · 9e1ee46 · 9e1ee46
1 parent 9ab6f31
commit 9e1ee46
Show file tree

Hide file tree

Showing 74 changed files with 1,291 additions and 262 deletions.
diff --git a/.github/workflows/dockers-benchmark-job-image.yml b/.github/workflows/dockers-benchmark-job-image.yml
@@ -1,5 +1,5 @@
 #
-# Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+# Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

diff --git a/apis/grpc/v1/benchmark/benchmark.pb.go b/apis/grpc/v1/benchmark/benchmark.pb.go
diff --git a/apis/grpc/v1/benchmark/benchmark_vtproto.pb.go b/apis/grpc/v1/benchmark/benchmark_vtproto.pb.go
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+// Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

diff --git a/apis/proto/v1/benchmark/benchmark.proto b/apis/proto/v1/benchmark/benchmark.proto
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+// Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

diff --git a/cmd/tools/benchmark/job/main.go b/cmd/tools/benchmark/job/main.go
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+// Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

diff --git a/cmd/tools/benchmark/job/sample.yaml b/cmd/tools/benchmark/job/sample.yaml
@@ -1,5 +1,5 @@
 #
-# Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+# Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

diff --git a/cmd/tools/benchmark/operator/main.go b/cmd/tools/benchmark/operator/main.go
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+// Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

diff --git a/go.mod b/go.mod
@@ -388,6 +388,7 @@ require (
 	go.opentelemetry.io/otel/trace v1.11.2
 	go.uber.org/automaxprocs v0.0.0-00010101000000-000000000000
 	go.uber.org/goleak v1.2.1
+	go.uber.org/ratelimit v0.2.0
 	go.uber.org/zap v1.24.0
 	gocloud.dev v0.0.0-00010101000000-000000000000
 	golang.org/x/net v0.18.0
@@ -420,6 +421,7 @@ require (
 	github.com/Masterminds/goutils v1.1.1 // indirect
 	github.com/Masterminds/semver v1.5.0 // indirect
 	github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b // indirect
+	github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/campoy/embedmd v1.0.0 // indirect
 	github.com/cenkalti/backoff/v4 v4.1.3 // indirect

diff --git a/go.sum b/go.sum
@@ -170,6 +170,8 @@ github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b h1:slYM766cy2nI3BwyR
 github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b/go.mod h1:1KcenG0jGWcpt8ov532z81sp/kMMUG485J2InIOyADM=
 github.com/akavel/rsrc v0.8.0/go.mod h1:uLoCtb9J+EyAqh+26kdrTgmzRBFPGOolLWKpdxkKq+c=
 github.com/akavel/rsrc v0.10.2/go.mod h1:uLoCtb9J+EyAqh+26kdrTgmzRBFPGOolLWKpdxkKq+c=
+github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 h1:MzBOUgng9orim59UnfUTLRjMpd09C5uEVQ6RPGeCaVI=
+github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129/go.mod h1:rFgpPQZYZ8vdbc+48xibu8ALc3yeyd64IhHS+PU6Yyg=
 github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
 github.com/andybalholm/stroke v0.0.0-20221221101821-bd29b49d73f0/go.mod h1:ccdDYaY5+gO+cbnQdFxEXqfy0RkoV25H3jLXUDNM3wg=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=

diff --git a/internal/config/benchmark.go b/internal/config/benchmark.go
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+// Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -17,33 +17,32 @@
 // Package config providers configuration type and load configuration logic
 package config
 
-import "github.com/vdaas/vald/internal/k8s/client"
-
 // BenchmarkJob represents the configuration for the internal benchmark search job.
 type BenchmarkJob struct {
-	Target             *BenchmarkTarget    `json:"target,omitempty" yaml:"target"`
-	Dataset            *BenchmarkDataset   `json:"dataset,omitempty" yaml:"dataset"`
-	Dimension          int                 `json:"dimension,omitempty" yaml:"dimension"`
-	Replica            int                 `json:"replica,omitempty" yaml:"replica"`
-	Repetition         int                 `json:"repetition,omitempty" yaml:"repetition"`
-	JobType            string              `json:"job_type,omitempty" yaml:"job_type"`
-	InsertConfig       *InsertConfig       `json:"insert_config,omitempty" yaml:"insert_config"`
-	UpdateConfig       *UpdateConfig       `json:"update_config,omitempty" yaml:"update_config"`
-	UpsertConfig       *UpsertConfig       `json:"upsert_config,omitempty" yaml:"upsert_config"`
-	SearchConfig       *SearchConfig       `json:"search_config,omitempty" yaml:"search_config"`
-	RemoveConfig       *RemoveConfig       `json:"remove_config,omitempty" yaml:"remove_config"`
-	ClientConfig       *GRPCClient         `json:"client_config,omitempty" yaml:"client_config"`
-	Rules              []*BenchmarkJobRule `json:"rules,omitempty" yaml:"rules"`
-	BeforeJobName      string              `json:"before_job_name,omitempty" yaml:"before_job_name"`
+	Target             *BenchmarkTarget    `json:"target,omitempty"               yaml:"target"`
+	Dataset            *BenchmarkDataset   `json:"dataset,omitempty"              yaml:"dataset"`
+	Dimension          int                 `json:"dimension,omitempty"            yaml:"dimension"`
+	Replica            int                 `json:"replica,omitempty"              yaml:"replica"`
+	Repetition         int                 `json:"repetition,omitempty"           yaml:"repetition"`
+	JobType            string              `json:"job_type,omitempty"             yaml:"job_type"`
+	InsertConfig       *InsertConfig       `json:"insert_config,omitempty"        yaml:"insert_config"`
+	UpdateConfig       *UpdateConfig       `json:"update_config,omitempty"        yaml:"update_config"`
+	UpsertConfig       *UpsertConfig       `json:"upsert_config,omitempty"        yaml:"upsert_config"`
+	SearchConfig       *SearchConfig       `json:"search_config,omitempty"        yaml:"search_config"`
+	RemoveConfig       *RemoveConfig       `json:"remove_config,omitempty"        yaml:"remove_config"`
+	ObjectConfig       *ObjectConfig       `json:"object_config,omitempty"        yaml:"object_config"`
+	ClientConfig       *GRPCClient         `json:"client_config,omitempty"        yaml:"client_config"`
+	Rules              []*BenchmarkJobRule `json:"rules,omitempty"                yaml:"rules"`
+	BeforeJobName      string              `json:"before_job_name,omitempty"      yaml:"before_job_name"`
 	BeforeJobNamespace string              `json:"before_job_namespace,omitempty" yaml:"before_job_namespace"`
-	Client             client.Client       `json:"client,omitempty" yaml:"client"`
+	RPS                int                 `json:"rps,omitempty"                  yaml:"rps"`
 }
 
 // BenchmarkScenario represents the configuration for the internal benchmark scenario.
 type BenchmarkScenario struct {
-	Target  *BenchmarkTarget  `json:"target" yaml:"target"`
-	Dataset *BenchmarkDataset `jon:"dataset" yaml:"dataset"`
-	Jobs    []*BenchmarkJob   `job:"jobs" yaml:jobs`
+	Target  *BenchmarkTarget  `json:"target,omitempty"  yaml:"target"`
+	Dataset *BenchmarkDataset `json:"dataset,omitempty" yaml:"dataset"`
+	Jobs    []*BenchmarkJob   `json:"jobs,omitempty"    yaml:"jobs"`
 }
 
 // BenchmarkTarget defines the desired state of BenchmarkTarget
@@ -52,6 +51,11 @@ type BenchmarkTarget struct {
 	Port int    `json:"port,omitempty"`
 }
 
+func (t *BenchmarkTarget) Bind() *BenchmarkTarget {
+	t.Host = GetActualValue(t.Host)
+	return t
+}
+
 // BenchmarkDataset defines the desired state of BenchmarkDateset
 type BenchmarkDataset struct {
 	Name    string                 `json:"name,omitempty"`
@@ -60,6 +64,11 @@ type BenchmarkDataset struct {
 	Range   *BenchmarkDatasetRange `json:"range,omitempty"`
 }
 
+func (d *BenchmarkDataset) Bind() *BenchmarkDataset {
+	d.Name = GetActualValue(d.Name)
+	return d
+}
+
 // BenchmarkDatasetRange defines the desired state of BenchmarkDatesetRange
 type BenchmarkDatasetRange struct {
 	Start int `json:"start,omitempty"`
@@ -72,22 +81,45 @@ type BenchmarkJobRule struct {
 	Type string `json:"type,omitempty"`
 }
 
+func (r *BenchmarkJobRule) Bind() *BenchmarkJobRule {
+	r.Name = GetActualValue(r.Name)
+	r.Type = GetActualValue(r.Type)
+	return r
+}
+
 // InsertConfig defines the desired state of insert config
 type InsertConfig struct {
 	SkipStrictExistCheck bool   `json:"skip_strict_exist_check,omitempty"`
 	Timestamp            string `json:"timestamp,omitempty"`
 }
 
+func (cfg *InsertConfig) Bind() *InsertConfig {
+	cfg.Timestamp = GetActualValue(cfg.Timestamp)
+	return cfg
+}
+
 // UpdateConfig defines the desired state of update config
 type UpdateConfig struct {
-	SkipStrictExistCheck bool   `json:"skip_strict_exist_check,omitempty"`
-	Timestamp            string `json:"timestamp,omitempty"`
+	SkipStrictExistCheck  bool   `json:"skip_strict_exist_check,omitempty"`
+	Timestamp             string `json:"timestamp,omitempty"`
+	DisableBalancedUpdate bool   `json:"disable_balanced_update,omitempty"`
+}
+
+func (cfg *UpdateConfig) Bind() *UpdateConfig {
+	cfg.Timestamp = GetActualValue(cfg.Timestamp)
+	return cfg
 }
 
 // UpsertConfig defines the desired state of upsert config
 type UpsertConfig struct {
-	SkipStrictExistCheck bool   `json:"skip_strict_exist_check,omitempty"`
-	Timestamp            string `json:"timestamp,omitempty"`
+	SkipStrictExistCheck  bool   `json:"skip_strict_exist_check,omitempty"`
+	Timestamp             string `json:"timestamp,omitempty"`
+	DisableBalancedUpdate bool   `json:"disable_balanced_update,omitempty"`
+}
+
+func (cfg *UpsertConfig) Bind() *UpsertConfig {
+	cfg.Timestamp = GetActualValue(cfg.Timestamp)
+	return cfg
 }
 
 // SearchConfig defines the desired state of search config
@@ -99,19 +131,93 @@ type SearchConfig struct {
 	Timeout string  `json:"timeout,omitempty"`
 }
 
+func (cfg *SearchConfig) Bind() *SearchConfig {
+	cfg.Timeout = GetActualValue(cfg.Timeout)
+	return cfg
+}
+
 // RemoveConfig defines the desired state of remove config
 type RemoveConfig struct {
 	SkipStrictExistCheck bool   `json:"skip_strict_exist_check,omitempty"`
 	Timestamp            string `json:"timestamp,omitempty"`
 }
 
+func (cfg *RemoveConfig) Bind() *RemoveConfig {
+	cfg.Timestamp = GetActualValue(cfg.Timestamp)
+	return cfg
+}
+
+// ObjectConfig defines the desired state of object config
+type ObjectConfig struct {
+	FilterConfig FilterConfig `json:"filter_config,omitempty" yaml:"filter_config"`
+}
+
+func (cfg *ObjectConfig) Bind() *ObjectConfig {
+	cfg.FilterConfig = *cfg.FilterConfig.Bind()
+	return cfg
+}
+
+// FilterTarget defines the desired state of filter target
+type FilterTarget struct {
+	Host string `json:"host,omitempty" yaml:"host"`
+	Port int32  `json:"port,omitempty" yaml:"port"`
+}
+
+func (cfg *FilterTarget) Bind() *FilterTarget {
+	cfg.Host = GetActualValue(cfg.Host)
+	return cfg
+}
+
+// FilterConfig defines the desired state of filter config
+type FilterConfig struct {
+	Targets []*FilterTarget `json:"target,omitempty" yaml:"target"`
+}
+
+func (cfg *FilterConfig) Bind() *FilterConfig {
+	for i := 0; i < len(cfg.Targets); i++ {
+		cfg.Targets[i] = cfg.Targets[i].Bind()
+	}
+	return cfg
+}
+
 // Bind binds the actual data from the Job receiver fields.
 func (b *BenchmarkJob) Bind() *BenchmarkJob {
 	b.JobType = GetActualValue(b.JobType)
+	b.BeforeJobName = GetActualValue(b.BeforeJobName)
+	b.BeforeJobNamespace = GetActualValue(b.BeforeJobNamespace)
 
+	if b.Target != nil {
+		b.Target = b.Target.Bind()
+	}
+	if b.Dataset != nil {
+		b.Dataset = b.Dataset.Bind()
+	}
+	if b.InsertConfig != nil {
+		b.InsertConfig = b.InsertConfig.Bind()
+	}
+	if b.UpdateConfig != nil {
+		b.UpdateConfig = b.UpdateConfig.Bind()
+	}
+	if b.UpsertConfig != nil {
+		b.UpsertConfig = b.UpsertConfig.Bind()
+	}
+	if b.SearchConfig != nil {
+		b.SearchConfig = b.SearchConfig.Bind()
+	}
+	if b.RemoveConfig != nil {
+		b.RemoveConfig = b.RemoveConfig.Bind()
+	}
+	if b.ObjectConfig != nil {
+		b.ObjectConfig = b.ObjectConfig.Bind()
+	}
 	if b.ClientConfig != nil {
 		b.ClientConfig = b.ClientConfig.Bind()
 	}
+	if len(b.Rules) > 0 {
+		for i := 0; i < len(b.Rules); i++ {
+			b.Rules[i] = b.Rules[i].Bind()
+		}
+	}
 	return b
 }
 

diff --git a/internal/errors/benchmark.go b/internal/errors/benchmark.go
@@ -29,4 +29,9 @@ var (
 	ErrFailedToCreateJob = func(err error, jn string) error {
 		return Wrapf(err, "could not create job: %s ", jn)
 	}
+
+	// ErrMismatchBenchmarkAtomics represents a function to generate an error that mismatch each atomic.Pointer stored corresponding to benchmark tasks.
+	ErrMismatchBenchmarkAtomics = func(job, benchjob, benchscenario interface{}) error {
+		return Errorf("mismatch atomics: job=%v\tbenchjob=%v\tbenchscenario=%v", job, benchjob, benchscenario)
+	}
 )
diff --git a/internal/k8s/crd/benchmark/valdbenchmarkjob.yaml b/internal/k8s/crd/benchmark/valdbenchmarkjob.yaml
@@ -1,5 +1,5 @@
 #
-# Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+# Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -129,6 +129,8 @@ spec:
                       - type
                     type: object
                   type: array
+                rps:
+                  type: integer
                 insert_config:
                   description: InsertConfig defines the desired state of insert config
                   properties:
@@ -168,6 +170,8 @@ spec:
                       type: boolean
                     timestamp:
                       type: string
+                    disable_balanced_update:
+                      type: boolean
                   type: object
                 upsert_config:
                   description: UpsertConfig defines the desired state of upsert config
@@ -176,6 +180,8 @@ spec:
                       type: boolean
                     timestamp:
                       type: string
+                    disable_balanced_update:
+                      type: boolean
                   type: object
                 client_config:
                   description: ClientConfig represents the configurations for gRPC client.

diff --git a/internal/k8s/crd/benchmark/valdbenchmarkscenario.yaml b/internal/k8s/crd/benchmark/valdbenchmarkscenario.yaml
@@ -1,5 +1,5 @@
 #
-# Copyright (C) 2019-2022 vdaas.org vald team <[email protected]>
+# Copyright (C) 2019-2023 vdaas.org vald team <[email protected]>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -167,6 +167,8 @@ spec:
                             - type
                           type: object
                         type: array
+                      rps:
+                        type: integer
                       insert_config:
                         description: InsertConfig defines the desired state of insert config
                         properties:
@@ -206,6 +208,8 @@ spec:
                             type: boolean
                           timestamp:
                             type: string
+                          disable_balanced_update:
+                            type: boolean
                         type: object
                       upsert_config:
                         description: UpsertConfig defines the desired state of upsert config
@@ -214,6 +218,8 @@ spec:
                             type: boolean
                           timestamp:
                             type: string
+                          disable_balanced_update:
+                            type: boolean
                         type: object
                       client_config:
                         description: ClientConfig represents the configurations for gRPC client.