Skip to content

Commit

Permalink
Refactor helm template and operator logic (#2043)
Browse files Browse the repository at this point in the history
* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* ✨ create bench job search tools

Signed-off-by: vankichi <[email protected]>

* ✨ add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format

Signed-off-by: vankichi <[email protected]>

* ♻️ fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ fix proto

Signed-off-by: vankichi <[email protected]>

* 💚 add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* 💚 invest

Signed-off-by: vankichi <[email protected]>

* Revert "💚 invest"

This reverts commit f0f585c.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* ✨ apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* ✨ add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* ✨ add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* ✨ rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* ✨ create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* ✨ ♻️ add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* ✨ impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* ✨ create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* ♻️ remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* ✨ impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* 🐛 Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ✨ impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* ✨ create bench job search tools

Signed-off-by: vankichi <[email protected]>

* ✨ add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format

Signed-off-by: vankichi <[email protected]>

* ♻️ fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ fix proto

Signed-off-by: vankichi <[email protected]>

* 💚 add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* 💚 invest

Signed-off-by: vankichi <[email protected]>

* Revert "💚 invest"

This reverts commit f0f585c.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* ✨ apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* ✨ add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* ✨ add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* ✨ rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* ✨ create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* ✨ ♻️ add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* ✨ impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* ✨ create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* ♻️ remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* ✨ impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* 🐛 Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ✨ impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ♻️ Refactor helm template and operator logic

Signed-off-by: vankichi <[email protected]>

* ♻️ Add download original dataset URL option

Signed-off-by: vankichi <[email protected]>

* ♻️ Set docker image location at the benchmark operator configmap and use it when information is set

Signed-off-by: vankichi <[email protected]>

* add search algorithm benchmark and update search aggregation algo

Signed-off-by: kpango <[email protected]>

* ✨ Add search result aggregation option

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* Improve job performance (#2061)

* 🐛 Fix job function to apply rate limiter

* ♻️ Add pyroscope setting

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix build error

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix docker file and add concurrencyLimit for job goroutine

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix job_template.go by feedback

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix job logic by feedback

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
  • Loading branch information
4 people committed Dec 7, 2023
1 parent b772b7f commit 96a976d
Show file tree
Hide file tree
Showing 49 changed files with 1,510 additions and 470 deletions.
1 change: 1 addition & 0 deletions .github/workflows/dockers-benchmark-job-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ on:
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "!internal/k8s/**"
- "apis/grpc/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
Expand Down
26 changes: 25 additions & 1 deletion charts/vald-benchmark-operator/crds/valdbenchmarkjob.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,10 @@ spec:
type: string
wait_for_ready:
type: boolean
concurrency_limit:
type: integer
maximum: 65535
minimum: 0
dataset:
type: object
properties:
Expand All @@ -252,6 +256,7 @@ spec:
name:
type: string
enum:
- original
- fashion-mnist
range:
type: object
Expand All @@ -262,6 +267,11 @@ spec:
start:
type: integer
minimum: 1
required:
- start
- end
url:
type: string
required:
- name
- indexes
Expand All @@ -285,7 +295,7 @@ spec:
- upsert
- search
- remove
- get_object
- getobject
- exists
object_config:
type: object
Expand Down Expand Up @@ -319,6 +329,16 @@ spec:
search_config:
type: object
properties:
aggregation_algorithm:
type: string
enum:
- Unknown
- ConcurrentQueue
- SortSlice
- SortPoolSlice
- PairingHeap
enable_linear_search:
type: boolean
epsilon:
type: number
min_num:
Expand All @@ -342,6 +362,10 @@ spec:
required:
- host
- port
ttl_seconds_after_finished:
type: integer
maximum: 65535
minimum: 0
update_config:
type: object
properties:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,19 @@ spec:
type: string
tag:
type: string
job_image:
type: object
properties:
pullPolicy:
type: string
enum:
- Always
- Never
- IfNotPresent
repository:
type: string
tag:
type: string
logging:
type: object
properties:
Expand Down
10 changes: 8 additions & 2 deletions charts/vald-benchmark-operator/crds/valdbenchmarkscenario.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ spec:
plural: valdbenchmarkscenarios
singular: valdbenchmarkscenario
shortNames:
- vbo
- vbos
- vbs
- vbss
scope: Namespaced
versions:
- name: v1
Expand Down Expand Up @@ -74,6 +74,7 @@ spec:
name:
type: string
enum:
- original
- fashion-mnist
range:
type: object
Expand All @@ -84,6 +85,11 @@ spec:
start:
type: integer
minimum: 1
required:
- start
- end
url:
type: string
required:
- name
- indexes
Expand Down
38 changes: 35 additions & 3 deletions charts/vald-benchmark-operator/job-values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,12 @@
"wait_for_ready": { "type": "boolean" }
}
},
"concurrency_limit": {
"type": "integer",
"description": "concurrency_limit represents the goroutine limit count. It affects the job performance.",
"maximum": 65535,
"minimum": 0
},
"dataset": {
"type": "object",
"description": "dataset information",
Expand All @@ -305,7 +311,7 @@
"name": {
"type": "string",
"description": "the name of dataset",
"enum": ["fashion-mnist"]
"enum": ["original", "fashion-mnist"]
},
"range": {
"type": "object",
Expand All @@ -321,7 +327,12 @@
"description": "start index number",
"minimum": 1
}
}
},
"required": ["start", "end"]
},
"url": {
"type": "string",
"description": "the dataset url which is used for executing benchmark job with user defined hdf5 file"
}
},
"required": ["name", "indexes", "group", "range"]
Expand Down Expand Up @@ -351,7 +362,7 @@
"upsert",
"search",
"remove",
"get_object",
"getobject",
"exists"
]
},
Expand Down Expand Up @@ -404,6 +415,21 @@
"type": "object",
"description": "upsert config",
"properties": {
"aggregation_algorithm": {
"type": "string",
"description": "search result aggregation algorithm",
"enum": [
"Unknown",
"ConcurrentQueue",
"SortSlice",
"SortPoolSlice",
"PairingHeap"
]
},
"enable_linear_search": {
"type": "boolean",
"description": "enable linear search for calculation recall"
},
"epsilon": { "type": "number", "description": "epsilon" },
"min_num": {
"type": "integer",
Expand Down Expand Up @@ -435,6 +461,12 @@
},
"required": ["host", "port"]
},
"ttl_seconds_after_finished": {
"type": "integer",
"description": "limits the lifetime of a Job that has finished execution.",
"maximum": 65535,
"minimum": 0
},
"update_config": {
"type": "object",
"description": "update config",
Expand Down
9 changes: 7 additions & 2 deletions charts/vald-benchmark-operator/scenario-values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"name": {
"type": "string",
"description": "the name of dataset",
"enum": ["fashion-mnist"]
"enum": ["original", "fashion-mnist"]
},
"range": {
"type": "object",
Expand All @@ -36,7 +36,12 @@
"description": "start index number",
"minimum": 1
}
}
},
"required": ["start", "end"]
},
"url": {
"type": "string",
"description": "the dataset url which is used for executing benchmark job with user defined hdf5 file"
}
},
"required": ["name", "indexes", "group", "range"]
Expand Down
21 changes: 18 additions & 3 deletions charts/vald-benchmark-operator/schemas/job-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ target:
# @schema {"name": "dataset", "type": "object", "required": ["name", "indexes", "group", "range"]}
# dataset -- dataset information
dataset:
# @schema {"name": "dataset.name", "type": "string", "enum": ["fashion-mnist"] }
# @schema {"name": "dataset.name", "type": "string", "enum": ["original", "fashion-mnist"] }
# dataset.name -- the name of dataset
name: "fashion-mnist"
# @schema {"name": "dataset.indexes", "type": "integer", "minimum": 0}
Expand All @@ -36,7 +36,7 @@ dataset:
# @schema {"name": "dataset.group", "type": "string", "minLength": 1}
# dataset.group -- the hdf5 group name of dataset
group: "test"
# @schema {"name": "dataset.range", "type": "object", "range": ["start", "port"]}
# @schema {"name": "dataset.range", "type": "object", "required": ["start", "end"]}
# dataset.range -- the data range of indexes
range:
# @schema {"name": "dataset.range.start", "type": "integer", "minimum": 1}
Expand All @@ -45,6 +45,9 @@ dataset:
# @schema {"name": "dataset.range.end", "type": "integer", "minimum": 1}
# dataset.range.end -- end index number
end: 1000
# @schema {"name": "dataset.url", "type": "string"}
# dataset.url -- the dataset url which is used for executing benchmark job with user defined hdf5 file
url: ""
# @schema {"name": "dimension", "type": "integer", "minimum": 1}
# dimension -- vector dimension
dimension: 784
Expand All @@ -54,7 +57,7 @@ replica: 1
# @schema {"name": "repetition", "type": "integer", "minimum": 1}
# repetition -- the number of repeat job
repetition: 1
# @schema {"name": "job_type", "type": "string", "enum": ["insert", "update", "upsert", "search", "remove", "get_object", "exists"]}
# @schema {"name": "job_type", "type": "string", "enum": ["insert", "update", "upsert", "search", "remove", "getobject", "exists"]}
# job_type -- job type name
job_type: "search"
# @schema {"name": "insert_config", "type": "object"}
Expand Down Expand Up @@ -108,6 +111,12 @@ search_config:
# @schema {"name": "search_config.timeout", "type": "string"}
# search_config.timeout -- search operation timeout
timeout: "10s"
# @schema {"name": "search_config.enable_linear_search", "type": "boolean"}
# search_config.enable_linear_search -- enable linear search for calculation recall
enable_linear_search: true
# @schema {"name": "search_config.aggregation_algorithm", "type": "string", "enum": ["Unknown", "ConcurrentQueue", "SortSlice", "SortPoolSlice", "PairingHeap"]}
# search_config.aggregation_algorithm -- search result aggregation algorithm
aggregation_algorithm: "Unknown"

# @schema {"name": "remove_config", "type": "object"}
# remove_config -- remove config
Expand Down Expand Up @@ -358,3 +367,9 @@ rules: []
# @schema {"name": "rps", "type": "integer", "minimum": 0, "maximum": 65535}
# rps -- desired request per sec
rps: 1000
# @schema {"name": "concurrency_limit", "type": "integer", "minimum": 0, "maximum": 65535}
# concurrency_limit -- concurrency_limit represents the goroutine limit count. It affects the job performance.
concurrency_limit: 200
# @schema {"name": "ttl_seconds_after_finished", "type": "integer", "minimum": 0, "maximum": 65535}
# ttl_seconds_after_finished -- limits the lifetime of a Job that has finished execution.
ttl_seconds_after_finished: 10
9 changes: 7 additions & 2 deletions charts/vald-benchmark-operator/schemas/scenario-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ target:
# @schema {"name": "dataset", "type": "object", "required": ["name", "indexes", "group", "range"]}
# dataset -- dataset information
dataset:
# @schema {"name": "dataset.name", "type": "string", "enum": ["fashion-mnist"] }
# @schema {"name": "dataset.name", "type": "string", "enum": ["original", "fashion-mnist"] }
# dataset.name -- the name of dataset
name: "fashion-mnist"
# @schema {"name": "dataset.indexes", "type": "integer", "minimum": 0}
Expand All @@ -37,7 +37,7 @@ dataset:
# @schema {"name": "dataset.group", "type": "string", "minLength": 1}
# dataset.group -- the hdf5 group name of dataset
group: "test"
# @schema {"name": "dataset.range", "type": "object", "range": ["start", "port"]}
# @schema {"name": "dataset.range", "type": "object", "required": ["start", "end"]}
# dataset.range -- the data range of indexes
range:
# @schema {"name": "dataset.range.start", "type": "integer", "minimum": 1}
Expand All @@ -46,6 +46,9 @@ dataset:
# @schema {"name": "dataset.range.end", "type": "integer", "minimum": 1}
# dataset.range.end -- end index number
end: 1000
# @schema {"name": "dataset.url", "type": "string"}
# dataset.url -- the dataset url which is used for executing benchmark job with user defined hdf5 file
url: ""

# @schema {"name": "jobs", "type": "array", "items": {"type": "object"}}
jobs:
Expand Down Expand Up @@ -80,6 +83,7 @@ jobs:
num: 10
min_num: 10
timeout: "10s"
enable_linear_search: true
remove_config:
skip_strict_exist_check: false
timestamp: ""
Expand Down Expand Up @@ -167,3 +171,4 @@ jobs:
insecure_skip_verify: false
rules: []
rps: 1000
ttl_seconds_after_finished: 100
1 change: 1 addition & 0 deletions charts/vald-benchmark-operator/templates/clusterrole.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ rules:
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
Expand Down
3 changes: 3 additions & 0 deletions charts/vald-benchmark-operator/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,6 @@ data:
observability:
{{- $observability := dict "Values" .Values.observability}}
{{- include "vald.observability" $observability | nindent 6 }}
job_image:
image: "{{ .Values.job_image.repository }}:{{ .Values.job_image.tag }}"
pullPolicy: {{ .Values.job_image.pullPolicy }}
14 changes: 14 additions & 0 deletions charts/vald-benchmark-operator/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,16 @@ spec:
app.kubernetes.io/component: benchmark-operator
{{- with .Values.podAnnotations }}
annotations:
{{- if .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.server_config.metrics.pprof.enabeld }}
pyroscope.io/scrape: "true"
pyroscope.io/application-name: {{ .Values.name }}
pyroscope.io/profile-cpu-enabled: "true"
pyroscope.io/profile-mem-enabled: "true"
pyroscope.io/port: "{{ .Values.server_config.metrics.pprof.port }}"
{{- end}}
{{- end }}
spec:
serviceAccountName: {{ .Values.serviceAccount.name }}
Expand Down Expand Up @@ -113,6 +122,11 @@ spec:
volumeMounts:
- name: {{ .Values.name }}-config
mountPath: /etc/server
env:
- name: JOB_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
Expand Down
Loading

0 comments on commit 96a976d

Please sign in to comment.