Skip to content

Commit

Permalink
Implement continuous benchmark tool (#2216)
Browse files Browse the repository at this point in the history
* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* create helm template for benchmark operator (#2027)

* :sparkles: create helm template for benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: refactor helm template

Signed-off-by: vankichi <[email protected]>

* style: Format code with gofumpt and prettier

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Refactor helm template and operator logic (#2043)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :recycle: Refactor helm template and operator logic

Signed-off-by: vankichi <[email protected]>

* :recycle: Add download original dataset URL option

Signed-off-by: vankichi <[email protected]>

* :recycle: Set docker image location at the benchmark operator configmap and use it when information is set

Signed-off-by: vankichi <[email protected]>

* add search algorithm benchmark and update search aggregation algo

Signed-off-by: kpango <[email protected]>

* :sparkles: Add search result aggregation option

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* Improve job performance (#2061)

* :bug: Fix job function to apply rate limiter

* :recycle: Add pyroscope setting

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :bug: Fix build error

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* :bug: Fix docker file and add concurrencyLimit for job goroutine

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job_template.go by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job logic by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* style: Format code with prettier and gofumpt

* Update charts and improvement conbench (#2119)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* create helm template for benchmark operator (#2027)

* :sparkles: create helm template for benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: refactor helm template

Signed-off-by: vankichi <[email protected]>

* style: Format code with gofumpt and prettier

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Refactor helm template and operator logic (#2043)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :recycle: Refactor helm template and operator logic

Signed-off-by: vankichi <[email protected]>

* :recycle: Add download original dataset URL option

Signed-off-by: vankichi <[email protected]>

* :recycle: Set docker image location at the benchmark operator configmap and use it when information is set

Signed-off-by: vankichi <[email protected]>

* add search algorithm benchmark and update search aggregation algo

Signed-off-by: kpango <[email protected]>

* :sparkles: Add search result aggregation option

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* Improve job performance (#2061)

* :bug: Fix job function to apply rate limiter

* :recycle: Add pyroscope setting

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :bug: Fix build error

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* :bug: Fix docker file and add concurrencyLimit for job goroutine

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job_template.go by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job logic by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Refactor helm template and operator logic (#2043)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: fix proto

Signed-off-by: vankichi <[email protected]>

* :green_heart: add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* :green_heart: invest

Signed-off-by: vankichi <[email protected]>

* Revert ":green_heart: invest"

This reverts commit f0f585ccf71b1c95a88559941557a27774096e69.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* :sparkles: apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* :sparkles: add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* :sparkles: add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* :sparkles: rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* :sparkles: create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* :sparkles: :recycle: add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* :sparkles: impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* :sparkles: create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* :recycle: remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* :sparkles: impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* :recycle: resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* :recycle: rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* :bug: Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :sparkles: impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* :recycle: apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :recycle: Refactor helm template and operator logic

Signed-off-by: vankichi <[email protected]>

* :recycle: Add download original dataset URL option

Signed-off-by: vankichi <[email protected]>

* :recycle: Set docker image location at the benchmark operator configmap and use it when information is set

Signed-off-by: vankichi <[email protected]>

* add search algorithm benchmark and update search aggregation algo

Signed-off-by: kpango <[email protected]>

* :sparkles: Add search result aggregation option

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* Improve job performance (#2061)

* :bug: Fix job function to apply rate limiter

* :recycle: Add pyroscope setting

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :bug: Fix build error

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* :bug: Fix docker file and add concurrencyLimit for job goroutine

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job_template.go by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix job logic by feedback

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :recycle: Add noise to vector when update/upsert and update crd

Signed-off-by: vankichi <[email protected]>

* :recycle: add deepmerge func for override default config by user-defined config

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor deepmerge

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* style: Format code with prettier and gofumpt

* :recycle: change add noise func

Signed-off-by: vankichi <[email protected]>

* :recycle: use golang/sync/error for prevent OOM KILL

Signed-off-by: vankichi <[email protected]>

* :recycle: fix config adn add test

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* :recycle: Apply new internal/sync for continous benchmark (#2175)

Signed-off-by: vankichi <[email protected]>

* :whale: fix Dockerfile

Signed-off-by: vankichi <[email protected]>

* :recycle: Fix license and format

Signed-off-by: vankichi <[email protected]>

* :white_check_mark: add benchmark operator reconcile test

Signed-off-by: vankichi <[email protected]>

* :recycle: Refactor from deepsource feedback (#2222)

Signed-off-by: vankichi <[email protected]>

* :recycle: use modulo to calc idx, fix Makefile, add error handling

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* :recycle: Fix feedback

Signed-off-by: vankichi <[email protected]>

* Refactor/dockers/apply build for 2 platform (#2267)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* :sparkles: create bench job search tools

Signed-off-by: vankichi <[email protected]>

* :sparkles: add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* :recycle: fix format

Signed-off-by: vankichi <[email protected]>

* :recycle: fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed-off-by: vankichi <[email protected]>

* :recycle: refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* :recycle: fix

Signed…
  • Loading branch information
4 people authored Jan 11, 2024
1 parent 6f4b179 commit f943916
Show file tree
Hide file tree
Showing 121 changed files with 22,134 additions and 12 deletions.
82 changes: 82 additions & 0 deletions .github/workflows/dockers-benchmark-job-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
name: "Build docker image: benchmark-job"
on:
push:
branches:
- main
tags:
- "*.*.*"
- "v*.*.*"
- "*.*.*-*"
- "v*.*.*-*"
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-job-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/job/Dockerfile"
- "versions/GO_VERSION"
pull_request:
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-job-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/job/Dockerfile"
- "versions/GO_VERSION"
pull_request_target:
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-job-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/job/Dockerfile"
- "versions/GO_VERSION"

jobs:
build:
uses: ./.github/workflows/_docker-image.yaml
with:
target: benchmark-job
secrets: inherit
82 changes: 82 additions & 0 deletions .github/workflows/dockers-benchmark-operator-image.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
name: "Build docker image: benchmark-operator"
on:
push:
branches:
- main
tags:
- "*.*.*"
- "v*.*.*"
- "*.*.*-*"
- "v*.*.*-*"
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-operator-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/operator/Dockerfile"
- "versions/GO_VERSION"
pull_request:
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-operator-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/operator/Dockerfile"
- "versions/GO_VERSION"
pull_request_target:
paths:
- ".github/actions/docker-build/actions.yaml"
- ".github/workflows/_docker-image.yaml"
- ".github/workflows/dockers-benchmak-operator-image.yml"
- "go.mod"
- "go.sum"
- "internal/**"
- "!internal/**/*_test.go"
- "!internal/db/**"
- "apis/grpc/**"
- "pkg/benchmark/operator/**"
- "cmd/benchmark/operator/**"
- "pkg/benchmark/job/**"
- "cmd/benchmark/job/**"
- "dockers/tools/benchmark/operator/Dockerfile"
- "versions/GO_VERSION"

jobs:
build:
uses: ./.github/workflows/_docker-image.yaml
with:
target: benchmark-operator
secrets: inherit
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ INDEX_CREATION_IMAGE = $(NAME)-index-creation
INDEX_SAVE_IMAGE = $(NAME)-index-save
READREPLICA_ROTATE_IMAGE = $(NAME)-readreplica-rotate
MANAGER_INDEX_IMAGE = $(NAME)-manager-index
BENCHMARK_JOB_IMAGE = $(NAME)-benchmark-job
BENCHMARK_OPERATOR_IMAGE = $(NAME)-benchmark-operator
MAINTAINER = "$(ORG).org $(NAME) team <$(NAME)@$(ORG).org>"

VERSION ?= $(eval VERSION := $(shell cat versions/VALD_VERSION))$(VERSION)
Expand Down Expand Up @@ -72,6 +74,8 @@ TELEPRESENCE_VERSION := $(eval TELEPRESENCE_VERSION := $(shell cat versions
VALDCLI_VERSION := $(eval VALDCLI_VERSION := $(shell cat versions/VALDCLI_VERSION))$(VALDCLI_VERSION)
YQ_VERSION := $(eval YQ_VERSION := $(shell cat versions/YQ_VERSION))$(YQ_VERSION)
BUF_VERSION := $(eval BUF_VERSION := $(shell cat versions/BUF_VERSION))$(BUF_VERSION)
ZLIB_VERSION := $(eval ZLIB_VERSION := $(shell cat versions/ZLIB_VERSION))$(ZLIB_VERSION)
HDF5_VERSION := $(eval HDF5_VERSION := $(shell cat versions/HDF5_VERSION))$(HDF5_VERSION)

OTEL_OPERATOR_RELEASE_NAME ?= opentelemetry-operator
PROMETHEUS_RELEASE_NAME ?= prometheus
Expand Down
69 changes: 68 additions & 1 deletion Makefile.d/build.mk
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ binary/build: \
cmd/discoverer/k8s/discoverer \
cmd/gateway/lb/lb \
cmd/gateway/filter/filter \
cmd/manager/index/index
cmd/manager/index/index \
cmd/tools/benchmark/job/job \
cmd/tools/benchmark/operator/operator

cmd/agent/core/ngt/ngt: \
ngt/install \
Expand Down Expand Up @@ -322,6 +324,64 @@ cmd/index/job/readreplica/rotate/readreplica-rotate: \
$(dir $@)main.go
$@ -version

cmd/tools/benchmark/job/job: \
$(GO_SOURCES_INTERNAL) \
$(PBGOS) \
$(shell find ./cmd/tools/benchmark/job -type f -name '*.go' -not -name '*_test.go' -not -name 'doc.go') \
$(shell find ./pkg/tools/benchmark/job -type f -name '*.go' -not -name '*_test.go' -not -name 'doc.go')
CGO_ENABLED=$(CGO_ENABLED) \
GO111MODULE=on \
GOPRIVATE=$(GOPRIVATE) \
go build \
--ldflags "-w -linkmode 'external' \
-extldflags '-static -fPIC -pthread -fopenmp -std=gnu++20 -lhdf5 -lhdf5_hl -lm -ldl' \
-X '$(GOPKG)/internal/info.Version=$(VERSION)' \
-X '$(GOPKG)/internal/info.GitCommit=$(GIT_COMMIT)' \
-X '$(GOPKG)/internal/info.BuildTime=$(DATETIME)' \
-X '$(GOPKG)/internal/info.GoVersion=$(GO_VERSION)' \
-X '$(GOPKG)/internal/info.GoOS=$(GOOS)' \
-X '$(GOPKG)/internal/info.GoArch=$(GOARCH)' \
-X '$(GOPKG)/internal/info.CGOEnabled=${CGO_ENABLED}' \
-X '$(GOPKG)/internal/info.NGTVersion=$(NGT_VERSION)' \
-X '$(GOPKG)/internal/info.BuildCPUInfoFlags=$(CPU_INFO_FLAGS)' \
-buildid=" \
-mod=readonly \
-modcacherw \
-a \
-tags "cgo osusergo netgo static_build" \
-trimpath \
-o $@ \
$(dir $@)main.go
$@ -version

cmd/tools/benchmark/operator/operator: \
$(GO_SOURCES_INTERNAL) \
$(PBGOS) \
$(shell find ./cmd/tools/benchmark/operator -type f -name '*.go' -not -name '*_test.go' -not -name 'doc.go') \
$(shell find ./pkg/tools/benchmark/operator -type f -name '*.go' -not -name '*_test.go' -not -name 'doc.go')
CFLAGS="$(CFLAGS)" \
CXXFLAGS="$(CXXFLAGS)" \
CGO_ENABLED=1 \
CGO_CXXFLAGS="-g -Ofast -march=native" \
CGO_FFLAGS="-g -Ofast -march=native" \
CGO_LDFLAGS="-g -Ofast -march=native" \
GO111MODULE=on \
GOPRIVATE=$(GOPRIVATE) \
go build \
--ldflags "-w -extldflags=-static \
-X '$(GOPKG)/internal/info.CGOEnabled=${CGO_ENABLED}' \
-X '$(GOPKG)/internal/info.NGTVersion=$(NGT_VERSION)' \
-X '$(GOPKG)/internal/info.BuildCPUInfoFlags=$(CPU_INFO_FLAGS)' \
-buildid=" \
-mod=readonly \
-modcacherw \
-a \
-tags "cgo osusergo netgo static_build" \
-trimpath \
-o $@ \
$(dir $@)main.go
$@ -version

.PHONY: binary/build/zip
## build all binaries and zip them
binary/build/zip: \
Expand Down Expand Up @@ -356,3 +416,10 @@ artifacts/vald-manager-index-$(GOOS)-$(GOARCH).zip: cmd/manager/index/index
$(call mkdir, $(dir $@))
zip --junk-paths $@ $<

artifacts/vald-benchmark-job-$(GOOS)-$(GOARCH).zip: cmd/tools/benchmark/job/job
$(call mkdir, $(dir $@))
zip --junk-paths $@ $<

artifacts/vald-benchmark-operator-$(GOOS)-$(GOARCH).zip: cmd/tools/benchmark/operator/operator
$(call mkdir, $(dir $@))
zip --junk-paths $@ $<
14 changes: 13 additions & 1 deletion Makefile.d/dependencies.mk
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ update/libs: \
update/telepresence \
update/vald \
update/valdcli \
update/yq
update/yq \
update/zlib \
update/hdf5

.PHONY: go/download
## download Go package dependencies
Expand Down Expand Up @@ -163,6 +165,16 @@ update/telepresence:
update/yq:
curl --silent https://api.github.com/repos/mikefarah/yq/releases/latest | grep -Po '"tag_name": "\K.*?(?=")' > $(ROOTDIR)/versions/YQ_VERSION

.PHONY: update/zlib
## update zlib version
update/zlib:
curl --silent https://api.github.com/repos/madler/zlib/releases/latest | grep -Po '"tag_name": "\K.*?(?=")' | sed 's/v//g' > $(ROOTDIR)/versions/ZLIB_VERSION

.PHONY: update/hdf5
## update hdf5 version
update/hdf5:
curl --silent https://api.github.com/repos/HDFGroup/hdf5/releases/latest | grep -Po '"tag_name": "\K.*?(?=")' | sed 's/v//g' > $(ROOTDIR)/versions/HDF5_VERSION

.PHONY: update/vald
## update vald it's self version
update/vald:
Expand Down
25 changes: 25 additions & 0 deletions Makefile.d/docker.mk
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ docker/build: \
docker/build/gateway-lb \
docker/build/gateway-filter \
docker/build/manager-index \
docker/build/benchmark-job \
docker/build/benchmark-operator \
docker/build/operator/helm

.PHONY: docker/name/org
Expand Down Expand Up @@ -226,3 +228,26 @@ docker/build/readreplica-rotate:
@make DOCKERFILE="$(ROOTDIR)/dockers/index/job/readreplica/rotate/Dockerfile" \
IMAGE=$(READREPLICA_ROTATE_IMAGE) \
docker/build/image

.PHONY: docker/name/benchmark-job
docker/name/benchmark-job:
@echo "$(ORG)/$(BENCHMARK_JOB_IMAGE)"

.PHONY: docker/build/benchmark-job
## build benchmark job
docker/build/benchmark-job:
@make DOCKERFILE="$(ROOTDIR)/dockers/tools/benchmark/job/Dockerfile" \
IMAGE=$(BENCHMARK_JOB_IMAGE) \
DOCKER_OPTS="--build-arg ZLIB_VERSION=$(ZLIB_VERSION) --build-arg HDF5_VERSION=$(HDF5_VERSION)" \
docker/build/image

.PHONY: docker/name/benchmark-operator
docker/name/benchmark-operator:
@echo "$(ORG)/$(BENCHMARK_OPERATOR_IMAGE)"

.PHONY: docker/build/benchmark-operator
## build benchmark operator
docker/build/benchmark-operator:
@make DOCKERFILE="$(ROOTDIR)/dockers/tools/benchmark/operator/Dockerfile" \
IMAGE=$(BENCHMARK_OPERATOR_IMAGE) \
docker/build/image
9 changes: 9 additions & 0 deletions Makefile.d/functions.mk
Original file line number Diff line number Diff line change
Expand Up @@ -206,3 +206,12 @@ define gen-go-option-test-sources
fi; \
done
endef

define gen-vald-crd
mv charts/$1/crds/$2.yaml $(TEMP_DIR)/$2.yaml
GOPRIVATE=$(GOPRIVATE) \
go run -mod=readonly hack/helm/schema/crd/main.go \
charts/$1/$3.yaml > $(TEMP_DIR)/$2-spec.yaml
$(BINDIR)/yq eval-all 'select(fileIndex==0).spec.versions[0].schema.openAPIV3Schema.properties.spec = select(fileIndex==1).spec | select(fileIndex==0)' \
$(TEMP_DIR)/$2.yaml $(TEMP_DIR)/$2-spec.yaml > charts/$1/crds/$2.yaml
endef
Loading

0 comments on commit f943916

Please sign in to comment.