Skip to content

Commit

Permalink
Refactor helm template and operator logic (#2043)
Browse files Browse the repository at this point in the history 
* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* ✨ create bench job search tools

Signed-off-by: vankichi <[email protected]>

* ✨ add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format

Signed-off-by: vankichi <[email protected]>

* ♻️ fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ fix proto

Signed-off-by: vankichi <[email protected]>

* 💚 add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* 💚 invest

Signed-off-by: vankichi <[email protected]>

* Revert "💚 invest"

This reverts commit f0f585c.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* ✨ apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* ✨ add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* ✨ add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* ✨ rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* ✨ create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* ✨ ♻️ add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* ✨ impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* ✨ create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* ♻️ remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* ✨ impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* 🐛 Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ✨ impl status handle of continuous benchmark crds (#1955)

Signed-off-by: vankichi <[email protected]>

* Impl benchmark jobs (#1977)

* Implement base of continuous benchmark tool (#1776)

* Create Continuous Bench Search Job tool (#1733)

* ✨ create bench job search tools

Signed-off-by: vankichi <[email protected]>

* ✨ add load hdf5 functions

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format

Signed-off-by: vankichi <[email protected]>

* ♻️ fix docker and use hdf5 data

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor benchmark job

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ fix proto

Signed-off-by: vankichi <[email protected]>

* 💚 add benchmark job image build ci

Signed-off-by: vankichi <[email protected]>

* 💚 invest

Signed-off-by: vankichi <[email protected]>

* Revert "💚 invest"

This reverts commit f0f585c.

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply code review

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Hiroto Funakoshi <[email protected]>

* ✨ apply from feedback

Signed-off-by: vankichi <[email protected]>

* Update internal/config/benchmark.go

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ change directory path

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>

* Add crds for continuous benchmark tools (#1789)

* ✨ add crds for continuous benchmark operator

Signed-off-by: vankichi <[email protected]>

* ✨ add benchmark operator/job scheme

Signed-off-by: vankichi <[email protected]>

* ✨ rename package names and add doc.go

Signed-off-by: vankichi <[email protected]>

* ✨ create runtime object

Signed-off-by: vankichi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yusuke Kato <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>

* Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825)

* ✨ ♻️ add Job reconciler & use scenario instead of operator

Signed-off-by: vankichi <[email protected]>

* ♻️ fix format & rename file

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Add benchmark operator framework (#1916)

* ✨ impl benchmark reconciler

Signed-off-by: vankichi <[email protected]>

* ✨ create benchmark operator framework

Signed-off-by: vankichi <[email protected]>

* ♻️ remove unness changes

Signed-off-by: vankichi <[email protected]>

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* impl reconcile logic for create benchmark job (#1923)

* ✨ impl reconcile logic for create benchmark job

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark's crds

Signed-off-by: vankichi <[email protected]>

* ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor continuous benchmark job logic

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ update charts

Signed-off-by: vankichi <[email protected]>

* Format code with prettier and gofumpt

* ♻️ rafactor con bench config and bug fix reconcile logic

Signed-off-by: vankichi <[email protected]>

* 🐛 Bugfix: fix typo and recall function logic

Signed-off-by: vankichi <[email protected]>

* ♻️ refactor pkg benchmark job

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ✨ impl benchmark jobs

Signed-off-by: vankichi <[email protected]>

* ♻️ apply feedback

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* ♻️ Refactor helm template and operator logic

Signed-off-by: vankichi <[email protected]>

* ♻️ Add download original dataset URL option

Signed-off-by: vankichi <[email protected]>

* ♻️ Set docker image location at the benchmark operator configmap and use it when information is set

Signed-off-by: vankichi <[email protected]>

* add search algorithm benchmark and update search aggregation algo

Signed-off-by: kpango <[email protected]>

* ✨ Add search result aggregation option

Signed-off-by: vankichi <[email protected]>

* style: Format code with prettier and gofumpt

* Improve job performance (#2061)

* 🐛 Fix job function to apply rate limiter

* ♻️ Add pyroscope setting

Signed-off-by: vankichi <[email protected]>

* ♻️ fix

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix build error

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>

* 🐛 Fix docker file and add concurrencyLimit for job goroutine

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix job_template.go by feedback

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix job logic by feedback

Signed-off-by: vankichi <[email protected]>

* ♻️ Fix

Signed-off-by: vankichi <[email protected]>

---------

Signed-off-by: vankichi <[email protected]>
Signed-off-by: kpango <[email protected]>
Co-authored-by: Yusuke Kato <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
  • Loading branch information
@vankichi @kpango
@hlts2 @deepsource-autofix
4 people committed Jul 24, 2023
1 parent 1fb05bc commit 745fdc4
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 38 deletions.
2 changes: 2 additions & 0 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,8 @@ require (
golang.org/x/sys v0.10.0
golang.org/x/text v0.11.0
golang.org/x/tools v0.10.0
golang.org/x/text v0.9.0
golang.org/x/time v0.3.0
golang.org/x/time v0.3.0
gonum.org/v1/hdf5 v0.0.0-00010101000000-000000000000
gonum.org/v1/plot v0.10.1
Expand Down
38 changes: 0 additions & 38 deletions pkg/tools/benchmark/job/service/job.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,44 +245,6 @@ func (j *job) PreStart(ctx context.Context) error {
return err
}
}
log.Infof("[benchmark job] start download dataset of %s", j.hdf5.GetName().String())
if err := j.hdf5.Download(); err != nil {
return err
}
log.Infof("[benchmark job] success download dataset of %s", j.hdf5.GetName().String())
log.Infof("[benchmark job] start load dataset of %s", j.hdf5.GetName().String())
if err := j.hdf5.Read(); err != nil {
return err
}
log.Infof("[benchmark job] success load dataset of %s", j.hdf5.GetName().String())
// Wait for beforeJob completed if exists
if len(j.beforeJobName) != 0 {
var jobResource v1.ValdBenchmarkJob
log.Info("[benchmark job] check before benchjob is completed or not...")
j.eg.Go(safety.RecoverFunc(func() error {
dt := time.NewTicker(j.beforeJobDur)
defer dt.Stop()
for {
select {
case <-ctx.Done():
return nil
case <-dt.C:
err := j.k8sClient.Get(ctx, j.beforeJobName, j.beforeJobNamespace, &jobResource)
if err != nil {
return err
}
if jobResource.Status == v1.BenchmarkJobCompleted {
log.Infof("[benchmark job ] before job (%s) is completed, job service will start soon.", j.beforeJobName)
return nil
}
log.Infof("[benchmark job] before job (%s) is not completed...", j.beforeJobName)
}
}
}))
if err := j.eg.Wait(); err != nil {
return err
}
}
return nil
}

Expand Down

0 comments on commit 745fdc4

Please sign in to comment.