add benchmark and check program for core ngt #2179

kmrmt · 2023-09-12T08:28:26Z

Description:

Implement core algorithm NGT's benchmark and getting metrics about memory usage.
It has 2 way of measuring memory usage.
One way is to use cmd/tools/cli/benchmark/core/main.go while another way is to use internal/core/algorithm/ngt/ngt_bench_test.go.
If you want detailed metrics, I recommend you to use the first way.
And include some fix.

Related Issue:

Versions:

Go Version: 1.21.1
Docker Version: 20.10.8
Kubernetes Version: v1.28.1
NGT Version: 2.1.3

Checklist:

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

vdaas-ci · 2023-09-12T08:28:39Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

github-actions · 2023-09-12T08:34:34Z

cmd/tools/cli/benchmark/core/main.go

+		for i := 0; i < row; i++ {
+			vecs[i] = make([]float32, dim)
+			for j := 0; j < dim; j++ {
+				vecs[i][j] = float32(vec[i*dim+j])


🚫 [golangci] _{reported by reviewdog 🐶}
unnecessary conversion (unconvert)

github-actions · 2023-09-12T08:34:34Z

cmd/tools/cli/benchmark/core/main.go

+	"os/signal"
+	"runtime"
+	"strings"
+	"sync"


🚫 [golangci] _{reported by reviewdog 🐶}
import 'sync' is not allowed from list 'main': sync is allowed only by internal/sync (depguard)

github-actions · 2023-09-12T08:34:34Z

cmd/tools/cli/benchmark/core/main.go

+	vectors, _, _ := load("sift-128-euclidean.hdf5")
+	n, _ := ngt.New(
+		ngt.WithDimension(len(vectors[0])),
+		ngt.WithDefaultPoolSize(8),


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 8, in detected (gomnd)

github-actions · 2023-09-12T08:34:34Z

cmd/tools/cli/benchmark/core/main.go

+
+		var m runtime.MemStats
+		runtime.ReadMemStats(&m)
+		log.Infof("%v\t%v\t%v\t%v\t%v\t%v\t%v\t%v\t%v", header, vmpeak, vmhwm, vmrss, m.Alloc/1024, m.TotalAlloc/1024, m.HeapAlloc/1024, m.HeapSys/1024, m.HeapInuse/1024)


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 1024, in detected (gomnd)

github-actions · 2023-09-12T08:34:35Z

cmd/tools/cli/benchmark/core/main.go

+				}
+				output("      insert")
+
+				if err := n.CreateIndex(8); err != nil {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 8, in detected (gomnd)

github-actions · 2023-09-12T08:34:35Z

cmd/tools/cli/benchmark/core/main.go

+		return nil, nil, err
+	}
+
+	return


🚫 [golangci] _{reported by reviewdog 🐶}
naked return in func load with 60 lines of code (nakedret)

codecov · 2023-09-12T08:39:16Z

Codecov Report

Attention: 574 lines in your changes are missing coverage. Please review.

Comparison is base (a865801) 30.43% compared to head (7592fa3) 30.49%.

❗ Current head 7592fa3 differs from pull request most recent head d75690f. Consider uploading reports for the commit d75690f to get more accurate results

Files	Patch %	Lines
cmd/tools/cli/benchmark/core/main.go	0.00%	445 Missing ⚠️
internal/core/algorithm/ngt/option.go	29.09%	25 Missing and 14 partials ⚠️
internal/core/algorithm/ngt/ngt.go	71.75%	34 Missing and 3 partials ⚠️
internal/net/grpc/pool/pool.go	0.00%	29 Missing ⚠️
internal/net/grpc/client.go	0.00%	23 Missing ⚠️
internal/errors/file.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2179      +/-   ##
==========================================
+ Coverage   30.43%   30.49%   +0.05%     
==========================================
  Files         362      358       -4     
  Lines       35443    35325     -118     
==========================================
- Hits        10787    10772      -15     
+ Misses      24143    24052      -91     
+ Partials      513      501      -12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cloudflare-workers-and-pages · 2023-09-13T03:27:42Z

Deploying with Cloudflare Pages

Latest commit:	`d75690f`
Status:	✅ Deploy successful!
Preview URL:	https://5e55813c.vald.pages.dev
Branch Preview URL:	https://feature-benchmark-add-benchm.vald.pages.dev

View logs

github-actions · 2023-09-19T09:14:18Z

cmd/tools/cli/benchmark/core/main.go

+	wg.Add(1)
+	go func() {
+		wg.Done()
+		srv := &http.Server{


🚫 [golangci] _{reported by reviewdog 🐶}
G112: Potential Slowloris Attack because ReadHeaderTimeout is not configured in the http.Server (gosec)

github-actions · 2023-09-19T09:14:18Z

cmd/tools/cli/benchmark/core/main.go

+		}
+		go srv.ListenAndServe()
+		<-ctx.Done()
+		srv.Shutdown(context.Background())


🚫 [golangci] _{reported by reviewdog 🐶}
Non-inherited new context, use function like context.WithXXX instead (contextcheck)

This commit fixes the style issues introduced in f838ef9 according to the output from Prettier and Gofumpt. Details: #2179

vdaas-ci · 2023-09-20T22:41:11Z

[WARNING:INTCFG] Changes in interal/config may require you to change Helm charts. Please check.

github-actions · 2023-09-21T02:44:24Z

cmd/tools/cli/benchmark/core/main.go

+				}
+				output("      insert")
+
+				if err := n.CreateIndex(8); err != nil {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 8, in detected (gomnd)

github-actions · 2023-09-21T02:44:24Z

cmd/tools/cli/benchmark/core/main.go

+	cancel()
+
+	wg.Wait()
+	end := time.NewTicker(30 * time.Minute)


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 30, in detected (gomnd)

github-actions · 2023-09-21T02:44:24Z

cmd/tools/cli/benchmark/core/main.go

+	wg.Wait()
+	end := time.NewTicker(30 * time.Minute)
+	defer end.Stop()
+	ticker := time.NewTicker(5 * time.Second)


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 5, in detected (gomnd)

github-actions · 2023-09-21T10:55:55Z

cmd/tools/cli/benchmark/core/main.go

+	log.Info("   operation\tVmPeak\tVmHWM\tVmRSS\tAlloc\tTotalAlloc\tHeapAlloc\tHeapSys\tHeapInuse")
+	output("       start")
+	defer output("         end")
+	ctx, cancel := context.WithTimeout(context.Background(), time.Hour*2)


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 2, in detected (gomnd)

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions · 2023-10-30T13:34:08Z

cmd/tools/cli/benchmark/core/main.go

+	})
+}
+
+func run(ctx context.Context, load bool, path string, dim int, vectors [][]float32, ids []uint, dur time.Duration, output func(header string)) {


🚫 [golangci] _{reported by reviewdog 🐶}
run - path is unused (unparam)

github-actions · 2023-10-30T13:34:08Z

internal/test/comparator/comparators.go

@@ -15,6 +15,7 @@ package comparator

 import (
 	"reflect"
+	"sync/atomic"


🚫 [golangci] _{reported by reviewdog 🐶}
import 'sync/atomic' is not allowed from list 'main': sync is allowed only by internal/sync (depguard)

github-actions · 2023-10-30T13:34:08Z

cmd/tools/cli/benchmark/core/main_test.go

+// 	}
+// 	type test struct {
+// 		name       string
+// 		args       args


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (args) found (dupword)

github-actions · 2023-10-30T13:34:08Z

cmd/tools/cli/benchmark/core/main_test.go

+// 	}
+// 	type test struct {
+// 		name       string
+// 		args       args


🚫 [golangci] _{reported by reviewdog 🐶}
Duplicate words (args) found (dupword)

github-actions · 2023-10-30T13:34:08Z

internal/test/comparator/comparators.go

@@ -50,4 +51,10 @@
 		// skipcq: VET-V0008
 		return reflect.DeepEqual(x, y)
 	})
+
+	// skipcq: VET-V0008
+	AtomicUint64Comparator = Comparer(func(x, y atomic.Uint64) bool {


🚫 [golangci] _{reported by reviewdog 🐶}
AtomicUint64Comparator is a global variable (gochecknoglobals)

github-actions · 2023-10-30T13:34:09Z

cmd/tools/cli/benchmark/core/main.go

+			fn()
+		}
+	}
+	return


🚫 [golangci] _{reported by reviewdog 🐶}
S1023: redundant return statement (gosimple)

github-actions · 2023-10-30T13:34:09Z

internal/test/comparator/comparators.go

+	// skipcq: VET-V0008
+	AtomicUint64Comparator = Comparer(func(x, y atomic.Uint64) bool {
+		// skipcq: VET-V0008
+		return reflect.DeepEqual(x, y)


🚫 [golangci] _{reported by reviewdog 🐶}
copylocks: call of reflect.DeepEqual copies lock value: sync/atomic.Uint64 contains sync/atomic.noCopy (govet)

github-actions · 2023-10-30T13:34:10Z

cmd/tools/cli/benchmark/core/main.go

+	"gonum.org/v1/hdf5"
+)
+
+func main() {


🚫 [golangci] _{reported by reviewdog 🐶}
Function name: main, Cyclomatic Complexity: 31, Halstead Volume: 8179.08, Maintainability Index: 14 (maintidx)

github-actions · 2023-10-30T13:34:10Z

cmd/tools/cli/benchmark/core/main.go

+	})
+	run(ctx, true, path, len(vectors[0]), vectors, ids, time.Hour*2, output)
+
+	ids = ids[:0:0]


🚫 [golangci] _{reported by reviewdog 🐶}
SA4006: this value of ids is never used (staticcheck)

github-actions · 2023-10-30T13:34:10Z

cmd/tools/cli/benchmark/core/main.go

+
+	ids = ids[:0:0]
+	ids = nil
+	vectors = vectors[:0:0]


🚫 [golangci] _{reported by reviewdog 🐶}
SA4006: this value of vectors is never used (staticcheck)

vdaas-ci · 2023-10-30T14:02:13Z

Profile Report

type	vald-agent-ngt	vald-lb-gateway	vald-discoverer	vald-manager-index
cpu
heap

other images

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+		<-ctx.Done()
+		ctx, cancel := context.WithCancel(context.Background())
+		defer cancel()
+		srv.Shutdown(ctx)


🚫 [golangci] _{reported by reviewdog 🐶}
Non-inherited new context, use function like context.WithXXX instead (contextcheck)

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+	go func() {
+		const timeout = time.Second * 5
+		defer wg.Done()
+		srv := &http.Server{


🚫 [golangci] _{reported by reviewdog 🐶}
http.Server is missing fields DisableGeneralOptionsHandler, TLSConfig, ReadTimeout, WriteTimeout, IdleTimeout, MaxHeaderBytes, TLSNextProto, ConnState, ErrorLog, BaseContext, ConnContext (exhaustruct)

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+		waitForGC      = time.Minute * 5
+		timeToFinalize = time.Minute * 5
+	)
+	sleep(ctx, time.Second*5, waitForGC, func() {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 5, in detected (gomnd)

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+		runtime.GC()
+		output("gc")
+	})
+	sleep(ctx, time.Second*5, waitForGC, func() {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 5, in detected (gomnd)

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+		runtime.GC()
+		output("gc")
+	})
+	sleep(ctx, time.Second*5, timeToFinalize, func() {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 5, in detected (gomnd)

github-actions · 2023-10-31T06:02:32Z

cmd/tools/cli/benchmark/core/main.go

+	"gonum.org/v1/hdf5"
+)
+
+func main() {


🚫 [golangci] _{reported by reviewdog 🐶}
Function name: main, Cyclomatic Complexity: 31, Halstead Volume: 8412.52, Maintainability Index: 14 (maintidx)

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions · 2023-10-31T08:50:27Z

cmd/tools/cli/benchmark/core/main.go

+			output("save index")
+		})
+	}
+	sleep(ctx, time.Second*5, time.Minute*10, func() {


🚫 [golangci] _{reported by reviewdog 🐶}
mnd: Magic number: 5, in detected (gomnd)

github-actions · 2023-10-31T08:50:27Z

cmd/tools/cli/benchmark/core/main.go

+	"gonum.org/v1/hdf5"
+)
+
+func main() {


🚫 [golangci] _{reported by reviewdog 🐶}
Function name: main, Cyclomatic Complexity: 31, Halstead Volume: 8293.88, Maintainability Index: 14 (maintidx)

Signed-off-by: Kosuke Morimoto <[email protected]>

cmd/tools/cli/benchmark/core/main.go

internal/net/grpc/client.go

internal/core/algorithm/ngt/ngt_bench_test.go

cmd/tools/cli/benchmark/core/main.go

internal/net/grpc/pool/pool.go

Signed-off-by: Kosuke Morimoto <[email protected]>

ykadowak · 2023-11-10T04:45:04Z

I'll approve once the conflicts are resolved

kpango

LGTM

* add core ngt benchmark Signed-off-by: kpango <[email protected]> * add finalizer for pooling Signed-off-by: kpango <[email protected]> * fix benchmark Signed-off-by: Kosuke Morimoto <[email protected]> * fix test Signed-off-by: Kosuke Morimoto <[email protected]> * fix test Signed-off-by: Kosuke Morimoto <[email protected]> * fix test Signed-off-by: Kosuke Morimoto <[email protected]> * fix reviewdog report Signed-off-by: Kosuke Morimoto <[email protected]> * fix reviewdog report Signed-off-by: Kosuke Morimoto <[email protected]> * fix deepsource report Signed-off-by: Kosuke Morimoto <[email protected]> * fix according to comments Signed-off-by: Kosuke Morimoto <[email protected]> --------- Signed-off-by: kpango <[email protected]> Signed-off-by: Kosuke Morimoto <[email protected]> Co-authored-by: kpango <[email protected]>

github-actions bot added team/set SET team area/internal size/L type/feature New feature priority/medium labels Sep 12, 2023

github-actions bot reviewed Sep 12, 2023

View reviewed changes

kmrmt force-pushed the feature/benchmark/add-benchmark-core-ngt branch from 09e9263 to cb61392 Compare September 13, 2023 03:27

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch from cb61392 to af77978 Compare September 19, 2023 09:07

github-actions bot reviewed Sep 19, 2023

View reviewed changes

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch from af77978 to f838ef9 Compare September 20, 2023 22:40

github-actions bot added area/helm area/agent/core labels Sep 20, 2023

deepsource-autofix bot added a commit that referenced this pull request Sep 20, 2023

style: format code with Prettier and Gofumpt

9a82a53

This commit fixes the style issues introduced in f838ef9 according to the output from Prettier and Gofumpt. Details: #2179

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch 4 times, most recently from bccccb4 to 5bc1bd0 Compare September 20, 2023 22:53

github-actions bot added size/XL and removed size/L labels Sep 20, 2023

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch from 5bc1bd0 to a1be637 Compare September 21, 2023 02:35

github-actions bot reviewed Sep 21, 2023

View reviewed changes

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch from a1be637 to 2ccc077 Compare September 21, 2023 10:48

github-actions bot reviewed Sep 21, 2023

View reviewed changes

kpango force-pushed the feature/benchmark/add-benchmark-core-ngt branch 3 times, most recently from 1cd6811 to f00fad7 Compare September 24, 2023 23:38

fix test

182b2bc

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions bot reviewed Oct 30, 2023

View reviewed changes

kmrmt added actions/e2e-deploy actions/e2e-chaos actions/e2e-profiling actions/e2e-max-dim labels Oct 30, 2023

fix reviewdog report

110efce

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions bot reviewed Oct 31, 2023

View reviewed changes

fix reviewdog report

02b0b5f

Signed-off-by: Kosuke Morimoto <[email protected]>

github-actions bot reviewed Oct 31, 2023

View reviewed changes

kmrmt requested review from kpango, a team and ykadowak and removed request for a team October 31, 2023 08:55

fix deepsource report

52eb1c4

Signed-off-by: Kosuke Morimoto <[email protected]>

ykadowak reviewed Nov 8, 2023

View reviewed changes

fix according to comments

c10d65c

Signed-off-by: Kosuke Morimoto <[email protected]>

kpango added 3 commits November 13, 2023 08:38

Merge branch 'main' into feature/benchmark/add-benchmark-core-ngt

af069b8

Merge branch 'main' into feature/benchmark/add-benchmark-core-ngt

7592fa3

Merge branch 'main' into feature/benchmark/add-benchmark-core-ngt

d75690f

kpango approved these changes Dec 4, 2023

View reviewed changes

kpango merged commit 405e499 into main Dec 4, 2023
95 of 97 checks passed

kpango deleted the feature/benchmark/add-benchmark-core-ngt branch December 4, 2023 10:00

kpango restored the feature/benchmark/add-benchmark-core-ngt branch December 4, 2023 10:00

kpango deleted the feature/benchmark/add-benchmark-core-ngt branch December 4, 2023 10:00

kpango restored the feature/benchmark/add-benchmark-core-ngt branch December 4, 2023 10:00

kpango mentioned this pull request Dec 6, 2023

[patch] release v1.7.9 #2257

Merged

add benchmark and check program for core ngt #2179

add benchmark and check program for core ngt #2179

Conversation

kmrmt commented Sep 12, 2023 • edited Loading

Description:

Related Issue:

Versions:

Checklist:

Special notes for your reviewer:

vdaas-ci commented Sep 12, 2023

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

github-actions bot Sep 12, 2023

Choose a reason for hiding this comment

codecov bot commented Sep 12, 2023 • edited Loading

Codecov Report

cloudflare-workers-and-pages bot commented Sep 13, 2023 • edited Loading

Deploying with Cloudflare Pages

github-actions bot Sep 19, 2023

Choose a reason for hiding this comment

github-actions bot Sep 19, 2023

Choose a reason for hiding this comment

vdaas-ci commented Sep 20, 2023

github-actions bot Sep 21, 2023

Choose a reason for hiding this comment

github-actions bot Sep 21, 2023

Choose a reason for hiding this comment

github-actions bot Sep 21, 2023

Choose a reason for hiding this comment

github-actions bot Sep 21, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

github-actions bot Oct 30, 2023

Choose a reason for hiding this comment

vdaas-ci commented Oct 30, 2023

Profile Report

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

github-actions bot Oct 31, 2023

Choose a reason for hiding this comment

ykadowak commented Nov 10, 2023

kpango left a comment

Choose a reason for hiding this comment

kmrmt commented Sep 12, 2023 •

edited

Loading

codecov bot commented Sep 12, 2023 •

edited

Loading

cloudflare-workers-and-pages bot commented Sep 13, 2023 •

edited

Loading