Skip to content

Commit

Permalink
Add E2E tests for read replica feature (#2298)
Browse files Browse the repository at this point in the history
* Add mirror gateway definitions and Implementations (#2262)

* add mirror definition and update proto dependencies

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway component (#1949)

* add mirror component

Signed-off-by: hlts2 <[email protected]>

* fix mirror dockerfile path

Signed-off-by: hlts2 <[email protected]>

* fix deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix insert rpc handler

Signed-off-by: hlts2 <[email protected]>

* deleted unused file

Signed-off-by: hlts2 <[email protected]>

* fix api resource name

Signed-off-by: hlts2 <[email protected]>

* add mirror service and fix handler logic

Signed-off-by: hlts2 <[email protected]>

* fix values and helm template for new configuration

Signed-off-by: hlts2 <[email protected]>

* Format code with gofumpt and prettier

* fix template rendering bug of mirror configmap

Signed-off-by: hlts2 <[email protected]>

* fix nil pointer error

Signed-off-by: hlts2 <[email protected]>

* bugfix to advertize different address

Signed-off-by: hlts2 <[email protected]>

* returns error when there is no other mirror gateways

Signed-off-by: hlts2 <[email protected]>

* add info log

Signed-off-by: hlts2 <[email protected]>

* extract only mirror addresses

Signed-off-by: hlts2 <[email protected]>

* fix log variable

Signed-off-by: hlts2 <[email protected]>

* add logging

Signed-off-by: hlts2 <[email protected]>

* bugfix mirror connection logic

Signed-off-by: hlts2 <[email protected]>

* add remove api proxy implementation

Signed-off-by: hlts2 <[email protected]>

* add error handling for register API

Signed-off-by: hlts2 <[email protected]>

* fix debug message

Signed-off-by: hlts2 <[email protected]>

* refactor discover logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add remove api

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* add remove api and remove api rollback logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add update rpc handler

Signed-off-by: hlts2 <[email protected]>

* refactor context variable for grpc method

Signed-off-by: hlts2 <[email protected]>

* fix trace span name

Signed-off-by: hlts2 <[email protected]>

* add getObjects method

Signed-off-by: hlts2 <[email protected]>

* refactor update and remove handler

Signed-off-by: hlts2 <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add multi handler implementation

Signed-off-by: hlts2 <[email protected]>

* fix mirror client trace span

Signed-off-by: hlts2 <[email protected]>

* rename discover service to mirror service

Signed-off-by: hlts2 <[email protected]>

* fix build error of bidirectional stream

Signed-off-by: hlts2 <[email protected]>

* add more error handling to search handler

Signed-off-by: hlts2 <[email protected]>

* add debug comment

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* fix named return bug

Signed-off-by: hlts2 <[email protected]>

* add debug log

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add force status wrap

Signed-off-by: hlts2 <[email protected]>

* execute upsert method to local cluster's lb

Signed-off-by: hlts2 <[email protected]>

* add target debug log

Signed-off-by: hlts2 <[email protected]>

* fix delete unwrap of circuitbreaker

Signed-off-by: hlts2 <[email protected]>

* refactor error handling for new status parse function

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* refactor error handling

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix error handling for broadcast operation of getobject

Signed-off-by: hlts2 <[email protected]>

* fix error join bug of getobject

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix status handling and add address to resource name

Signed-off-by: hlts2 <[email protected]>

* use vald client for update operation

Signed-off-by: hlts2 <[email protected]>

* add rollback test for mirror crud

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix variable name

Signed-off-by: hlts2 <[email protected]>

* refactor mirror servie

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix trace attribute and refactor variable name

Signed-off-by: hlts2 <[email protected]>

* add error handling when crud rpc fails

Signed-off-by: hlts2 <[email protected]>

* deleted unused value

Signed-off-by: hlts2 <[email protected]>

* make format

Signed-off-by: hlts2 <[email protected]>

* Revert "make format"

This reverts commit f605563.

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* apply suggestion

Signed-off-by: hlts2 <[email protected]>

* bugfix and refactor mirror client

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Rename ip to host of proto payload field for mirror gateway (#2031)

* fix field name and yaml indent

Signed-off-by: hlts2 <[email protected]>

* make proto/add format

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* make proto/add & make format

Signed-off-by: hlts2 <[email protected]>

* fix docker build for mirror

Signed-off-by: hlts2 <[email protected]>

* Revert "fix docker build for mirror"

This reverts commit 53ab94f.

* fix docker build error for mirror gateway

Signed-off-by: hlts2 <[email protected]>

* Add Custom Resource for mirror target (#2023)

* add mirror definition and update proto dependencies

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway component (#1949)

* add mirror component

Signed-off-by: hlts2 <[email protected]>

* fix mirror dockerfile path

Signed-off-by: hlts2 <[email protected]>

* fix deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix insert rpc handler

Signed-off-by: hlts2 <[email protected]>

* deleted unused file

Signed-off-by: hlts2 <[email protected]>

* fix api resource name

Signed-off-by: hlts2 <[email protected]>

* add mirror service and fix handler logic

Signed-off-by: hlts2 <[email protected]>

* fix values and helm template for new configuration

Signed-off-by: hlts2 <[email protected]>

* Format code with gofumpt and prettier

* fix template rendering bug of mirror configmap

Signed-off-by: hlts2 <[email protected]>

* fix nil pointer error

Signed-off-by: hlts2 <[email protected]>

* bugfix to advertize different address

Signed-off-by: hlts2 <[email protected]>

* returns error when there is no other mirror gateways

Signed-off-by: hlts2 <[email protected]>

* add info log

Signed-off-by: hlts2 <[email protected]>

* extract only mirror addresses

Signed-off-by: hlts2 <[email protected]>

* fix log variable

Signed-off-by: hlts2 <[email protected]>

* add logging

Signed-off-by: hlts2 <[email protected]>

* bugfix mirror connection logic

Signed-off-by: hlts2 <[email protected]>

* add remove api proxy implementation

Signed-off-by: hlts2 <[email protected]>

* add error handling for register API

Signed-off-by: hlts2 <[email protected]>

* fix debug message

Signed-off-by: hlts2 <[email protected]>

* refactor discover logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add remove api

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* add remove api and remove api rollback logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add update rpc handler

Signed-off-by: hlts2 <[email protected]>

* refactor context variable for grpc method

Signed-off-by: hlts2 <[email protected]>

* fix trace span name

Signed-off-by: hlts2 <[email protected]>

* add getObjects method

Signed-off-by: hlts2 <[email protected]>

* refactor update and remove handler

Signed-off-by: hlts2 <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add multi handler implementation

Signed-off-by: hlts2 <[email protected]>

* fix mirror client trace span

Signed-off-by: hlts2 <[email protected]>

* rename discover service to mirror service

Signed-off-by: hlts2 <[email protected]>

* fix build error of bidirectional stream

Signed-off-by: hlts2 <[email protected]>

* add more error handling to search handler

Signed-off-by: hlts2 <[email protected]>

* add debug comment

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* fix named return bug

Signed-off-by: hlts2 <[email protected]>

* add debug log

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add force status wrap

Signed-off-by: hlts2 <[email protected]>

* execute upsert method to local cluster's lb

Signed-off-by: hlts2 <[email protected]>

* add target debug log

Signed-off-by: hlts2 <[email protected]>

* fix delete unwrap of circuitbreaker

Signed-off-by: hlts2 <[email protected]>

* refactor error handling for new status parse function

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* refactor error handling

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix error handling for broadcast operation of getobject

Signed-off-by: hlts2 <[email protected]>

* fix error join bug of getobject

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix status handling and add address to resource name

Signed-off-by: hlts2 <[email protected]>

* use vald client for update operation

Signed-off-by: hlts2 <[email protected]>

* add rollback test for mirror crud

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix variable name

Signed-off-by: hlts2 <[email protected]>

* refactor mirror servie

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix trace attribute and refactor variable name

Signed-off-by: hlts2 <[email protected]>

* add error handling when crud rpc fails

Signed-off-by: hlts2 <[email protected]>

* deleted unused value

Signed-off-by: hlts2 <[email protected]>

* make format

Signed-off-by: hlts2 <[email protected]>

* Revert "make format"

This reverts commit f605563.

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* apply suggestion

Signed-off-by: hlts2 <[email protected]>

* bugfix and refactor mirror client

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add discovery logic for other mirror target

Signed-off-by: hlts2 <[email protected]>

* add k8s template file

Signed-off-by: hlts2 <[email protected]>

* add crd for mirror target resource

Signed-off-by: hlts2 <[email protected]>

* add new values for mirror discovery logic

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* fix schema register error and fix manifest to multi vald cluster

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* bugfix: adapt the service account and add early return

Signed-off-by: hlts2 <[email protected]>

* bugfix: usage of errors join

Signed-off-by: hlts2 <[email protected]>

* bugfix: type conversion bug

Signed-off-by: hlts2 <[email protected]>

* fix metadata hash logic and add mirror group function

Signed-off-by: hlts2 <[email protected]>

* add status handling for network connection

Signed-off-by: hlts2 <[email protected]>

* refactoring

Signed-off-by: hlts2 <[email protected]>

* bugfix: status update handling

Signed-off-by: hlts2 <[email protected]>

* bugfix: update status using status writer

Signed-off-by: hlts2 <[email protected]>

* add status timestamp

Signed-off-by: hlts2 <[email protected]>

* fix yaml field format

Signed-off-by: hlts2 <[email protected]>

* improve status changes handling

Signed-off-by: hlts2 <[email protected]>

* reduced number of status updates

Signed-off-by: hlts2 <[email protected]>

* added status check handling before changing status

Signed-off-by: hlts2 <[email protected]>

* update vald mirror target definition and add new cluster role

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* deploy multiple vald clusters and mirror target resource to connect with each other

Signed-off-by: hlts2 <[email protected]>

* fix e2d test to mirror-gateway and crd view field

Signed-off-by: hlts2 <[email protected]>

* define mirror app name

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix golangci warning

Signed-off-by: hlts2 <[email protected]>

* crd generate command for mirror target using mirror-target-values

Signed-off-by: hlts2 <[email protected]>

* add string field validation

Signed-off-by: hlts2 <[email protected]>

* fix build error

Signed-off-by: hlts2 <[email protected]>

* add configuration comment

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* add image tag for debug

Signed-off-by: hlts2 <[email protected]>

* fix rebase error

Signed-off-by: hlts2 <[email protected]>

* fix wait timeout option

Signed-off-by: hlts2 <[email protected]>

* add new option to set mirror address

Signed-off-by: hlts2 <[email protected]>

* add new comment

Signed-off-by: hlts2 <[email protected]>

* bugfix for ingress configuration

Signed-off-by: hlts2 <[email protected]>

* not create lb ingress when mirror is enabled

Signed-off-by: hlts2 <[email protected]>

* not create filter ingress when mirror is enabled

Signed-off-by: hlts2 <[email protected]>

* add error handling for unimplemented codes error

Signed-off-by: hlts2 <[email protected]>

* bugfix register addr process

Signed-off-by: hlts2 <[email protected]>

* fix build error

Signed-off-by: hlts2 <[email protected]>

* Revert "update docker file for build error"

This reverts commit f40986d.

* add new package for docker build error

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway example (#2082)

* add example code

Signed-off-by: hlts2 <[email protected]>

* execute search and getobject request to all vald clusters

Signed-off-by: hlts2 <[email protected]>

* fixed to send data to each cluster based on the dataset

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* add mirror target metrics

Signed-off-by: hlts2 <[email protected]>

* fix example ngt dimension

Signed-off-by: hlts2 <[email protected]>

* make proto/all & make format

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* fix mirror image tag

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway document (#2105)

* add mirror document

Signed-off-by: hlts2 <[email protected]>

* fix svg to png

Signed-off-by: hlts2 <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* style: Format code with prettier and gofumpt

* Update docs/overview/component/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* style: Format code with prettier and gofumpt

* Apply suggestions from code review

* Update docs/overview/component/mirror-gateway.md

* fix expand bug for mirror tutorial

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix filename typo

Signed-off-by: hlts2 <[email protected]>

* add new link

Signed-off-by: hlts2 <[email protected]>

* add cluster role settings for mirror gateway

Signed-off-by: hlts2 <[email protected]>

* add hdf5 installation section

Signed-off-by: hlts2 <[email protected]>

* Update docs/user-guides/mirroring-configuration.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* update link

Signed-off-by: hlts2 <[email protected]>

* delete quote

Signed-off-by: hlts2 <[email protected]>

* update mirror component overview image

Signed-off-by: hlts2 <[email protected]>

* update mirror component overview image

Signed-off-by: hlts2 <[email protected]>

* Update docs/overview/component/mirror-gateway.md

Co-authored-by: Kevin Diu <[email protected]>

* Update docs/overview/component/mirror-gateway.md

* Update docs/api/mirror-gateway.md

Co-authored-by: Yusuke Kadowaki <[email protected]>

* Update docs/api/mirror-gateway.md

* Update docs/user-guides/mirroring-configuration.md

Co-authored-by: Kevin Diu <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kevin Diu <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>

* merge ingress configuration for lb and filter

Signed-off-by: hlts2 <[email protected]>

* fix duplicate definition error

Signed-off-by: hlts2 <[email protected]>

* make proto/all & make schema update & make format

Signed-off-by: hlts2 <[email protected]>

* fix test execution error for mirror handler

Signed-off-by: hlts2 <[email protected]>

* make proto/all format and update schema and fails build error

Signed-off-by: hlts2 <[email protected]>

* use resuable workflow to build mirror image

Signed-off-by: hlts2 <[email protected]>

* use internal sync package and refactor

Signed-off-by: hlts2 <[email protected]>

* fix error handling

Signed-off-by: hlts2 <[email protected]>

* add network policy settings and update deploy command

Signed-off-by: hlts2 <[email protected]>

* add default setting logic

Signed-off-by: hlts2 <[email protected]>

* add new fileld to crd schema and update crd

Signed-off-by: hlts2 <[email protected]>

* fix make command to deploy vald with mirror components

Signed-off-by: hlts2 <[email protected]>

* add e2e test for mirror

Signed-off-by: hlts2 <[email protected]>

* add debug command to get pod status

Signed-off-by: hlts2 <[email protected]>

* bugfix command

Signed-off-by: hlts2 <[email protected]>

* disable sidecar e2e and add mirror e2e command

Signed-off-by: hlts2 <[email protected]>

* update cluster role

Signed-off-by: hlts2 <[email protected]>

* add cluster-info dump command

Signed-off-by: hlts2 <[email protected]>

* Revert "add cluster-info dump command"

This reverts commit 7ece16f.

* debug: add logs command

Signed-off-by: hlts2 <[email protected]>

* Revert "debug: add logs command"

This reverts commit 454eaf3.

* add RemoveByTimestamp for mirror

Signed-off-by: hlts2 <[email protected]>

* fix: update sleep time for operator deploy

Signed-off-by: hlts2 <[email protected]>

* update wait time for vmt

Signed-off-by: hlts2 <[email protected]>

* Returns all connection information in the registration rpc (#2199)

* fix: returns all connection information in the registration rpc

Signed-off-by: hlts2 <[email protected]>

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* refactor: deleted unnecessary parameter

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Add handler logic to continue processing (#2204)

* feat: add new crud logic

Signed-off-by: hlts2 <[email protected]>

* refactor code

Signed-off-by: hlts2 <[email protected]>

* refactor: error handling logic

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix error handling for rpc

Signed-off-by: hlts2 <[email protected]>

* feat: add test for RemoveByTimestamp

Signed-off-by: hlts2 <[email protected]>

* fix: fails error test

Signed-off-by: hlts2 <[email protected]>

* refactor: test logic

Signed-off-by: hlts2 <[email protected]>

* fix: add new condition and add test case

Signed-off-by: hlts2 <[email protected]>

* refactor code

Signed-off-by: hlts2 <[email protected]>

* fix: error handling and comment

Signed-off-by: hlts2 <[email protected]>

* fix: test case name and test case order

Signed-off-by: hlts2 <[email protected]>

* fix: execute format

Signed-off-by: hlts2 <[email protected]>

* fix: add new comment

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Update Mirror Gateway document (#2207)

* fix: update mirror document

Signed-off-by: hlts2 <[email protected]>

* fix: format

Signed-off-by: hlts2 <[email protected]>

* feat: add removeByTimestamp section

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning and refactor document

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* feat: add mirror gateway troubleshooting docs

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 2c492f7 according to the output
from Gofumpt and Prettier.

Details: #2207

* feat: add contents of troubleshooting document

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* fix: grammar

Signed-off-by: hlts2 <[email protected]>

* fix: link path

Signed-off-by: hlts2 <[email protected]>

* fix: invalid link path

Signed-off-by: hlts2 <[email protected]>

* fix: refactor docs

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary contents

Signed-off-by: hlts2 <[email protected]>

* fix: status handling logic document

Signed-off-by: hlts2 <[email protected]>

* fix: document refactor

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix status handling document

Signed-off-by: hlts2 <[email protected]>

* fix: refactor sentence

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* feat: schema and protobuf update

Signed-off-by: hlts2 <[email protected]>

* fix: reviewdog warning and add comment

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 7d7a690 according to the output
from Gofumpt and Prettier.

Details: #2237

* fix: reviewdog warning

Signed-off-by: hlts2 <[email protected]>

* fix: reviewdog warning

Signed-off-by: hlts2 <[email protected]>

* fix: move mirror client mock definition to internal package

Signed-off-by: hlts2 <[email protected]>

* feat: add new handler base logic

Signed-off-by: hlts2 <[email protected]>

* fix: refactor stream list object rpc

Signed-off-by: hlts2 <[email protected]>

* fix: divide logic to other function

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e62a657 according to the output
from Gofumpt and Prettier.

Details: #2237

* make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: return grpc error

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* feat: add mirror service test

Signed-off-by: hlts2 <[email protected]>

* fix: add mock definition code

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in d6580ec according to the output
from Gofumpt and Prettier.

Details: #2237

* fix: use internal method

Signed-off-by: hlts2 <[email protected]>

* fix: delete Exist method because it is not needed

Signed-off-by: hlts2 <[email protected]>

* fix: use address cache insted of connectedAddr method

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* feat: improve connection management and add new test

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* fix: refactor error handling

Signed-off-by: hlts2 <[email protected]>

* feat: add test for syncronizee based on address and small refactor

Signed-off-by: hlts2 <[email protected]>

* fix: improve error handling and example image tag

Signed-off-by: hlts2 <[email protected]>

* fix: schema update and format

Signed-off-by: hlts2 <[email protected]>

* Update charts/vald/values/multi-vald/dev-vald-with-mirror.yaml

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 4c11568 according to the output
from Gofumpt and Prettier.

Details: #2262

* fix: reviewdog warning and deepsource warning

Signed-off-by: hlts2 <[email protected]>

* feat: use new command to docker build

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: update schema and format

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary RPCs due to failure of previous rebasing

Signed-off-by: hlts2 <[email protected]>

* fix: proto and execute make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: revert unnecessary changes

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary code

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: apply feedback for comment

Signed-off-by: hlts2 <[email protected]>

* feat: add internal hash package to generate hash value

Signed-off-by: hlts2 <[email protected]>

* fix: ignore deepsource warning

Signed-off-by: hlts2 <[email protected]>

* feat: add function and method comment

Signed-off-by: hlts2 <[email protected]>

* Update pkg/gateway/mirror/handler/grpc/handler.go

* fix: schema and proto update and format

Signed-off-by: hlts2 <[email protected]>

* feat: add ingress rule for lb and filter

Signed-off-by: hlts2 <[email protected]>

* fix: deleted wildcard path

Signed-off-by: hlts2 <[email protected]>

* fix: ingress unified into one manifest

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: lb ingress port settings and small refactor

Signed-off-by: hlts2 <[email protected]>

* feat: add cluster wide ingress resource name

Signed-off-by: hlts2 <[email protected]>

* fix: add deploy ingress resource using make command

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Kevin Diu <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>

* Update values-readreplica.yaml with new index path and in-memory mode

* Update POD_NAME environment variable in e2e.yml

* Add readreplica e2e logic

* Refactor e2e test for read replica

* Update E2E test name and add read replica rotation

* Update E2E test parameters for improved performance

* Refactor setup-e2e action.yaml

* Refactor

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in a68a83d according to the output
from Gofumpt and Prettier.

Details: #2298

* Add kubectl package for Kubernetes operations

* Remove unused param

* Fix function parameter order in kubectl package

* Refactor kubectl.go: Add copyright notice and fix error handling

* FIx error handling

* Prevent command injection

* Apply format

* Seperate k8s client initialization in TestE2EReadReplica

* Fix comment

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in d127dbc according to the output
from Gofumpt and Prettier.

Details: #2298

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: Hiroto Funakoshi <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Kevin Diu <[email protected]>
  • Loading branch information
5 people authored Jan 22, 2024
1 parent a9da833 commit 2ca7187
Show file tree
Hide file tree
Showing 9 changed files with 522 additions and 6 deletions.
128 changes: 128 additions & 0 deletions .github/actions/e2e-deploy-vald-readreplica/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
name: "Deploy Vald Read Replica for E2E test"
description: "A action to deploy vald read replica for E2E test"

inputs:
require_minio:
description: "If Minio is required, set this to true."
required: false
default: "false"
helm_extra_options:
description: "Extra options that passed to Helm command."
required: false
default: ""
values:
description: "Path to the values.yaml that passed to Helm command."
required: false
default: "false"
wait_for_selector:
description: "Label selector used for specifying a pod waited for"
required: false
default: "app=vald-lb-gateway"
wait_for_timeout:
description: "Timeout used for waiting for pods"
required: false
default: "600s"
use_local_charts:
description: "If you want to use local charts, set this to true."
required: false
default: "true"
default_image_tag:
description: "Default image tag. e.g) nightly, vx.x, vx.x.x"
required: true
default: "nightly"
outputs:
POD_NAME:
description: "A pod name that waited for"
value: ${{ steps.get_real_pod_name.outputs.POD_NAME }}

runs:
using: "composite"
steps:
- name: Deploy Minio
id: deploy_minio
shell: bash
if: ${{ inputs.require_minio == 'true' }}
run: |
make K8S_SLEEP_DURATION_FOR_WAIT_COMMAND=10 k8s/external/minio/deploy
- name: Dump Helm values
shell: bash
run: |
cat ${{ inputs.values }}
- name: Deploy vald read replica from remote charts
shell: bash
id: deploy_vald_readreplica_remote
if: ${{ inputs.use_local_charts == 'false' }}
run: |
helm install \
--values ${VALUES} \
--set defaults.image.tag=${DEFAULT_IMAGE_TAG} \
${HELM_EXTRA_OPTIONS} \
--generate-name charts/vald-readreplica
sleep 3
kubectl wait --for=condition=ready pod -l ${WAIT_FOR_SELECTOR} --timeout=${WAIT_FOR_TIMEOUT}
kubectl get pods
podname=`kubectl get pods --selector=${WAIT_FOR_SELECTOR} | tail -1 | awk '{print $1}'`
echo "POD_NAME=${podname}" >> $GITHUB_OUTPUT
env:
DEFAULT_IMAGE_TAG: ${{ inputs.default_image_tag }}
VALUES: ${{ inputs.values }}
HELM_EXTRA_OPTIONS: ${{ inputs.helm_extra_options }}
WAIT_FOR_SELECTOR: ${{ inputs.wait_for_selector }}
WAIT_FOR_TIMEOUT: ${{ inputs.wait_for_timeout }}

- name: Deploy vald read replica from local charts
shell: bash
id: deploy_vald_readreplica_local
if: ${{ inputs.use_local_charts == 'true' }}
run: |
make k8s/vald-readreplica/deploy VERSION=${DEFAULT_IMAGE_TAG} HELM_VALUES=${VALUES} HELM_EXTRA_OPTIONS="${HELM_EXTRA_OPTIONS}"
sleep 3
kubectl wait --for=condition=ready pod -l ${WAIT_FOR_SELECTOR} --timeout=${WAIT_FOR_TIMEOUT}
kubectl get pods
podname=`kubectl get pods --selector=${WAIT_FOR_SELECTOR} | tail -1 | awk '{print $1}'`
echo "POD_NAME=${podname}" >> $GITHUB_OUTPUT
env:
DEFAULT_IMAGE_TAG: ${{ inputs.default_image_tag }}
VALUES: ${{ inputs.values }}
HELM_EXTRA_OPTIONS: ${{ inputs.helm_extra_options }}
WAIT_FOR_SELECTOR: ${{ inputs.wait_for_selector }}
WAIT_FOR_TIMEOUT: ${{ inputs.wait_for_timeout }}

- name: Get real pod name
shell: bash
id: get_real_pod_name
env:
PODNAME_LOCAL_DEPLOY: ${{ steps.deploy_vald_readreplica_local.outputs.POD_NAME }}
PODNAME_REMOTE_DEPLOY: ${{ steps.deploy_vald_readreplica_remote.outputs.POD_NAME }}
# Set GITHUB_OUTPUT to the not empty one, PODNAME_LOCAL_DEPLOY or PODNAME_REMOTE_DEPLOY
run: |
if [[ -n "${PODNAME_LOCAL_DEPLOY}" ]]; then
echo "POD_NAME=${PODNAME_LOCAL_DEPLOY}" >> $GITHUB_OUTPUT
else
echo "POD_NAME=${PODNAME_REMOTE_DEPLOY}" >> $GITHUB_OUTPUT
fi
11 changes: 11 additions & 0 deletions .github/actions/setup-e2e/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ inputs:
description: "If k3d is not required, set this to false"
required: false
default: "true"
require_minikube:
description: "If minikube is not required, set this to true and set require_k3d to false"
required: false
default: "false"
ingress_port:
description: 'If it is not "0", ingress will be exposed to the specified port'
required: false
Expand Down Expand Up @@ -94,6 +98,13 @@ runs:
agents: 3
ingress_port: ${{ inputs.ingress_port }}

- name: Setup Minikube environment
if: ${{ inputs.require_minikube == 'true' }}
shell: bash
run: |
make minikube/install
make minikube/start
- name: Check Kubernetes cluster
shell: bash
run: |
Expand Down
89 changes: 89 additions & 0 deletions .github/helm/values/values-readreplica.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

defaults:
logging:
level: info
networkPolicy:
enabled: true

gateway:
lb:
enabled: true
minReplicas: 1
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi
gateway_config:
index_replica: 3

agent:
minReplicas: 3
maxReplicas: 10
podManagementPolicy: Parallel
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi
ngt:
auto_index_duration_limit: 2m
auto_index_check_duration: 30s
auto_index_length: 1000
dimension: 784
index_path: /var/ngt/index
enable_in_memory_mode: false
persistentVolume:
enabled: true
accessMode: ReadWriteOnce
storageClass: csi-hostpath-sc
size: 1Gi
readreplica:
enabled: true
snapshot_classname: "csi-hostpath-snapclass"
replica: 1

discoverer:
minReplicas: 1
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi

manager:
index:
replicas: 1
resources:
requests:
cpu: 100m
memory: 30Mi
indexer:
auto_index_duration_limit: 2m
auto_index_check_duration: 30s
auto_index_length: 1000
readreplica:
rotator:
enabled: true
initContainers: []
env:
- name: MY_TARGET_REPLICA_ID
value: "0"
56 changes: 56 additions & 0 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,61 @@ jobs:
env:
POD_NAME: ${{ steps.deploy_vald.outputs.POD_NAME }}

e2e-stream-crud-with-readreplica:
name: "E2E test (Stream CRUD) with read replica"
needs: [dump-contexts-to-log]
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4

- name: Set Git config
run: |
git config --global --add safe.directory ${GITHUB_WORKSPACE}
- name: Setup E2E environment
id: setup_e2e
uses: ./.github/actions/setup-e2e
with:
require_k3d: "false"
require_minikube: "true"

- name: Deploy Vald
id: deploy_vald
uses: ./.github/actions/e2e-deploy-vald
with:
helm_extra_options: ${{ steps.setup_e2e.outputs.HELM_EXTRA_OPTIONS }}
values: .github/helm/values/values-readreplica.yaml
wait_for_selector: app=vald-agent-ngt

- name: Deploy Vald Read Replica
id: deploy_vald_readreplica
uses: ./.github/actions/e2e-deploy-vald-readreplica
with:
default_image_tag: ${{ steps.setup_e2e.outputs.DEFAULT_IMAGE_TAG }}
helm_extra_options: ${{ steps.setup_e2e.outputs.HELM_EXTRA_OPTIONS }}
values: .github/helm/values/values-readreplica.yaml
wait_for_selector: app=vald-lb-gateway

- name: Run E2E CRUD with read replica rotation
run: |
make hack/benchmark/assets/dataset/${{ env.DATASET }}
make E2E_BIND_PORT=8081 \
E2E_DATASET_NAME=${{ env.DATASET }} \
E2E_INSERT_COUNT=1000 \
E2E_SEARCH_COUNT=1000 \
E2E_SEARCH_BY_ID_COUNT=1000 \
E2E_GET_OBJECT_COUNT=100 \
E2E_UPDATE_COUNT=100 \
E2E_UPSERT_COUNT=100 \
E2E_REMOVE_COUNT=100 \
E2E_WAIT_FOR_CREATE_INDEX_DURATION=3m \
E2E_TARGET_POD_NAME=${POD_NAME} \
E2E_TARGET_NAMESPACE=default \
e2e/readreplica
env:
POD_NAME: ${{ steps.deploy_vald_readreplica.outputs.POD_NAME }}

e2e-stream-crud-with-mirror:
name: "E2E test (Stream CRUD) with mirror"
needs: [dump-contexts-to-log]
Expand Down Expand Up @@ -391,6 +446,7 @@ jobs:
- e2e-stream-crud-skip-exist-check
- e2e-stream-crud-under-index-management-jobs
- e2e-stream-crud-with-mirror
- e2e-stream-crud-with-readreplica
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down
5 changes: 5 additions & 0 deletions Makefile.d/e2e.mk
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ e2e/insert/search:
e2e/index/job/correction:
$(call run-e2e-crud-test,-run TestE2EIndexJobCorrection)

.PHONY: e2e/readreplica
## run readreplica e2e
e2e/readreplica:
$(call run-e2e-crud-test,-run TestE2EReadReplica)

.PHONY: e2e/maxdim
## run e2e/maxdim
e2e/maxdim:
Expand Down
2 changes: 1 addition & 1 deletion pkg/index/job/readreplica/rotate/service/rotator.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (

snapshotv1 "github.com/kubernetes-csi/external-snapshotter/client/v6/apis/volumesnapshot/v1"
"github.com/vdaas/vald/internal/errors"
client "github.com/vdaas/vald/internal/k8s/client"
"github.com/vdaas/vald/internal/k8s/client"
"github.com/vdaas/vald/internal/log"
"github.com/vdaas/vald/internal/observability/trace"
"github.com/vdaas/vald/internal/sync/errgroup"
Expand Down
Loading

0 comments on commit 2ca7187

Please sign in to comment.