Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert binlog.py to go version #41

Merged
merged 72 commits into from
Dec 18, 2019
Merged

Convert binlog.py to go version #41

merged 72 commits into from
Dec 18, 2019

Conversation

arindamnayak
Copy link

Ported binlog.py to go version.

arindamnayak and others added 30 commits October 31, 2019 12:17
* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>
* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>
* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>
* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>
* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>
* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>
* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>
* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>
* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>
* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>
* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>
* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>
* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>
* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: saurabh <[email protected]>
Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you write a description for each test case similar to how you already have for TestCharset? Maybe also rename TestMain to TestBinlog? or whatever is appropriate.

Also requires a rebase/merge to get unit tests passing.

Comment on lines +254 to +259
// Insert something that will replicate incorrectly if the charset is not
// propagated through binlog streamer to the destination.

// Vitess tablets default to using utf8, so we insert something crazy and
// pretend it's latin1. If the binlog player doesn't also pretend it's
// latin1, it will be inserted as utf8, which will change its value.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Nice documentation.

@arindamnayak
Copy link
Author

Can you write a description for each test case similar to how you already have for TestCharset? Maybe also rename TestMain to TestBinlog? or whatever is appropriate.

Also requires a rebase/merge to get unit tests passing.

Update the documentations. TestMain contains all setup related code, we can't customize that name as it accepts m *testing.M .

It depends on a upstream PR (initial sharding PR), once that is merged and if this PR looks good to you, I will create PR against upstream, where all these unit test case failure will be fixed automatically.

Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arindamnayak arindamnayak changed the base branch from tal_test_initial_sharding_v3 to tal_test_master December 18, 2019 06:07
@ajeetj
Copy link

ajeetj commented Dec 18, 2019

LGTM, merging in tal_test_master

@ajeetj ajeetj merged commit 76d6f62 into tal_test_master Dec 18, 2019
arindamnayak added a commit that referenced this pull request Dec 18, 2019
Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>
ajeetj pushed a commit that referenced this pull request Dec 19, 2019
ajeetj added a commit that referenced this pull request Dec 20, 2019
* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>

* initial commit for sharding test

Signed-off-by: Arindam Nayak <[email protected]>

* added reconciliation assert

Signed-off-by: Arindam Nayak <[email protected]>

* added insert and stream replication

Signed-off-by: Arindam Nayak <[email protected]>

* added rest assertions including migrateservedtypes

Signed-off-by: Arindam Nayak <[email protected]>

* completed remaining assertions and fixed non-workings

Signed-off-by: Arindam Nayak <[email protected]>

* converted to v3 way of resharding

Signed-off-by: Arindam Nayak <[email protected]>

* restructured the testcases

Signed-off-by: Arindam Nayak <[email protected]>

* added testcase with bytes as sharding key type

Signed-off-by: Arindam Nayak <[email protected]>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <[email protected]>

* removed unused variables

Signed-off-by: saurabh <[email protected]>

* removing comment

Signed-off-by: saurabh <[email protected]>

* addressed review comments

Signed-off-by: saurabh <[email protected]>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <[email protected]>

* added v3 sharding test

Signed-off-by: Arindam Nayak <[email protected]>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <[email protected]>

* inital code for resharding

Signed-off-by: Ajeet jain <[email protected]>

* completed the cluster code

Signed-off-by: Ajeet jain <[email protected]>

* more progress

Signed-off-by: Ajeet jain <[email protected]>

* pull graceful shutdown code

Signed-off-by: Ajeet jain <[email protected]>

* added fe more cases in resharding_test.go

Signed-off-by: Ajeet jain <[email protected]>

* fixed merge issues

Signed-off-by: Ajeet jain <[email protected]>

* running mysql in parallel mode and fe more assertions

Signed-off-by: Ajeet jain <[email protected]>

* used rbr mysql mode

Signed-off-by: Arindam Nayak <[email protected]>

* almost finished with reshading v2 version

Signed-off-by: Ajeet jain <[email protected]>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <[email protected]>

* removed unused variables

Signed-off-by: saurabh <[email protected]>

* removing comment

Signed-off-by: saurabh <[email protected]>

* addressed review comments

Signed-off-by: saurabh <[email protected]>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <[email protected]>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <[email protected]>

* changed code to use v3 way of sharding

Signed-off-by: Arindam Nayak <[email protected]>

* converted to v3 way

Signed-off-by: Ajeet jain <[email protected]>

* added multi split diff and multi sharding

Signed-off-by: Arindam Nayak <[email protected]>

* converted to v3 way

Signed-off-by: Ajeet jain <[email protected]>

* TabletManager remaining test cases

TabletManager remaining test cases in GO
Signed-off-by: Ajeet jain <[email protected]>

* moved sql start to non-blocking mode (#35)

* moved sql start to non-blocking mode

Signed-off-by: Arindam Nayak <[email protected]>

* added testcase for vtgate vars

Signed-off-by: Arindam Nayak <[email protected]>

* implement pr suggestion

Signed-off-by: Arindam Nayak <[email protected]>

* trying to work with throttler

Signed-off-by: Ajeet jain <[email protected]>

* use ks1 mysql ports for other tablets in ks2

Signed-off-by: Arindam Nayak <[email protected]>

* converted on int keys

Signed-off-by: Ajeet jain <[email protected]>

* Rebase test_master with latest Cluster code (#22)

* pick latest fix from cluster_for_test

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* Tabletmanager2 test cases in GO using cluster (#23)

* tabletmanager lock unlock table test case

Signed-off-by: Ajeet jain <[email protected]>

Signed-off-by: Arindam Nayak <[email protected]>

* Vtctld Test cases in Go using cluster (#25)

* converted vtctld_test.py to go

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* Converted schema.py testcase (#26)

* migrated one of testcase from schema.py to schema_test.go

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* Tablet Manager test cases in Go using cluster (#27)

* converted tabletmanager test cases to go

Signed-off-by: Ajeet jain <[email protected]>

Signed-off-by: Arindam Nayak <[email protected]>

* Converted keyspace test to go (#28)

* ported testcase of keyspace_test.py

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* Converted sharded test from py to go (#29)

* Converted sharded test from py to go

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* Added testcase for mysqlctl process

Signed-off-by: saurabh <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* removed unused variables

Signed-off-by: saurabh <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* removing comment

Signed-off-by: saurabh <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* addressed review comments

Signed-off-by: saurabh <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* removed cleandir flag from vttablet teardown

Signed-off-by: Arindam Nayak <[email protected]>

* vtgate buffer testcase migrated in go

Signed-off-by: saurabh <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* TabletManager remaining test cases

TabletManager remaining test cases in GO
Signed-off-by: Ajeet jain <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* moved sql start to non-blocking mode (#35)

* moved sql start to non-blocking mode

Signed-off-by: Arindam Nayak <[email protected]>
Signed-off-by: Arindam Nayak <[email protected]>

* added testcase for vtgate vars

Signed-off-by: Arindam Nayak <[email protected]>

* redirect process errors to log files (#38)

Signed-off-by: Arindam Nayak <[email protected]>

* debug the test error in ci

Signed-off-by: Arindam Nayak <[email protected]>

* fix initial sharding multi

Signed-off-by: Arindam Nayak <[email protected]>

* fixed protected mysqlctl teardown

Signed-off-by: Arindam Nayak <[email protected]>

* removed sleep time

Signed-off-by: Ajeet jain <[email protected]>

* refactor mysqlctl start and stop

Signed-off-by: Arindam Nayak <[email protected]>

* Fixed multisplitdiff command

Signed-off-by: Arindam Nayak <[email protected]>

* added log statement for error condition

Signed-off-by: Arindam Nayak <[email protected]>

* Ignoring Python tests which are moved to GO (#40)

Signed-off-by: Ajeet jain <[email protected]>

* readme for go endtoend test cases

Signed-off-by: Ajeet jain <[email protected]>

* Update README.md

* Fixed byte column test cases

Signed-off-by: Ajeet jain <[email protected]>

* lint fixes

Signed-off-by: Ajeet jain <[email protected]>

* updated comments

Signed-off-by: Arindam Nayak <[email protected]>

* restructure of launch cluster

Signed-off-by: Ajeet jain <[email protected]>

* fix tear down code

Signed-off-by: Ajeet jain <[email protected]>

* Refactor code

Signed-off-by: Arindam Nayak <[email protected]>

* fix the throttler test cases

Signed-off-by: Ajeet jain <[email protected]>

* fix the static checks

Signed-off-by: Ajeet jain <[email protected]>

* disable running python test cases in travis

Signed-off-by: Ajeet jain <[email protected]>

* fixed static checks

Signed-off-by: Ajeet jain <[email protected]>

* removed unrequired line to set RBR mode

Signed-off-by: Ajeet jain <[email protected]>

* Add matrix build for mysql

Signed-off-by: Morgan Tocker <[email protected]>

* Add additional flavors, move flaky test

Signed-off-by: Morgan Tocker <[email protected]>

* vindex: add required NeedVCursor function

There's currently no way to know if a vindex requires the vcursor.
We need this knowledge to prevent vreplication from choosing
a vindex that requires a VCursor. This change requires all vindexes
to report if they need the vcursor or not.

Signed-off-by: Sugu Sougoumarane <[email protected]>

* vrepl: choose best vindex instead of primary

Signed-off-by: Sugu Sougoumarane <[email protected]>

* few of the reparent test cases

Signed-off-by: Ajeet jain <[email protected]>

* vttablet-up.sh : Adds detection of mysqld launch failure.  Given failure, tests for further for          presence of apparmor.  Exits with error and recommendation to disable.

Signed-off-by: ChrisR <[email protected]>

* completed all the reparent tests

Signed-off-by: Ajeet jain <[email protected]>

* Remove MariaDB 10.1

Re-enable unit race
Change CI git checkout to be v2.0
Disable test caching

Signed-off-by: Morgan Tocker <[email protected]>

* Revert some of the MariaDB changes, fix unit_race

Signed-off-by: Morgan Tocker <[email protected]>

* Add debugging for macOS issue

Signed-off-by: Morgan Tocker <[email protected]>

* Add additional flaky test

Signed-off-by: Morgan Tocker <[email protected]>

* Add sleep for macOS

Signed-off-by: Morgan Tocker <[email protected]>

* Disable macOS local example for now (flaky)

Signed-off-by: Morgan Tocker <[email protected]>

* Disable unit_race again

Signed-off-by: Morgan Tocker <[email protected]>

* updated test cases for reparent

Signed-off-by: Ajeet jain <[email protected]>

* Resharding endtoend testcases in Go migrated from Python (#44)


Signed-off-by: Ajeet jain <[email protected]>

* fix vtgate schema testcases

Signed-off-by: Ajeet jain <[email protected]>

* Convert binlog.py to go version (#41)


Signed-off-by: Arindam Nayak <[email protected]>

* fixed range based reparenting

Signed-off-by: Ajeet jain <[email protected]>

* removed unrequired comments

Signed-off-by: Ajeet jain <[email protected]>

* Modified apparmor detection to passive, added additional investigation info.

Signed-off-by: ChrisR <[email protected]>

* make-release-packages.sh: Disable shellcheck 1091 and Add README pointing to local install instructions

Signed-off-by: Gary Edgar <[email protected]>

* review comments addressed

Signed-off-by: Ajeet jain <[email protected]>

* Splitting github endtoend action into 2 files

Signed-off-by: Ajeet jain <[email protected]>

* vindex: NeedVCursor->NeedsVCursor

Signed-off-by: Sugu Sougoumarane <[email protected]>

* Set correct tablet type to fix failing test

Signed-off-by: Arindam Nayak <[email protected]>

* Fixing initial sharding multi

Signed-off-by: Arindam Nayak <[email protected]>

* fix the assertions

Signed-off-by: Arindam Nayak <[email protected]>

* fixed master-spare test case

Signed-off-by: Ajeet jain <[email protected]>

* Testing dockerized test in github workflow

Signed-off-by: Arindam Nayak <[email protected]>

* Vertical split testcase migrated in go (#46)

Vertical split endtoend testcases in Go migrated from Python

Signed-off-by: saurabh <[email protected]>

* Adjusted shard number for testcase

Signed-off-by: Arindam Nayak <[email protected]>

* Reverting changes for config.json

Signed-off-by: Arindam Nayak <[email protected]>

* Refactor vitess/lite Dockerfiles. (vitessio#5592)

Signed-off-by: Anthony Yeh <[email protected]>

Co-authored-by: Arindam Nayak <[email protected]>
Co-authored-by: saurabh408 <[email protected]>
Co-authored-by: Morgan Tocker <[email protected]>
Co-authored-by: Sugu Sougoumarane <[email protected]>
Co-authored-by: ChrisR <[email protected]>
Co-authored-by: Gary Edgar <[email protected]>
Co-authored-by: Anthony Yeh <[email protected]>
Co-authored-by: Deepthi Sigireddi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants