Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drainer/: Reduce memory usage (#735) #737

Merged
merged 23 commits into from
Sep 25, 2019
Merged

drainer/: Reduce memory usage (#735) #737

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0dfc864
Change unreasonable buff size and reduce memory usage
lichunzhu Sep 2, 2019
212d716
Merge branch 'master' of https://github.com/pingcap/tidb-binlog into …
lichunzhu Sep 2, 2019
4520757
wait for 0.01s for goroutines to update status
lichunzhu Sep 2, 2019
04d71a1
Merge branch 'master' of https://github.com/pingcap/tidb-binlog into …
lichunzhu Sep 9, 2019
8a6124a
change defaultBinlogItemCount to 8; revise /tests/status/run.sh to sk…
lichunzhu Sep 9, 2019
b860497
fix check error
lichunzhu Sep 9, 2019
dafce05
Merge branch 'master' of https://github.com/pingcap/tidb-binlog into …
lichunzhu Sep 11, 2019
f9fc0e0
Merge branch 'master' of https://github.com/pingcap/tidb-binlog into …
lichunzhu Sep 11, 2019
e21f41c
Merge branch 'czli/drainer/reduceMemoryUsage' of https://github.com/l…
lichunzhu Sep 11, 2019
9bd6252
eliminate loader & syncer buffer usage
lichunzhu Sep 11, 2019
e513e0d
Merge branch 'czli/drainer/reduceMemoryUsage' of https://github.com/l…
lichunzhu Sep 12, 2019
4b74231
reduce memory usage for success channel
lichunzhu Sep 12, 2019
1e51514
add txn manager
lichunzhu Sep 16, 2019
60bd55e
remove info clear and reduce success channel buffer
lichunzhu Sep 18, 2019
be6146d
Merge branch 'master' into czli/drainer/reduceMemoryUsage
july2993 Sep 23, 2019
9b4a2e5
Add more comments and logs for txnManager. Refine txnManager's code.
lichunzhu Sep 24, 2019
e6aded0
simplify atomic operations
lichunzhu Sep 24, 2019
6ddab70
Merge branch 'master' into czli/drainer/reduceMemoryUsage
IANTHEREAL Sep 24, 2019
03427a7
add two unit tests for txnManager to test whether txnManager can quit…
lichunzhu Sep 25, 2019
21d4efc
reduce wait time
lichunzhu Sep 25, 2019
6789bdf
simplify load_test
lichunzhu Sep 25, 2019
81e586c
simplify failMsg
lichunzhu Sep 25, 2019
64aa654
Update pkg/loader/load_test.go
lichunzhu Sep 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions pkg/loader/load_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,7 @@ func (s *txnManagerSuite) TestRunTxnManager(c *check.C) {
for i := 0; i < 5; i++ {
select {
case input <- txn:
case <-time.After(50 * time.Microsecond):
case <-time.After(10 * time.Microsecond):
c.Fatal("txnManager gets blocked while receiving txns")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the 5th txn get blocked sending to input, why didn't this test fail?

Copy link
Contributor Author

@lichunzhu lichunzhu Sep 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 5th txn was picked from input but can't be added to the txnManager.cacheChan. Although it gets blocked at cond.Wait() but it's still picked out from input. By the way, we can't know a txn's size when it is not picked out from input.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check c.Assert(output, check.HasLen, 4) can make sure it's not added.

}
}
Expand All @@ -413,13 +413,13 @@ func (s *txnManagerSuite) TestRunTxnManager(c *check.C) {
case t := <-output:
txnManager.pop(t)
c.Assert(t, check.DeepEquals, txn)
case <-time.After(50 * time.Microsecond):
default:
c.Fatal("Fail to pick txn from txnManager")
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

select {
case t := <-output:
case <-time.After...
}

This pattern appears a lot in this test, is it possible to extract this as a helper function to simplify the test?

// Now txn won't be blocked but txnManager should be blocked at cond.Wait()
select {
case input <- txn:
case <-time.After(50 * time.Microsecond):
case <-time.After(10 * time.Microsecond):
c.Fatal("txnManager gets blocked while receiving txns")
}
// close txnManager and output should be closed when txnManager is closed
Expand Down Expand Up @@ -455,21 +455,21 @@ func (s *txnManagerSuite) TestAddBigTxn(c *check.C) {
}
select {
case input <- txnBig:
case <-time.After(50 * time.Microsecond):
case <-time.After(10 * time.Microsecond):
c.Fatal("txnManager gets blocked while receiving txns")
}
select {
case t := <-output:
txnManager.pop(t)
c.Assert(t, check.DeepEquals, txnSmall)
case <-time.After(50 * time.Microsecond):
default:
c.Fatal("Fail to pick txn from txnManager")
}
select {
case t := <-output:
txnManager.pop(t)
c.Assert(t, check.DeepEquals, txnBig)
case <-time.After(50 * time.Microsecond):
case <-time.After(10 * time.Microsecond):
c.Fatal("Fail to pick txn from txnManager")
}
txnManager.Close()
Expand Down Expand Up @@ -501,7 +501,7 @@ func (s *txnManagerSuite) TestCloseLoaderInput(c *check.C) {
case t := <-output:
txnManager.pop(t)
c.Assert(t, check.DeepEquals, txn)
case <-time.After(50 * time.Microsecond):
case <-time.After(10 * time.Microsecond):
c.Fatal("Fail to pick txn from txnManager")
}
// output should be closed when input is closed
Expand Down