linghtning/backend: optimize local writer concurrency and memory usage #753

glorv · 2021-02-25T04:17:16Z

What problem does this PR solve?

Optimize local writer performance and memory usage.

What is changed and how it works?

Optimize local writer by:

Open an index LocalWriter for each chunk to avoid the bottle neck at write index kvs
Replace the async ApplendRows with sync operation, to optimize the memory usage and simplify the logic
decrease the encode/deliverLoop chan to decrease memory usage
use a memory buffer to restore the temp kv pairs before write them to SST file
Manually compaction small SST files into a single SST file and then ingest it into pebble

Bench result:
The benchmark tests were run on a 40core machine with an NVME disk. and based on the following three data and table schema:

DataSet1. 14k warehouse tpcc data

DataSet2. 1k warehouse order_line table with 3 indexes. Thus each row generates 4 kvs.

PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
KEY `idx_d_i` (`ol_d_id`, `ol_i_id`),
KEY `idx_d_w_supply` (`ol_d_id`, `ol_w_id`, `ol_supply_w_id`)

DataSet3. 1k warehouse order_line table with 3 indexes. Thus each row generates 8 kvs.

PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
KEY `idx_d_i` (`ol_d_id`,`ol_i_id`),
KEY `idx_d_w_supply` (`ol_d_id`,`ol_w_id`,`ol_supply_w_id`),
KEY `idx_o_amount` (`ol_o_id`,`ol_amount`),
KEY `idx_d_supply` (`ol_d_id`,`ol_supply_w_id`),
KEY `idx_o_d_i` (`ol_o_id`,`ol_d_id`,`ol_i_id`),
KEY `idx_i_id` (`ol_i_id`)

Bench Result:

DataSet	Branch	Peak Memory	Cost Time
DataSet1	master	34GB	2h25m
DataSet1	opt-local-writer	29GB	1h46m
DataSet2	master	30GB	24m57s
DataSet2	opt-local-writer	30GB	9m32s
DataSet3	master	64GB	1h22m
DataSet3	opt-local-writer	33GB	33m

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has interface methods change

Side effects

Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch

Release Note

Optimize lightning local backend memory efficiency and performance

…riter

kennytm · 2021-04-29T07:37:13Z

/merge

ti-chi-bot · 2021-04-29T07:37:15Z

@kennytm: /merge in this pull request requires 3 /lgtm.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

kennytm · 2021-04-29T07:37:28Z

Huh.

kennytm

rest LGTM

kennytm · 2021-04-29T07:49:15Z

pkg/lightning/backend/local/local.go

@@ -434,7 +820,7 @@ func NewLocalBackend(
 	g glue.Glue,
 	maxOpenFiles int,
 ) (backend.Backend, error) {
-	localFile := cfg.SortedKVDir
+	File := cfg.SortedKVDir


why rename the variable. as least use a lower case name.

This must be done by the IDE when I refactor the LocalFile struct..

😂😂😂

kennytm · 2021-04-29T07:51:31Z

pkg/lightning/backend/local/local.go

@@ -628,27 +1036,62 @@ func (local *local) openEngineDB(engineUUID uuid.UUID, readOnly bool) (*pebble.D
 		// set threshold to half of the max open files to avoid trigger compaction
 		L0CompactionThreshold: math.MaxInt32,
 		L0StopWritesThreshold: math.MaxInt32,
+		LBaseMaxBytes:         16 << 40,


Suggested change

LBaseMaxBytes: 16 << 40,

LBaseMaxBytes: 16 * units.TiB,

kennytm · 2021-04-29T07:51:50Z

pkg/lightning/backend/local/local.go

+			// 16GB
+			TargetFileSize: 16 << 30,


Suggested change

// 16GB

TargetFileSize: 16 << 30,

TargetFileSize: 16 * units.GiB,

kennytm · 2021-04-29T07:53:36Z

pkg/lightning/backend/local/local.go

+		iter, err := reader.NewIter([]byte{'t'}, []byte{'u'})
+		if err != nil {
+			return nil, errors.Trace(err)
+		}
+		key, val := iter.SeekGE([]byte{'t'})


🤔 seems unnecessary to restrict the range. we don't store non-t prefixed stuff in the SST.

@glorv (line 2748 is included)

I supposed the NewIter must receive a not empty bound, but found we can just pass nil, nil instead. 😂

kennytm · 2021-04-29T07:56:30Z

pkg/lightning/restore/restore.go

+	compactionLowerThreshold = 512 << 20 // 512M
+	compactionUpperThreshold = 32 << 30  // 32GB


Suggested change

compactionLowerThreshold = 512 << 20 // 512M

compactionUpperThreshold = 32 << 30 // 32GB

compactionLowerThreshold = 512 * units.MiB

compactionUpperThreshold = 32 * units.GiB

glorv · 2021-04-29T07:58:22Z

Huh.

I add the require-LGT3 label for this PR since the change is quite big. Please approve this PR, then we can merge it. @kennytm 😅

kennytm · 2021-04-29T08:00:10Z

@glorv see above 😅

glorv · 2021-04-29T08:20:20Z

/run-all-tests

kennytm · 2021-04-29T09:42:01Z

/lgtm

ti-chi-bot · 2021-04-29T09:42:03Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Little-Wallace
kennytm
lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

kennytm · 2021-04-29T09:42:13Z

/merge

ti-chi-bot · 2021-04-29T09:42:15Z

This pull request has been accepted and is ready to merge.

Commit hash: 1dff359

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2021-04-29T09:45:15Z

cherry pick to release-4.0 in PR #1074

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2021-04-29T09:45:37Z

cherry pick to release-5.0 in PR #1075

optmize local writer

8236d66

glorv added needs-cherry-pick-release-4.0 needs-cherry-pick-5.0-rc type/enhancement component/import labels Feb 25, 2021

glorv added 3 commits February 25, 2021 13:34

fix lint

eedb83f

Merge branch 'master' into opt-local-writer

60a1bfe

fix lint

b4f3da4

glorv mentioned this pull request Feb 25, 2021

backend/local: optimize local writer concurrency and memory usage pingcap/tidb-lightning#560

Closed

glorv removed the needs-cherry-pick-5.0-rc label Feb 25, 2021

glorv added 4 commits February 25, 2021 22:14

Merge branch 'master' into opt-local-writer

cc7a64d

avoid flush local writer caused panic

ef411bc

Merge branch 'master' into opt-local-writer

1f3ddc9

fix flush queue

4777bcf

glorv force-pushed the opt-local-writer branch 2 times, most recently from 9f34e01 to 0f2442c Compare March 1, 2021 03:06

add a unit test for local writer ingestSST

a22bbf1

glorv force-pushed the opt-local-writer branch from 0f2442c to a22bbf1 Compare March 1, 2021 03:10

glorv requested review from Little-Wallace, kennytm, 3pointer and lance6716 March 2, 2021 02:17

glorv added 2 commits March 2, 2021 19:37

fix duplicated chunk counter

c6805d2

Merge branch 'master' into opt-local-writer

39c5680

This was referenced Mar 10, 2021

pkg/lightning: temporarily disable local writer #843

Closed

lightning/backend/local: use batch write instead of ingest on overlap #857

Closed

ti-chi-bot added the needs-rebase label Mar 24, 2021

glorv added 2 commits March 25, 2021 19:14

batch ingest ssts

092cee3

Merge branch 'master' of ssh://github.com/pingcap/br into opt-local-w…

364b704

…riter

ti-chi-bot added the size/XXL label Mar 25, 2021

simplify addMetas

51fbf23

kennytm reviewed Apr 29, 2021

View reviewed changes

resolve comments

58f8ba9

do not explicit set the reader.NewIter bound

1dff359

ti-chi-bot added status/LGT3 LGTM3. This PR looks very good to our bot. and removed status/LGT2 LGTM2 labels Apr 29, 2021

ti-chi-bot added the status/can-merge label Apr 29, 2021

ti-chi-bot merged commit 6fd7b9a into pingcap:master Apr 29, 2021

ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Apr 29, 2021

cherry pick pingcap#753 to release-4.0

b7dc514

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Apr 29, 2021

linghtning/backend: optimize local writer concurrency and memory usage (#753) #1074

Closed

ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Apr 29, 2021

cherry pick pingcap#753 to release-5.0

51ecb3b

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Apr 29, 2021

linghtning/backend: optimize local writer concurrency and memory usage (#753) #1075

Closed

glorv deleted the opt-local-writer branch April 29, 2021 10:00

glorv restored the opt-local-writer branch April 29, 2021 11:45

glorv deleted the opt-local-writer branch April 29, 2021 11:46

This was referenced May 13, 2021

lightning OOM during importing 10TB tpcc data #937

Closed

Lightning: stuck after runs out disk space #977

Closed

ti-chi-bot mentioned this pull request Jun 10, 2021

lightning: optimize lightning memory and perf (#1020) #1195

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linghtning/backend: optimize local writer concurrency and memory usage #753

linghtning/backend: optimize local writer concurrency and memory usage #753

glorv commented Feb 25, 2021

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

kennytm commented Apr 29, 2021

kennytm left a comment

kennytm Apr 29, 2021

glorv Apr 29, 2021

kennytm Apr 29, 2021

glorv Apr 29, 2021

kennytm Apr 29, 2021

kennytm Apr 29, 2021

kennytm Apr 29, 2021

kennytm Apr 29, 2021

glorv Apr 29, 2021

kennytm Apr 29, 2021

glorv commented Apr 29, 2021 •

edited

Loading

kennytm commented Apr 29, 2021

glorv commented Apr 29, 2021

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

ti-srebot commented Apr 29, 2021

ti-srebot commented Apr 29, 2021

	// 16GB
	TargetFileSize: 16 << 30,
	TargetFileSize: 16 * units.GiB,

		compactionLowerThreshold = 512 << 20 // 512M
		compactionUpperThreshold = 32 << 30 // 32GB

linghtning/backend: optimize local writer concurrency and memory usage #753

linghtning/backend: optimize local writer concurrency and memory usage #753

Conversation

glorv commented Feb 25, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release Note

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

kennytm commented Apr 29, 2021

kennytm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorv commented Apr 29, 2021 • edited Loading

kennytm commented Apr 29, 2021

glorv commented Apr 29, 2021

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

kennytm commented Apr 29, 2021

ti-chi-bot commented Apr 29, 2021

ti-srebot commented Apr 29, 2021

ti-srebot commented Apr 29, 2021

glorv commented Apr 29, 2021 •

edited

Loading