master: add etcd compaction and quota config

[lance6716]

What problem does this PR solve?

part of #1519

What is changed and how it works?

• set default values and expose etcd config for auto-compaction and backend-quota
• enable auto-compaction for a short value in integration test, to see if DM could handle compaction error

Check List

Tests

• Unit test
• Integration test to check DM could handle compaction error

Code changes

• Has exported variable/fields change
• Has persistent data change

Side effects

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation
• Need to be included in the release note

[lance6716]

I tested setting and not setting auto comapction configurations, with writing different value to a same key. The DBSize of endpoint status has no difference most time, while for one configuration old revision key is compacted and other is not.

Will check DBSize of endpoint status in integration test later.

[lance6716]

etcd-io/etcd#8496

according to above link, auto compaction does no help to disk problems, and defregment will stop etcd serving thus need carefully trigger, we might let user configurate a big quota to reduce the chance, and write an FAQ on etcd defregment or provide a helper command

--

update: we might implement rolling defregment in future

[lance6716]

/run-all-tests

[lichunzhu]

/lgtm
But if we want to add this in v2.0.2, we'd better and periodically defragment too.

[lance6716]

But if we want to add this in v2.0.2, we'd better and periodically defragment too.

I think adding this and lacking defragment didn't make things worse. defragment will stop etcd's serving, thus we should let master retire leader to run a defragment. I'm not clear about the logic now

[lichunzhu]

Users can't achieve their goals if they don't do defragment manually. They may think auto-compaction can avoid Error: etcdserver: mvcc: database space exceeded problem. But actually, it's insufficient.

[lance6716]

Users can't achieve their goals if they don't do defragment manually. They may think auto-compaction can avoid Error: etcdserver: mvcc: database space exceeded problem. But actually, it's insufficient.

we could tell them auto-compaction features are the same behaviour with etcd/pd, so their expectation is wrong.

[lichunzhu]

Users can't achieve their goals if they don't do defragment manually. They may think auto-compaction can avoid Error: etcdserver: mvcc: database space exceeded problem. But actually, it's insufficient.

we could tell them auto-compaction features are the same behaviour with etcd/pd, so their expectation is wrong.

OK. We should add this to our documents. I'm curious about how pd handles this problem? I mean how to tell users to use defragment. I check their code there is no periodically defragment, either.

[GMHDBJD]

We may no need to config `auto-compaction-retention in so many integration tests. I think most users will not manually config it.
rest LGTM

[lance6716]

We may no need to config `auto-compaction-retention in so many integration tests. I think most users will not manually config it.
rest LGTM

now we set the efault value to 1h so DM may face etcd compacted error. I set it to a shorter value in integration test to expose failure of etcd compacted error

[GMHDBJD]

/lgtm

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• GMHDBJD
• lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

[lance6716]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 40b63e8

[ti-srebot]

cherry pick to release-2.0 in PR #1534