Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: make the memory tracker of Jsonobjectagg more accurate #23024

Merged
merged 11 commits into from
Mar 4, 2021
Merged

executor: make the memory tracker of Jsonobjectagg more accurate #23024

merged 11 commits into from
Mar 4, 2021

Conversation

wshwsh12
Copy link
Contributor

@wshwsh12 wshwsh12 commented Mar 1, 2021

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Proposal: xxx

What's Changed:
We need estimate the memory usage of map[string]interface{} in partialResult4JsonObjectAgg, and trace it.

How it Works:

Related changes

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM
  • Breaking backward compatibility

Release note

  • No release note

@wshwsh12 wshwsh12 requested a review from a team as a code owner March 1, 2021 08:17
@wshwsh12 wshwsh12 requested review from lzmhhh123 and removed request for a team March 1, 2021 08:17
@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 1, 2021
@wshwsh12 wshwsh12 changed the title executor: make the meory tracker of Jsonobjectagg more accurate. executor: make the memory tracker of Jsonobjectagg more accurate Mar 1, 2021
@wshwsh12 wshwsh12 requested review from XuHuaiyu and ichn-hu and removed request for lzmhhh123 March 1, 2021 09:32
ichn-hu
ichn-hu requested changes Mar 2, 2021
View reviewed changes
Copy link
Contributor

@ichn-hu ichn-hu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

executor/aggfuncs/func_json_objectagg.go Outdated Show resolved Hide resolved
executor/aggfuncs/func_json_objectagg.go Outdated Show resolved Hide resolved
@wshwsh12 wshwsh12 requested a review from ichn-hu March 2, 2021 09:18
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 2, 2021
XuHuaiyu
XuHuaiyu reviewed Mar 2, 2021
View reviewed changes
executor/aggfuncs/set_with_memory_usage.go Outdated
const (
// Maximum average load of a bucket that triggers growth is 6.5.
// Represent as loadFactorNum/loadFactDen, to allow integer math.
loadFactorNum = 13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the var defined in aggregate.go?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. It will cause cycle import.
I try to move the two var to package set and export it. PTAL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can move it to util/hack.go

And add a comment for these 2 consts to describe that they are from the golang definition.

@XuHuaiyu XuHuaiyu added sig/execution SIG execution type/enhancement The issue or PR belongs to an enhancement. labels Mar 2, 2021
@wshwsh12 wshwsh12 requested a review from XuHuaiyu March 3, 2021 06:18
@ti-chi-bot
Copy link
Member

@XuHuaiyu: Please use /LGTM instead of LGTM when you want to approve the pull request by comment.
If you use the GitHub review feature, please approve the PR directly, the comment will not take effect in the GitHub review feature.
If you have any qustions please refer to lgtm command help or lgtm plugin design.

If you have approved this PR, please ignore this reply. This reply is being used as a temporary reply during the migration of the new bot and will be removed on April 1.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

XuHuaiyu
XuHuaiyu approved these changes Mar 4, 2021
View reviewed changes
Copy link
Contributor

@XuHuaiyu XuHuaiyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • XuHuaiyu
  • ichn-hu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 4, 2021
@XuHuaiyu
Copy link
Contributor

XuHuaiyu commented Mar 4, 2021

/merge

@ti-chi-bot
Copy link
Member

@XuHuaiyu: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 10a4646

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 4, 2021
@ti-chi-bot
Copy link
Member

@wshwsh12: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@wshwsh12
Copy link
Contributor Author

wshwsh12 commented Mar 4, 2021

/run-tics-test

@ti-chi-bot ti-chi-bot merged commit 8761adc into pingcap:master Mar 4, 2021
@wshwsh12 wshwsh12 deleted the jsonobjectagg branch April 7, 2021 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ichn-hu ichn-hu ichn-hu approved these changes

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

Assignees
No one assigned
Labels
sig/execution SIG execution size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wshwsh12 @ti-chi-bot @XuHuaiyu @ichn-hu