Enable LTO in release build process #4924

solotzg · 2022-05-18T13:00:33Z

What problem does this PR solve?

Issue Number: ref #4909 after #4890

Problem Summary:

What is changed and how it works?

new feature flag thinlto in ./tiflash version.
clean useless modules: reduce binary size from 329M to 293M.
inline function invoke automatically.

Benchmark

use cgroup to limit cpu usage to 500% at most.

mkdir -p /sys/fs/cgroup/cpu/tzg_test
lsof -i:9000 | grep 'TiFlash' | grep -v 'grep' | awk '{print $2}' > /sys/fs/cgroup/cpu/tzg_test/cgroup.procs
echo "500000" > /sys/fs/cgroup/cpu/tzg_test/cpu.cfs_quota_us

MySQL [tpch_10]> desc lineitem;
+-----------------+---------------+------+------+---------+-------+
| Field           | Type          | Null | Key  | Default | Extra |
+-----------------+---------------+------+------+---------+-------+
| L_ORDERKEY      | bigint(20)    | NO   | PRI  | NULL    |       |
| L_PARTKEY       | bigint(20)    | NO   |      | NULL    |       |
| L_SUPPKEY       | bigint(20)    | NO   |      | NULL    |       |
| L_LINENUMBER    | bigint(20)    | NO   | PRI  | NULL    |       |
| L_QUANTITY      | decimal(15,2) | NO   |      | NULL    |       |
| L_EXTENDEDPRICE | decimal(15,2) | NO   |      | NULL    |       |
| L_DISCOUNT      | decimal(15,2) | NO   |      | NULL    |       |
| L_TAX           | decimal(15,2) | NO   |      | NULL    |       |
| L_RETURNFLAG    | char(1)       | NO   |      | NULL    |       |
| L_LINESTATUS    | char(1)       | NO   |      | NULL    |       |
| L_SHIPDATE      | date          | NO   |      | NULL    |       |
| L_COMMITDATE    | date          | NO   |      | NULL    |       |
| L_RECEIPTDATE   | date          | NO   |      | NULL    |       |
| L_SHIPINSTRUCT  | char(25)      | NO   |      | NULL    |       |
| L_SHIPMODE      | char(10)      | NO   |      | NULL    |       |
| L_COMMENT       | varchar(44)   | NO   |      | NULL    |       |
+-----------------+---------------+------+------+---------+-------+
16 rows in set (0.00 sec)

run tpch-10-q1

	LTO	original
AVG Time Cost(s)	6.042	6.322

improve 4.63%

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Test release build under x86 and arm64

docker run --rm -it -m 40G -v `pwd`:/home/jenkins/agent/workspace/tiflash-build-common/tiflash hub.pingcap.net/tiflash/tiflash-llvm-base:amd64 sh
sh-4.2# NPROC=12 /home/jenkins/agent/workspace/tiflash-build-common/tiflash/release-centos7-llvm/scripts/build-release.sh

sh-4.2# ./tiflash version
TiFlash
Release Version: v6.1.0-alpha-186-g7eb89fa-dirty
Edition:         Community
Git Commit Hash: 7eb89fac9cc36b789fb0aaf449ac95e1b5897de2
Git Branch:      lto
UTC Build Time:  2022-05-18 09:43:31
Enable Features: jemalloc avx avx512 unwind thinlto
Profile:         RELWITHDEBINFO
...

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: Zhigao Tong <[email protected]>

ti-chi-bot · 2022-05-18T13:00:36Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

SchrodingerZhu
zanmato1984

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

zanmato1984

LGTM

zanmato1984 · 2022-05-19T04:25:08Z

But I suggest merging this PR after code freeze for 6.1. Leave the uncertainty behind it.

zanmato1984 · 2022-05-19T04:25:15Z

/hold

solotzg · 2022-05-23T06:24:58Z

/merge

ti-chi-bot · 2022-05-23T06:24:59Z

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-05-23T06:25:01Z

This pull request has been accepted and is ready to merge.

Commit hash: b269457

solotzg · 2022-05-23T10:29:58Z

/merge

ti-chi-bot · 2022-05-23T10:29:58Z

@solotzg: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-05-23T10:34:44Z

Coverage for changed files

Filename                                                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/jenkins/agent/workspace/tiflash-build-common/tiflash/dbms/src/Common/TiFlashBuildInfo.cpp          18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                    18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18224      9786             46.30%    204134  97943        52.02%

full coverage report (for internal network access only)

sre-bot · 2022-05-23T10:54:10Z

Coverage for changed files

Filename                                                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/jenkins/agent/workspace/tiflash-build-common/tiflash/dbms/src/Common/TiFlashBuildInfo.cpp          18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                    18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18224      9785             46.31%    204181  97962        52.02%

full coverage report (for internal network access only)

solotzg · 2022-05-23T11:23:28Z

/unhold

sre-bot · 2022-05-23T11:40:27Z

Coverage for changed files

Filename                                                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/jenkins/agent/workspace/tiflash-build-common/tiflash/dbms/src/Common/TiFlashBuildInfo.cpp          18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                    18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18222      9784             46.31%    204112  97935        52.02%

full coverage report (for internal network access only)

sre-bot · 2022-05-23T11:57:33Z

Coverage for changed files

Filename                                                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/jenkins/agent/workspace/tiflash-build-common/tiflash/dbms/src/Common/TiFlashBuildInfo.cpp          18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                    18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18222      9784             46.31%    204122  97946        52.02%

full coverage report (for internal network access only)

sre-bot · 2022-06-06T10:52:08Z

Coverage for changed files

Filename                                                                                            Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/jenkins/agent/workspace/tiflash-build-common/tiflash/dbms/src/Common/TiFlashBuildInfo.cpp          18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                    18                 0   100.00%          10                 0   100.00%          55                 0   100.00%           0                 0         -

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18222      9784             46.31%    204122  97963        52.01%

full coverage report (for internal network access only)

solotzg added 2 commits May 18, 2022 17:25

Enable LTO in release build process

7eb89fa

Signed-off-by: Zhigao Tong <[email protected]>

rollback

ea1612a

solotzg added the type/enhancement The issue or PR belongs to an enhancement. label May 18, 2022

solotzg requested review from zanmato1984 and SchrodingerZhu May 18, 2022 13:00

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 18, 2022

solotzg mentioned this pull request May 18, 2022

Enhancement For Code Quality & Performance #4909

Open

6 tasks

Merge branch 'master' into lto

d1b0996

SchrodingerZhu approved these changes May 19, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 19, 2022

zanmato1984 approved these changes May 19, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 19, 2022

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 19, 2022

solotzg added 3 commits May 19, 2022 13:32

Merge branch 'master' into lto

2fdd69a

Merge branch 'master' into lto

9211f1c

Merge branch 'master' into lto

b269457

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 23, 2022

Merge branch 'master' into lto

b1e67f1

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 23, 2022

Merge branch 'master' into lto

062cceb

Merge branch 'master' into lto

4d7bc7e

ti-chi-bot merged commit d1975b8 into pingcap:master May 23, 2022

solotzg mentioned this pull request Jun 6, 2022

Add Precompiled header for modules & Refine by PImpl to accelerate release build #5047

Merged

12 tasks

solotzg mentioned this pull request Jun 17, 2022

Add options to support Profile Guided Optimization #5160

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable LTO in release build process #4924

Enable LTO in release build process #4924

solotzg commented May 18, 2022 •

edited

Loading

ti-chi-bot commented May 18, 2022 •

edited

Loading

zanmato1984 left a comment

zanmato1984 commented May 19, 2022

zanmato1984 commented May 19, 2022

solotzg commented May 23, 2022

ti-chi-bot commented May 23, 2022

ti-chi-bot commented May 23, 2022

solotzg commented May 23, 2022

ti-chi-bot commented May 23, 2022

sre-bot commented May 23, 2022

sre-bot commented May 23, 2022

solotzg commented May 23, 2022

sre-bot commented May 23, 2022

sre-bot commented May 23, 2022

sre-bot commented Jun 6, 2022

Enable LTO in release build process #4924

Enable LTO in release build process #4924

Conversation

solotzg commented May 18, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Benchmark

Check List

Release note

ti-chi-bot commented May 18, 2022 • edited Loading

zanmato1984 left a comment

Choose a reason for hiding this comment

zanmato1984 commented May 19, 2022

zanmato1984 commented May 19, 2022

solotzg commented May 23, 2022

ti-chi-bot commented May 23, 2022

ti-chi-bot commented May 23, 2022

solotzg commented May 23, 2022

ti-chi-bot commented May 23, 2022

sre-bot commented May 23, 2022

Coverage for changed files

Coverage summary

sre-bot commented May 23, 2022

Coverage for changed files

Coverage summary

solotzg commented May 23, 2022

sre-bot commented May 23, 2022

Coverage for changed files

Coverage summary

sre-bot commented May 23, 2022

Coverage for changed files

Coverage summary

sre-bot commented Jun 6, 2022

Coverage for changed files

Coverage summary

solotzg commented May 18, 2022 •

edited

Loading

ti-chi-bot commented May 18, 2022 •

edited

Loading