operator: print callstack when exiting before preparing done #51371

YuJuncen · 2024-02-27T11:28:58Z

What problem does this PR solve?

Issue Number: close #51370

Problem Summary:
It is hard to debug what happens when the init pod get timed out.

What changed and how does it work?

This PR will print all gorouintes to stdout once the prepare-for-snapshot-backup command exits before finishing the full prepare stage.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

When we are not yet finished:

When we have finished prepare stage:

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Yu Juncen <[email protected]>

tiprow · 2024-02-27T11:29:13Z

Hi @YuJuncen. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Yu Juncen <[email protected]>

codecov · 2024-02-27T11:49:14Z

Codecov Report

Attention: Patch coverage is 40.90909% with 26 lines in your changes missing coverage. Please review.

Project coverage is 73.0579%. Comparing base (15a8459) to head (84f6513).
Report is 1950 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #51371        +/-   ##
================================================
+ Coverage   70.6430%   73.0579%   +2.4149%     
================================================
  Files          1459       1465         +6     
  Lines        434200     453774     +19574     
================================================
+ Hits         306732     331518     +24786     
+ Misses       108243     101896      -6347     
- Partials      19225      20360      +1135

Flag	Coverage Δ
integration	`49.0359% <9.3023%> (?)`
unit	`70.6749% <36.3636%> (+0.2453%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`50.5688% <40.9090%> (+4.6943%)`	⬆️

BornChanger · 2024-02-28T06:06:07Z

br/pkg/utils/misc.go

+func StartExitSingleListener(ctx context.Context) (context.Context, context.CancelFunc) {
+	cx, cancel := context.WithCancel(ctx)
+	sc := make(chan os.Signal, 1)
+	signal.Notify(sc,


To confirm, will a normal init job exit print the go routines also?

No, goroutine dump won't be printed once prepare done.

BornChanger · 2024-02-28T06:23:32Z

/cc @WangLe1321 ptal.

WangLe1321 · 2024-02-28T06:45:25Z

br/pkg/utils/misc.go

+		fmt.Println()
+		printDelimate(fmt.Sprintf("Got signal %v to exit.", sig))
+		printDelimate(fmt.Sprintf("Required Goroutine Dump = %v", dumpGoroutine))
+		if DumpGoroutineWhenExit.Load() {


Why not just use dumpGoroutine?

WangLe1321 · 2024-02-28T06:48:09Z

br/pkg/utils/misc.go

+		printDelimate(fmt.Sprintf("Required Goroutine Dump = %v", dumpGoroutine))
+		if DumpGoroutineWhenExit.Load() {
+			printDelimate("Start Dumping Goroutine")
+			os.Stdout.Write(AllStackInfo())


Why not use fmt.Println(AllStackInfo())

debug.PrintStack() uses write. I think there isn't many difference between them.

Signed-off-by: Yu Juncen <[email protected]>

ti-chi-bot · 2024-03-06T03:33:28Z

@WangLe1321: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

3pointer

rest LGTM

3pointer · 2024-03-06T08:36:11Z

br/pkg/task/operator/cmd.go

@@ -141,6 +142,7 @@ func AdaptEnvForSnapshotBackup(ctx context.Context, cfg *PauseGcConfig) error {
 		if cfg.OnAllReady != nil {
 			cfg.OnAllReady()
 		}
+		utils.DumpGoroutineWhenExit.Store(false)


it seems no need set it back

ti-chi-bot · 2024-03-06T08:36:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, BornChanger, WangLe1321

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~br/OWNERS~~ [3pointer,BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-03-06T08:36:37Z

[LGTM Timeline notifier]

Timeline:

2024-03-06 03:38:34.839954412 +0000 UTC m=+224741.862200799: ☑️ agreed by BornChanger.
2024-03-06 08:36:36.71644493 +0000 UTC m=+242623.738691318: ☑️ agreed by 3pointer.

tiprow · 2024-03-06T08:39:06Z

@YuJuncen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tidb_parser_test	`84f6513`	link	true	`/test tidb_parser_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

BornChanger · 2024-03-06T09:37:16Z

/retest

tiprow · 2024-03-06T09:37:39Z

@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

YuJuncen · 2024-03-06T10:44:02Z

/run-cherry-picker

YuJuncen · 2024-03-06T10:44:24Z

/run-cherry-picker

ti-chi-bot · 2024-03-06T10:45:00Z

In response to a cherrypick label: new pull request created to branch release-6.5: #51549.

…#51549) close #51370

…#51371) (pingcap#46) (pingcap#61) close pingcap#51370 Co-authored-by: 山岚 <[email protected]>

…#51371) (pingcap#46) close pingcap#51370 Co-authored-by: 山岚 <[email protected]>

ti-chi-bot · 2024-11-03T04:49:24Z

In response to a cherrypick label: new pull request created to branch release-7.5: #57081.

ti-chi-bot · 2024-11-03T04:49:29Z

In response to a cherrypick label: new pull request created to branch release-7.1: #57082.

Signed-off-by: ti-chi-bot <[email protected]>

…#57081) close #51370

YuJuncen added 2 commits February 27, 2024 15:43

print callstack when exiting

8d86bfb

Signed-off-by: Yu Juncen <[email protected]>

added more logs

6b593c8

Signed-off-by: Yu Juncen <[email protected]>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 27, 2024

better logging

961680d

Signed-off-by: Yu Juncen <[email protected]>

BornChanger reviewed Feb 28, 2024

View reviewed changes

BornChanger mentioned this pull request Feb 28, 2024

EBS BR support to multiple k8s cluster TiDB pingcap/tidb-operator#5003

Open

WangLe1321 reviewed Feb 28, 2024

View reviewed changes

YuJuncen added 2 commits February 29, 2024 17:59

address comments

ce9c7db

Signed-off-by: Yu Juncen <[email protected]>

make clippy happy

84f6513

Signed-off-by: Yu Juncen <[email protected]>

WangLe1321 approved these changes Mar 6, 2024

View reviewed changes

BornChanger approved these changes Mar 6, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 6, 2024

3pointer reviewed Mar 6, 2024

View reviewed changes

3pointer approved these changes Mar 6, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 6, 2024

ti-chi-bot bot merged commit ff794e5 into pingcap:master Mar 6, 2024
32 of 34 checks passed

YuJuncen added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Mar 6, 2024

ti-chi-bot mentioned this pull request Mar 6, 2024

operator: print callstack when exiting before preparing done (#51371) #51549

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Mar 23, 2024

operator: print callstack when exiting before preparing done (#51371) (…

ea8ec37

…#51549) close #51370

mittalrishabh pushed a commit to mittalrishabh/tidb that referenced this pull request May 6, 2024

operator: print callstack when exiting before preparing done (pingcap…

7967b0b

…#51371) (pingcap#46) (pingcap#61) close pingcap#51370 Co-authored-by: 山岚 <[email protected]>

mittalrishabh pushed a commit to mittalrishabh/tidb that referenced this pull request May 30, 2024

operator: print callstack when exiting before preparing done (pingcap…

d95943b

…#51371) (pingcap#46) close pingcap#51370 Co-authored-by: 山岚 <[email protected]>

BornChanger added needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels Nov 3, 2024

ti-chi-bot mentioned this pull request Nov 3, 2024

operator: print callstack when exiting before preparing done (#51371) #57081

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Nov 3, 2024

This is an automated cherry-pick of pingcap#51371

830eb74

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Nov 3, 2024

operator: print callstack when exiting before preparing done (#51371) #57082

Open

13 tasks

ti-chi-bot bot removed the needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. label Nov 7, 2024

ti-chi-bot bot pushed a commit that referenced this pull request Dec 2, 2024

operator: print callstack when exiting before preparing done (#51371) (…

8603039

…#57081) close #51370

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: print callstack when exiting before preparing done #51371

operator: print callstack when exiting before preparing done #51371

YuJuncen commented Feb 27, 2024 •

edited

Loading

tiprow bot commented Feb 27, 2024

codecov bot commented Feb 27, 2024 •

edited

Loading

BornChanger Feb 28, 2024

YuJuncen Feb 29, 2024

BornChanger commented Feb 28, 2024

WangLe1321 Feb 28, 2024

WangLe1321 Feb 28, 2024

YuJuncen Feb 29, 2024

ti-chi-bot bot commented Mar 6, 2024

3pointer left a comment

3pointer Mar 6, 2024

ti-chi-bot bot commented Mar 6, 2024

ti-chi-bot bot commented Mar 6, 2024

tiprow bot commented Mar 6, 2024

BornChanger commented Mar 6, 2024

tiprow bot commented Mar 6, 2024

YuJuncen commented Mar 6, 2024

YuJuncen commented Mar 6, 2024

ti-chi-bot commented Mar 6, 2024

ti-chi-bot commented Nov 3, 2024

ti-chi-bot commented Nov 3, 2024

operator: print callstack when exiting before preparing done #51371

operator: print callstack when exiting before preparing done #51371

Conversation

YuJuncen commented Feb 27, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Feb 27, 2024

codecov bot commented Feb 27, 2024 • edited Loading

Codecov Report

BornChanger Feb 28, 2024

Choose a reason for hiding this comment

YuJuncen Feb 29, 2024

Choose a reason for hiding this comment

BornChanger commented Feb 28, 2024

WangLe1321 Feb 28, 2024

Choose a reason for hiding this comment

WangLe1321 Feb 28, 2024

Choose a reason for hiding this comment

YuJuncen Feb 29, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Mar 6, 2024

3pointer left a comment

Choose a reason for hiding this comment

3pointer Mar 6, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Mar 6, 2024

ti-chi-bot bot commented Mar 6, 2024

[LGTM Timeline notifier]

tiprow bot commented Mar 6, 2024

BornChanger commented Mar 6, 2024

tiprow bot commented Mar 6, 2024

YuJuncen commented Mar 6, 2024

YuJuncen commented Mar 6, 2024

ti-chi-bot commented Mar 6, 2024

ti-chi-bot commented Nov 3, 2024

ti-chi-bot commented Nov 3, 2024

YuJuncen commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading