Gangplank: ensure that the local port is available for SSH #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This time we'll go ahead and use the remote chosen port locally as well. This is because the minio bucket is consulted by the local running process for required artifacts before even starting a pod on the remote (i.e. why start the pod until the artifacts exist that it needs?). If they use different port numbers then we'd have to track them separately and behave differently whether we're running in the remote podman cosa or in the local cosa. Let's just use the same port locally as well since it is highly unlikely to have a port conflict locally because we're running in a container with no other services running. To achieve this goal we submerge the starting of the minio server into the ssh port forwarding code so we can detect the port before we start the server and set up the proxy.
This builds on PR coreos#2378 which maps the remote port to the local port. However, if the local port is in use, then Gangplank will fail to launch. Signed-off-by: Ben Howard <[email protected]>
dustymabe
force-pushed
the
dusty-gangplank-ports
branch
2 times, most recently
from
4086e49 to
ff56518
Compare
dustymabe
added a commit
that referenced
this pull request
May 3, 2023
Recently we saw a test with many soft lockup messages like: ``` [ 4159.779792] watchdog: BUG: soft lockup - CPU#0 stuck for 883s! [kworker/u2:0:10] [ 4159.780488] Modules linked in: [ 4159.780787] CPU: 0 PID: 10 Comm: kworker/u2:0 Tainted: G L ------- --- 6.4.0-0.rc0.20230502git865fdb08197e.11.fc39.x86_64 #1 [ 4159.780787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc37 04/01/2014 [ 4159.780787] Workqueue: ftrace_check_wq ftrace_check_work_func [ 4159.780787] RIP: 0010:get_symbol_pos+0x5d/0x140 [ 4159.780787] Code: 63 0c 8d 08 d0 6e a0 85 c9 79 0a 48 f7 d1 48 03 0d 50 c2 5a 01 48 39 cf 48 0f 42 f0 48 0f 43 d0 48 89 f0 48 29 d0 48 83 f8 01 <77> ca 48 85 d2 75 08 eb 4b 48 83 ea 01 74 45 8d 42 ff 48 98 48 63 [ 4159.780787] RSP: 0018:ffffb1970005bb80 EFLAGS: 00000202 [ 4159.780787] RAX: 0000000000000032 RBX: ffffffff9f22edb4 RCX: ffffffff9f22ed00 [ 4159.780787] RDX: 0000000000004436 RSI: 0000000000004468 RDI: ffffffff9f22edb4 [ 4159.780787] RBP: ffffb1970005bbce R08: 0000000000000000 R09: ffffb1970005bbc0 [ 4159.780787] R10: 0000000000000000 R11: 00000000000320a7 R12: 0000000000000000 [ 4159.780787] R13: 0000000000000000 R14: ffffb1970005bbc0 R15: 0000000000000000 [ 4159.780787] FS: 0000000000000000(0000) GS:ffff9fa07ec00000(0000) knlGS:0000000000000000 [ 4159.780787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4159.780787] CR2: ffff9fa04fe01000 CR3: 000000000e022000 CR4: 0000000000350ef0 [ 4159.780787] Call Trace: [ 4159.780787] <TASK> [ 4159.780787] kallsyms_lookup_buildid+0x4d/0x130 [ 4159.780787] test_for_valid_rec+0x64/0xb0 [ 4159.780787] ftrace_check_work_func+0x3b/0x60 [ 4159.780787] process_one_work+0x1c7/0x3d0 [ 4159.780787] worker_thread+0x51/0x390 [ 4159.780787] ? __pfx_worker_thread+0x10/0x10 [ 4159.780787] kthread+0xf7/0x130 [ 4159.780787] ? __pfx_kthread+0x10/0x10 [ 4159.780787] ret_from_fork+0x2c/0x50 [ 4159.780787] </TASK> ``` Let's try to detect and report this.
dustymabe
added a commit
that referenced
this pull request
May 4, 2023
Recently we saw a test with many soft lockup messages like: ``` [ 4159.779792] watchdog: BUG: soft lockup - CPU#0 stuck for 883s! [kworker/u2:0:10] [ 4159.780488] Modules linked in: [ 4159.780787] CPU: 0 PID: 10 Comm: kworker/u2:0 Tainted: G L ------- --- 6.4.0-0.rc0.20230502git865fdb08197e.11.fc39.x86_64 #1 [ 4159.780787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc37 04/01/2014 [ 4159.780787] Workqueue: ftrace_check_wq ftrace_check_work_func [ 4159.780787] RIP: 0010:get_symbol_pos+0x5d/0x140 [ 4159.780787] Code: 63 0c 8d 08 d0 6e a0 85 c9 79 0a 48 f7 d1 48 03 0d 50 c2 5a 01 48 39 cf 48 0f 42 f0 48 0f 43 d0 48 89 f0 48 29 d0 48 83 f8 01 <77> ca 48 85 d2 75 08 eb 4b 48 83 ea 01 74 45 8d 42 ff 48 98 48 63 [ 4159.780787] RSP: 0018:ffffb1970005bb80 EFLAGS: 00000202 [ 4159.780787] RAX: 0000000000000032 RBX: ffffffff9f22edb4 RCX: ffffffff9f22ed00 [ 4159.780787] RDX: 0000000000004436 RSI: 0000000000004468 RDI: ffffffff9f22edb4 [ 4159.780787] RBP: ffffb1970005bbce R08: 0000000000000000 R09: ffffb1970005bbc0 [ 4159.780787] R10: 0000000000000000 R11: 00000000000320a7 R12: 0000000000000000 [ 4159.780787] R13: 0000000000000000 R14: ffffb1970005bbc0 R15: 0000000000000000 [ 4159.780787] FS: 0000000000000000(0000) GS:ffff9fa07ec00000(0000) knlGS:0000000000000000 [ 4159.780787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4159.780787] CR2: ffff9fa04fe01000 CR3: 000000000e022000 CR4: 0000000000350ef0 [ 4159.780787] Call Trace: [ 4159.780787] <TASK> [ 4159.780787] kallsyms_lookup_buildid+0x4d/0x130 [ 4159.780787] test_for_valid_rec+0x64/0xb0 [ 4159.780787] ftrace_check_work_func+0x3b/0x60 [ 4159.780787] process_one_work+0x1c7/0x3d0 [ 4159.780787] worker_thread+0x51/0x390 [ 4159.780787] ? __pfx_worker_thread+0x10/0x10 [ 4159.780787] kthread+0xf7/0x130 [ 4159.780787] ? __pfx_kthread+0x10/0x10 [ 4159.780787] ret_from_fork+0x2c/0x50 [ 4159.780787] </TASK> ``` Let's try to detect and report this.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on PR coreos#2378 which maps the remote port to the local port.
However, if the local port is in use, then Gangplank will fail to
launch.