-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd: Add enter command #188
Conversation
@niemeyer @benhoyt Note that I've included my https://github.com/woky/go-flags/ fork which supports setting |
635cedb
to
42bae10
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't reviewed this in detail (let me know if it's ready for that and if you want that), but just left a couple of preliminary comments.
cmd/pebble/cmd_enter.go
Outdated
var runPanic interface{} | ||
|
||
go func() { | ||
defer func() { runPanicCh <- recover() }() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why don't we just let a panic crash the process?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is because in cmd/
we actually use panic({exitStatus(1)})
as kind of long jump. When exitStatus
error is recovered in main
, the main just exits with the given status code. The run command uses it for restart functionality and also to bail when something goes wrong. If I let the goroutine panic, the exitStatus
error wouldn't be recovered in main. And, since exitStatus
has Error
method, if run command wanted to exit with status 1, then the following would be printed
panic: internal error: exitStatus{1} being handled as normal error
goroutine 6 [running]:
main.(*cmdRun).run(0x0?, 0x0?)
/home/woky/devel/pebble/cmd/pebble/cmd_run.go:82 +0xa9
main.(*cmdEnter).Execute.func2()
/home/woky/devel/pebble/cmd/pebble/cmd_enter.go:140 +0x25
created by main.(*cmdEnter).Execute
/home/woky/devel/pebble/cmd/pebble/cmd_enter.go:138 +0x46a
exit status 2
instead of just
exit status 1
1d1fff6
to
ba90c45
Compare
@benhoyt I've addressed your comments. It won't be ready to merge unless either the PR to go-flags is merged or we decide to fork it (and how), but I think it's ready to review. |
903f427
to
8ce91b0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's surprisingly tidy, thanks Tomas.
Just a few minors and some high-level questions:
cmd/pebble/cmd_enter.go
Outdated
without starting "autostart" services (i.e. run --hold, unless --run flag is | ||
given), and the subcommand is run as a pebble client command that can | ||
communicate with the service manager instance in the background. The service | ||
manager is shut down after the commands finishes unless noted otherwise below. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is doing a reasonable job of describing the behavior, but as Ben already spotted it is also very confusing because the actual behavior of any one command is hard to grasp because it's an ANDing of a number of different conditional paragraphs that may affect the given command or not. The agreement we've reached at the end of our several meetings was supposed to be one where the behavior of enter would be the most natural option for the given command at hand, so if we did that justice, we shoud be able to have a more straightforward description here too.
With that said, let's leave that aside for now and focus on the implementation. Once this is ready to go in, we can do a pass together and find the best way to word this out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree it's not exactly straightforward read.
cmd/pebble/cmd_enter.go
Outdated
} | ||
|
||
if enterFlags&enterFlagRequireServiceAutostart != 0 && !cmd.Run { | ||
return fmt.Errorf("enter: this command can only run with autostarted services (hint: add --run to enter)") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps more directly:
enter: must use --run before %q subcommand
With that said, the memory I have of our final agreement (which arguably took a bit of back and forth, so I may easily be missing something) was that we'd make subcommands behave in their most natural way, and when there was no natural way at all we'd prevent the subcommand from running.
In particular here, the fact we'd have start and stop disagreeing on the requirement of --run, and the fact we're forcing --run to be provided to a subcommand that could not work without it, are both bad symptoms that this isn't quite what we need. Let's talk about it elsewhere to revisit this topic please.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wholeheartedly agree that the sub-command behavior as currently specified is confusing and hard to convey.
EDIT: Changed the messages according to your example.
err := commander.Execute(extraArgs) | ||
|
||
if err != nil || enterFlags&enterFlagKeepServiceManager == 0 { | ||
runStop() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't reached the end of the code yet, so maybe this will be answered later, but how are we keeping the run command going if we're returning from enter
early here? In other words, the fact we're not asking the run command to stop explicitly won't do much for us if this command is about to return and exit from Pebble itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is handled by runResultCh
(previously called runPanicCh
) that's written to in the goroutine closure above after the run
command ends, and that's being read from right after this statement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. Can we add some simple tests for a few of the pebble enter
use cases? Might be more trouble than it's worth, but worth some thought IMO.
cmd/pebble/cmd_enter.go
Outdated
Furthermore, some commands have restrictions and idiosyncrasies: | ||
|
||
exec, ls, plan, services, version | ||
Service manager logging is disabled unless -v/--verbose flag is provided. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The wording still needs a few style/grammar tweaks I think. But for now, just note that these indentations are removed when displaying the help (i.e., running pebble enter -h
).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noted, thanks. This is done by wrapText
function in go-flags. Let's keep it indented and fix go-flags later.
9367d14
to
299c308
Compare
@niemeyer @benhoyt I've addressed most of your comments (and replied to the rest). Please take a look again. Note the CI failures. This because of data race detector detecting race condition when setting logger with Data race detector output
I've never debugged data races in Golang so I could use some help. However, my theory is that this is a non-issue and it only surfaced because this is the first test where pebble is started as a service manager (overlord) and so I could disable the the data race detector in the new test file with |
8a06b17
to
2b49c9d
Compare
I've fixed the data race detector warning by synchronizing access to the global The currently failing test in
EDIT: It didn't happen again (after I force pushed with updated commit date). |
907ceb7
to
2300ba6
Compare
Looks good to me (apart from perhaps the help wording which it sounds like you and Gustavo will work on). And thanks for the logging fix! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the proposed help text, and we forgot to talk about stop
vs. --run
yesterday.
Otherwise, this looks good to go.
) | ||
|
||
const shortEnterHelp = "Run pebble command in new pebble environment" | ||
const longEnterHelp = ` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, here is the proposal
Short help: Run subcommand under a container environment
Long help:
The enter command facilitates the use of Pebble as an entrypoint for containers.
When used without a subcommand it mimics the behavior of the run command
alone, while if used with a subcommand it runs that subcommand in the most
appropriate environment taking in account its purpose.
These subcommands are currently supported:
help (1)(2)
version (1)(2)
plan (1)(2)
services (1)(2)
ls (1)(2)
exec (2)
start (3)
stop (3)
(1) Services are not started.
(2) No logs on stdout unless -v is used.
(3) Services continue running after the subcommand succeeds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good.
nitpick:
< taking in account
> taking into account
cmd/pebble/cmd_enter.go
Outdated
Service manager continues running after the command finishes. | ||
|
||
stop | ||
Service autostart (the --run flag) is required. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We forgot to address this wart in our call yesterday. We need to figure why start/stop were proposed to be different, as it's very awkward to require a flag at all times for stop and not for start.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recalling the spec (RK007) discussions, the logic was that pebble enter stop [svc ...]
would have an equivalent behaviour to pebble run --hold; pebble stop [svc ...]
, which was deemed to be undesired and thus marked as an "error" combination.
Service autostart (the --run flag) is required.
Putting this in other words, --run
is required as a consequence of pebble enter
acting as pebble run --hold
. Without it, enter stop
doesn't do anything cause there are no "started" services.
03f1289
to
3253d55
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright, thanks for all the reviews and help here.
This is good to go, pending only the go-flags support.
Update: Sorry, forgot to mention the one trivial below:
Given the lack of response from upstream, let's temporarily fork go-flags at https://github.com/canonical/go-flags, and submit/review the required extensions there please. |
3253d55
to
6b1d386
Compare
Thanks. Opened PR canonical/go-flags#1 |
470e6ff
to
d331c3c
Compare
d331c3c
to
0eb575e
Compare
0eb575e
to
d870575
Compare
Updated go.{mod|sum} for canonical/go-flags fork. Tests pass! Should be good to go. :o) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, Tomas. One trivial only before we merge:
8b953d1
to
82ce86c
Compare
b95b567
to
dba41c0
Compare
One more trivial, sorry about that: formatting checks are failing. |
Also, run go mod tidy.
This obviates the need to pass double dash (--) when exec command includes command line switches.
In main.go we define Stdin, Stdout and Stderr reader/writers. This is to aid with testing. In cmd_exec.go we use os.Std* files as reader/writers. This makes it impossible to test output from exec commands. Change cmd_exec.go to use reader/writers defined in main.go.
Currently run() from main.go is exported to tests as RunMain, but we cannot test pebble exits with specific error code with run(). Make main() exit via osExit function variable and export RealMain that catches and returns the exit code.
From the long help message: Usage: pebble enter [enter-OPTIONS] [<subcommand>...] The enter command facilitates the use of Pebble as an entrypoint for containers. When used without a subcommand it mimics the behavior of the run command alone, while if used with a subcommand it runs that subcommand in the most appropriate environment taking into account its purpose. These subcommands are currently supported: help (1)(2) version (1)(2) plan (1)(2) services (1)(2) ls (1)(2) exec (2) start (3) stop (3) (1) Services are not started. (2) No logs on stdout unless -v is used. (3) Services continue running after the subcommand succeeds. [enter command options] --create-dirs Create pebble directory on startup if it doesn't exist --hold Do not start default services automatically --http= Start HTTP API listening on this address (e.g., ":4000") -v, --verbose Log all output from services to stdout --run Start default services before executing subcommand
In practice logger.SetDefault() is only called once in main() and never again. This sets up global logger variable which is then read by logger print functions (Panicf, Noticef, Debugf) in other parts of pebble. However, test runs are not fully isolated and multiple tests can call logger.SetDefaul() in the same process. Until now, this wasn't a problem, because no test called logger.SetDefault() and logger print functions in different goroutines. But when we start pebble as a service manager inside a test, like in cmd/pebble/cmd_enter_test.go, several goroutines are created, some of which call logger print functions. In this case it's likely that logger.SetDefault() gets called in another test after it was read from different goroutine in a logger print function call. When that happens, the golang data race detetor, which is active during test runs in CI, reports the problem included at the bottom. Since logger.SetLogger() is public it can in theory be called by other code. Synchronize access to it by newly introduced mutex. Also drop mutex from defaultLogger structure as it's private data type and access to it inside logger packages is already protected by the newly introduced mutex. [1] https://go.dev/doc/articles/race_detector Golang data race detector report: ================== WARNING: DATA RACE Write at 0x00000117add0 by goroutine 336: github.com/canonical/pebble/internal/logger.SetLogger() /home/woky/devel/devel/pebble/internal/logger/logger.go:83 +0x15e github.com/canonical/pebble/cmd/pebble.run() /home/woky/devel/devel/pebble/cmd/pebble/main.go:343 +0x36 github.com/canonical/pebble/cmd/pebble.main() /home/woky/devel/devel/pebble/cmd/pebble/main.go:324 +0x58 github.com/canonical/pebble/cmd/pebble.RealMain() /home/woky/devel/devel/pebble/cmd/pebble/export_test.go:77 +0xdc github.com/canonical/pebble/cmd/pebble_test.(*PebbleSuite).TestEnterHelpCommand() /home/woky/devel/devel/pebble/cmd/pebble/cmd_enter_test.go:79 +0x22c runtime.call16() /usr/lib/go/src/runtime/asm_amd64.s:728 +0x48 reflect.Value.Call() /usr/lib/go/src/reflect/value.go:370 +0xc7 gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:775 +0xadd gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:669 +0xec Previous read at 0x00000117add0 by goroutine 319: github.com/canonical/pebble/internal/logger.Debugf() /home/woky/devel/devel/pebble/internal/logger/logger.go:66 +0x6e github.com/canonical/pebble/internal/overlord/cmdstate.(*execution).controlLoop() /home/woky/devel/devel/pebble/internal/overlord/cmdstate/handlers.go:464 +0x10e4 github.com/canonical/pebble/internal/overlord/cmdstate.(*execution).do.func9() /home/woky/devel/devel/pebble/internal/overlord/cmdstate/handlers.go:272 +0x84 Goroutine 336 (running) created at: gopkg.in/check%2ev1.(*suiteRunner).forkCall() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:666 +0x5c4 gopkg.in/check%2ev1.(*suiteRunner).forkTest() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:757 +0x16b gopkg.in/check%2ev1.(*suiteRunner).runTest() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:812 +0x439 gopkg.in/check%2ev1.(*suiteRunner).run() /home/woky/go/pkg/mod/gopkg.in/[email protected]/check.go:618 +0x3e6 gopkg.in/check%2ev1.Run() /home/woky/go/pkg/mod/gopkg.in/[email protected]/run.go:92 +0x49 gopkg.in/check%2ev1.RunAll() /home/woky/go/pkg/mod/gopkg.in/[email protected]/run.go:84 +0x12b gopkg.in/check%2ev1.TestingT() /home/woky/go/pkg/mod/gopkg.in/[email protected]/run.go:72 +0x565 github.com/canonical/pebble/cmd/pebble_test.Test() /home/woky/devel/devel/pebble/cmd/pebble/main_test.go:24 +0x2e testing.tRunner() /usr/lib/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /usr/lib/go/src/testing/testing.go:1629 +0x47 Goroutine 319 (finished) created at: github.com/canonical/pebble/internal/overlord/cmdstate.(*execution).do() /home/woky/devel/devel/pebble/internal/overlord/cmdstate/handlers.go:272 +0xfc5 github.com/canonical/pebble/internal/overlord/cmdstate.(*CommandManager).doExec() /home/woky/devel/devel/pebble/internal/overlord/cmdstate/handlers.go:118 +0xb09 github.com/canonical/pebble/internal/overlord/cmdstate.(*CommandManager).doExec-fm() <autogenerated>:1 +0x4d github.com/canonical/pebble/internal/overlord/state.(*TaskRunner).run.func1() /home/woky/devel/devel/pebble/internal/overlord/state/taskrunner.go:195 +0xb3 gopkg.in/tomb%2ev2.(*Tomb).run() /home/woky/go/pkg/mod/gopkg.in/[email protected]/tomb.go:163 +0x51 gopkg.in/tomb%2ev2.(*Tomb).Go.func2() /home/woky/go/pkg/mod/gopkg.in/[email protected]/tomb.go:159 +0x47 ==================
This commit fixes handling of the following two cases: 1. pebble enter subcommand --help 2. pebble enter help subcommand Help messages are handled in main.go when the executed command returns flags.Error{Type:flags.ErrHelp}. When we pass the subcommand's return value back to the main parser, the error handling there will use the currently active command to print the help message. The active command is enter but we want to print the subcommands help message. Whenever the subcommand returns an error value, set it as active command in the main parser, so that the error handling there works with the failing subcommand.
dba41c0
to
a346470
Compare
Whoops. Fixed. (And squashed into first commit.) |
The enter command behaves just like run when run without positional
arguments, with the exception that the --hold flag is not supported.
That is, it runs as service manager. When run with positional arguments,
the service manager is started in background without starting autostart
services (i.e. like run with the --hold flag) and positional arguments
are executed as pebble command. Only subset of pebble commands is
supported, namely exec, help, ls, plan, services, start, stop and
version. Autostart services can be started with the --run flag. start
command must be run with the --run flag and ls, plan, services and stop
commands cannot be run with the --run flag. Normally, the service
manager is shut down when the command finishes, except for start and
stop commands where the service manager continues running. The enter
command is intended to be used as entrypoint in container images.