Implement systemd notification

[astorath]

Is your feature request related to a problem? Please describe.
Using vault binary as a systemd unit have some control problem:

• vault does not have daemon mode
• vault does not support systemd notification system

So when service starts unsuccessfully (due to config error for instance), systemctl can't report the problem right away and systemd keeps restarting failing unit forever.

Describe the solution you'd like
I think the best solution is to implement systemd notification when vault (or vault agent) is ready to accept requests.

Example from consul: agent.go

Describe alternatives you've considered
I've implemented a simple bash wrapper to check for active port after vault startup. This is a dirty solution:

• it makes startup take longer
• checking port availability is not the same as checking service is ready
• requires some unnecessary intermediate script

Explain any additional use-cases
You can implement WATCHDOG notification to make systemd monitor unit health.

Additional context
I see that github.com/coreos/go-systemd module is in

vault/vendor/modules.txt

Line 185 in ceb4aea

github.com/coreos/go-systemd/journal

and

vault/go.sum

Line 104 in 0a58f30

github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7 h1:u9SHYsPQNyt5tgDm3YN7+9dYrpK96E5wFilTFWIDZOM=

, maybe there were some attempts to implement it already?

[aphorise]

Should be implemented in above PR (make a note for my own reference).

[sgmiller]

Implemented using a cleaner dep, and also notifying on shutdown/config reload: #11517

[aphorise]

@astorath can you confirm if the merged changes already accommodate for what you were after within the latest 1.7.x or higher releases?

[qk4l]

@sgmiller Thank you for implementation.
I've tried this functional (1.8.4) and found the problem with reload.
Currently there is not SdNotifyReady call after reload, so systemd wait, timeout and then call restart.
https://github.com/coreos/go-systemd/blob/main/daemon/sdnotify.go#L39

Some logs...

Oct 24 20:42:18 example.com systemd[1]: Reloading vault.service.
Oct 24 20:42:18 example.com vault[32106]: ==> Vault reload triggered
Oct 24 20:42:18 example.com vault[32106]: 2021-10-24T20:42:18.709+0300 [DEBUG] sent systemd notification: notification=RELOADING=1
...
Oct 24 20:45:19 example.com systemd[1]: vault.service reload operation timed out. Stopping.
Oct 24 20:47:13 example.com systemd[1]: Stopped vault.service.
Oct 24 20:47:13 example.com systemd[1]: Starting vault.service...
Oct 24 20:47:14 example.com systemd[1]: Started vault.service.

[ncabatoff]

This was fixed in #11517 and should've been closed then. Closing.