Remove crontab

[dbutenhof]

PBENCH-1086

Use systemd to periodically run the pbench-index service instead of requiring that we generate a crontab file and install components to use it.

We add the sd_notify service to the server shell to better synchronize with systemd so that it knows our service isn't "running" once we've been exec-d, but only when we tell it. I also added some STATUS updates, although these don't matter much unless someone is watching with systemctl status. The critical bit is that we issue READY=1 when we want systemd to consider the server "active". This triggers the new pbench-index.timer, which will begin to run every minute (more or less as with crontab). The sd_notify delay ensures that the server will have initialized PostgreSQL and Elasticsearch to avoid any conflicts. This introduces the Type = notify and NotifyAccess settings. (I chose all as the NotifyAccess because it appears that we're now seeing notifications from gunicorn as well, and all allows systemd to recogize sd_notify calls from children of the main service PID instead of writing a warning into the journal.)

We use User = pbench system services rather than user services principally because the latter would complicate the build and deployment without giving any real benefits. Because buildah builds images without an initd pid 1, the necessary loginctl enable-linger pbench and systemctl --user commands aren't available during container build. While I experimented with deferring these until we stand up the real container during deployment, that never felt "clean".

Note that I removed the pidfile and kill configuration from the service file. These are actually unnecessary unless the service is doing something unusual: instead we simply tell systemd to use SIGTERM on the primary service PID. (Though even that isn't strictly necessary as SIGTERM is the default.)

[webbnh]

Looks great! Just small stuff and nits.

[webbnh]

👍