Restart crashed plugins

[josefkarasek]

Description

Changes proposed in this pull request:

• Use built-in hashicorp Ping() API call to monitor health of source plugins
• Release resources of crashed plugins
• Restart crashed plugins
• Define configurable strategies to approach restart policies

Testing

Add source cm-watcher and executor echo to your comm platform.

Add to your config:

plugins:
    agentRestartPolicy:
      # -- Restart policy type. Allowed values: "restartAgent", "deactivatePlugin".
      type: "deactivatePlugin"
      # -- Number of restarts before policy takes into effect.
      threshold: 5

Recompile plugins and start botkube locally

make build-plugins-single gen-plugins-index
go run cmd/botkube-agent/main.go

Watch botkube pod logs.

Executor testing

@Botkube echo @panic will cause the echo plugin to panic and exit. Wait a few seconds and it will be restarted.
Check again with @Botkube echo hello.

Source testing

Create cm with annotation die: "true".

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    die: "true"
  name: watcher
  namespace: botkube
EOF

When this cm exists, cm-watcher plugin will continue to crash. Remove the cm. The plugin should restart.
Create the cm without the annotation - plugin should send message to specified channel.

Related issue(s)

#878

[mszostok]

Very impressive! 🚀 I like the implementation, and I left only minor comments.

I see such todos:

• add option to print plugin status or add it to list executors/sources as a new column
• add e2e tests cases
• update documentation

let me know if we should take over those items 👍

P.S. in the PR desc you have agentRestartPolicy but it should be restartPolicy and also for the current impl types should start with upper case.

[mszostok]

why commented? in general, it makes sense to have it 🤔

[mszostok]

should we do sth about this TODO? or it's more a note?

[mszostok]

in general we shouldn't panic as it will not run the proper clean-up logic, but this would require a full refactor of the main func, so it's sth to address later 😞

[mszostok]

should it be restarted and start from fresh once the plugin after e.g. 2 restarts become healthy?

Because in the current approach I can easily deactivate a plugin that is just flaky 🤔 because it is for the whole plugin history.

[mszostok]

maybe we can normalize it, to all small letters? so even if I type restartAgent instead of RestartAgent it will work.

[mszostok]

Suggested change

	m.log.Fatalf("Plugin %q has been restarted %d times and selected agentRestartPolicy is %q. Exiting...", plugin, restarts, m.policy.Type)
	m.log.Fatalf("Plugin %q has been restarted %d times and selected restartPolicy is %q. Exiting...", plugin, restarts, m.policy.Type)

[mszostok]

would be nice to print the "status" with the number of retries and max retries like (attempt no 2 of max 10)

[mszostok]

Code merged in #1236