How should we gather logs centrally from clusters?

[dnegreira]

Hi everyone,

More recently we have been having a few questions or requests to gather sosreport data from some services like kubeflow or other software running on kubernetes, these services might run on big clusters of k8s, so it would make sense to gather all the kubectl logs centrally rather than go and run sosreport on each of the k8s hosts.

I see that the openshift plugin is quite well done in this regard, and can serve as an inspiration for future plugins to gather the data, but I am afraid that we might duplicate the work that we need to do in other plugins.
What I mean with this, and again using the openshift plugin as an example, we have the options kubeconfig, host, add-namespace, etc, which can all be re-used by any plugin that needs/wants to query kubernetes in this case, so I am afraid that if we do this, per plugin, that we will have lots of unnecessary duplicated code.

My questions are:

• What should be the best way forward if we want to collect centrally the commands from kubectl, and maybe from other clustering services/commands in the future?
• Should we have a plugin per service we want to collect from kubernetes (IE, openshift, kubeflow), and how could we avoid duplication of code by using a common set of options that we could use between the different plugins we would want to retrieve centrally?

[pmoravec]

I dont know kubernetes / openshift much, but if multiple plugins would duplicate same code that operates over similar objects, it makes sense to add it to the Plugin class..? (well, that class alone is huge enough, we might think to refactor or hiearchise it, maybe, but that is for another discussion).

I mean, do something similar like was done for iterating and/or getting info from devices: https://github.com/sosreport/sos/blob/main/sos/report/__init__.py#L441 .