Display formatted logs for Toolkit Components

[w0wka91]

It would be nice if the CLI would be able to display formatted logs for all of the Toolkit Components.
We could grab the logs from the specific pod, pass the logs into a json processor and show it to the user.

The API could be something like:

flux log (helm|source|kustomize|notification) --watch

[stefanprodan]

I think such a command would be a good UX improvement if we allow users to filter the logs by level, kind, name and namespace.

Print kustomize-controller reconciliation errors:

flux logs kustomize-controller --level=error --watch=true --tail=100

Print reconciliations errors of a Kustomization named podinfo from the default namespace:

flux logs kustomize-controller --level=error --kind=Kustomization --name=podinfo --namespace=default

We need to decide how we want to display the logs in the console output.
Here is the json log of a reconciliation error:

{"level":"error","ts":"2021-01-20T08:27:45.548Z","logger":"controller.kustomization","msg":"Reconciler error","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"default","error":"apply failed: The Deployment \"podinfo\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"podinfo\", \"app.kubernetes.io/name\":\"podinfo\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n"}

[w0wka91]

It could be also useful to filter logs by level without specifying the component

Print all recently occured errors:
flux logs --level=error --watch=true --tail=100

We could provide a default output-format and let the user overwrite it via an environment variable?!
Default could be something like this:

%TIMESTAMP %LEVEL %KIND - %MESSAGE

The output would look like this:
2021-01-20 08:27:45 ERROR Kustomization - apply failed: The Deployment \"podinfo\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"podinfo\", \"app.kubernetes.io/name\":\"podinfo\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Another approach would be to simply pass the current json log to a json processor like: jq

[stefanprodan]

Showing the error field is not enough, we need msg as well.

Another approach would be to simply pass the current json log to a json processor like: jq

Then I see no reason to implement flux logs, a combination of kubectl logs | jq <filters> is what I currently use. If we go for a command like flux logs we should filter and parse the json is a readable format so people don't have to pipe it into jq.

[w0wka91]

Showing the error field is not enough, we need msg as well.

Output for log level ERROR:

2021-01-20 08:27:45 ERROR Kustomization - Reconciler error
  apply failed: The Deployment \"podinfo\" is invalid: spec.selector: Invalid value: 
    v1.LabelSelector{MatchLabels:map[string]string{\"app\":\"podinfo\", \"app.kubernetes.io/name\":\"podinfo\"}, 
    MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Output for log level INFO:

2021-01-20 08:27:45 INFO Kustomization - Kustomization applied in 345.21018ms. 
  helmrepository.source.toolkit.fluxcd.io/podinfo: unchanged
  helmrepository.source.toolkit.fluxcd.io/bitnami: unchanged

[stefanprodan]

I propose this format:

<TS> <LEVEL> <kind/name.namespace> - <MSG> <ERROR>

[w0wka91]

Sounds good to me