[Story] Print container logs with Scorecard test output

[ebaron]

It would be useful to see the logs for each container running in the Scorecard tests, whether the tests pass or fail. It will be noisy, but I believe the only test output for Scorecard tests is the one log. Perhaps we can format these nicely with section headers for each container, with some whitespace before and after.

There's a Kubernetes API you can use to get the logs for each container:
https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#get-read-log-of-the-specified-pod

[tthvo]

It will be noisy, but I believe the only test output for Scorecard tests is the one log.

I wonder how about the scorecard can create a ConfigMap (with predefined labels) and put the logs with entries for each container in the cryostat pod.

At the end of tests, we can check if any "log" configmap exists, then dump it to local files for inspection. What do you think?

EDIT: Might hit configmap size limit of 1MB tho. Probably not :D

[ebaron]

I was just thinking of reading the logs into memory while running the tests. There is a follow parameter that works like kubectl logs -f, and we could create a goroutine for each container to read them asynchronously. What do you think of that approach?

[tthvo]

I was just thinking of reading the logs into memory while running the tests. There is a follow parameter that works like kubectl logs -f, and we could create a goroutine for each container to read them asynchronously. What do you think of that approach?

Right. That sounds good to me! I guess I was a bit concerned about large quantity of logs being stored in memory, especially in a more resource-limit cluster. Maybe, it's not too large to be concerned about.

Otherwise, I am thinking that goroutines can write to files on the pod's ephemeral storage and operator-sdk scorecard can default to run with:

-x, --skip-cleanup             Disable resource cleanup after tests are run 

Then, before running scorecard tests, we can run clean up first with a prompt to confirm if any resources exist (or force clean if certain env var is set). Logs can then access by exec into the scorecard pod.

[tthvo]

Maybe I am overthinking. I can help with this and see it how it goes^^