Safely providing logging output from failed hub health checks

[sgibson91]

Summary

At the moment, our hub health checks in deployer only output pass or fail, without any indication to what caused the failure. This is because when pytest fails like this, it exposes any secrets on the command line. At the minute to diagnose the failure, we have to get the logs directly from the hub pod. Which is fine, but if you don't already have access to the clusters setup, it can be disruptive to flow.

We should think about safe ways for the deployer to provide feedback when a hub health check fails.

[yuvipanda]

This would be great help. The problem is that pytest fails with showing all params, which include secrets. I think we can safely move away from pytest maybe - I don't think we use any of its features at all

[sgibson91]

The problem is that pytest fails with showing all params, which include secrets.

Ah yes, thank you. Updating top comment.

[sgibson91]

Can you remind me what the undesired output looks like? Is it just the options defined in conftest.py are printed out at the top of the output and the api_token in-particular is sensitive? That is we get something like the below before the pytest feedback:

hub_url = https://example.com
hub_type = basehub
api_token = SUPER_SECRET_VALUE

[sgibson91]

Or better yet, if there's a way I can locally force a failed test without actually deploying anything - that would be grand 😁

[sgibson91]

If my above assumption is correct, I think "all" we have to do is add --tb=short to the pytest invocation and the input parameters are suppressed and the expense of slightly shortening the rest of pytest's output

https://github.com/2i2c-org/pilot-hubs/blob/b4244164aa7a526ef5d6df1e7fb0d167ace567b8/deployer/hub.py#L419-L425

[sgibson91]

See minimal working example here https://gist.github.com/sgibson91/5b31ee1b18832ae785565da99fe605a2

[damianavila]

and the input parameters are suppressed and the expense of slightly shortening the rest of pytest's output

I think it is a good enough compromise...