Resurface CassandraTask error messages in the Failed condition

[adejanovski]

What is missing?

When a CassandraTask fails, we don't have any information in the CassandraTask status to make it easier to understand the core issue. This then requires to go through the cass-operator logs in search of the error message.
We'd need to update the Failed condition Message field with the error.

Why is this needed?

This would make investigation a lot easier without requiring to go through logs.

[burmanm]

That information should be there, if there is a message to be set.

https://github.com/k8ssandra/cass-operator/blob/master/internal/controllers/control/cassandratask_controller.go#L375

[adejanovski]

I'm seeing cases where the task fails with an NPE reported in the cass-operator logs but no message in the Failed condition.
Let's investigate a bit more, I'll come up with a way to reproduce this.

[adejanovski]

Here's a sample status from a failed CassandraTask:

status:
  completionTime: '2024-01-23T07:59:44Z'
  conditions:
    - lastTransitionTime: '2024-01-23T07:59:44Z'
      message: ''
      reason: Running
      status: 'False'
      type: Running
    - lastTransitionTime: '2024-01-23T07:59:44Z'
      message: ''
      reason: Complete
      status: 'True'
      type: Complete
    - lastTransitionTime: '2024-01-23T07:59:44Z'
      message: ''
      reason: Failed
      status: 'True'
      type: Failed
  failed: 1
  startTime: '2024-01-23T07:59:43Z'

And here's the error in the cass-operator logs:

2024-01-23T09:32:42.286Z	ERROR	Job failed to successfully complete the task	... {"error": "task failed: java.lang.IllegalArgumentException: Keyspace ALL does not exist"}
github.com/k8ssandra/cass-operator/internal/controllers/control.(*CassandraTaskReconciler).reconcileEveryPodTask
	/workspace/internal/controllers/control/cassandratask_controller.go:660
github.com/k8ssandra/cass-operator/internal/controllers/control.(*CassandraTaskReconciler).Reconcile
	/workspace/internal/controllers/control/cassandratask_controller.go:339
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

[burmanm]

Only way to return a message in this case is to stop the task execution also when a non-retry event happens. What happens here is that we might get that error on the first pod, but currently we then move on to the next pod to try the same task again - so writing that error message from the first pod didn't make sense.