Failed to list old offloaded nodes" err="Error 1103 (42000): Incorrect table name ''

[mikutas]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

After we upgraded argo-workflows from v3.4.5 to v3.4.6 (helm chart 0.22.14 -> 0.22.16), workflow-controller came to output following error log.

"Failed to list old offloaded nodes" err="Error 1103 (42000): Incorrect table name ''"

Version

v3.4.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

(I guess it doesnt' affect)

Logs from the workflow controller

"Failed to list old offloaded nodes" err="Error 1103 (42000): Incorrect table name ''"

Logs from in your workflow's wait container

(I guess it doesnt' affect)

[mikutas]

probably related? #10734

[acobaugh]

FYI, we hit this one pretty hard (many of our workflows have to be offloaded due to size), and we're still hitting this even after switching to latest-linux-amd64@sha256:43768569c601d8ab0ad5b406a6e60ef6df50569cc83e892d2690fabcf4c8bf70

In our case with postgres, the error is Failed to list old offloaded nodes" err="pq: zero-length delimited identifier at or near \"\"\"\"

[acobaugh]

Please disregard my previous comment, the helm chart breaks when passing tag@sha256 style tags. After manually editing the Deployment, our errors seem to have gone away with #10887 applied. Would be great to see this fixed in a patch release soon-ish, since it seems like node status offloading is really broken without it.