workload: tolerate ambiguous commit errors with --tolerate-ambiguous-errors

[knz]

Fixes #49865

Release note (cli change): cockroach workload now recognizes the
command-line flag --tolerate-ambiguous-errors to ignore ambiguous
commit errors. This is useful when testing workload during "chaos
testing", when nodes are stopped/restarted ungracefully.

[cockroach-teamcity]

This change is 

[knz]

@nvanbenschoten I'm not exactly sure how to test this and verify that it works. Do you have a suggestion?

[irfansharif]

The code changes here LGTM but I'll leave it to you and Nathan to duke it out over #49865. It seems we added this flag to paper over chaos conditions, but we're now seeing ambiguous errors even without chaos? That makes make question the utility of this.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @nvanbenschoten)

[nvanbenschoten]

Cross-posting two things from #49865:

Re chaos tests: Like @knz, I was also surprised to see that we were already passing --tolerate-errors in the chaos tests that were hitting ambiguous errors and that we'd now want to pass this new flag to. That indicates that this flag won't help the problem and that errors are making their way back to the test through some other mechanism.

Re non-chaos tests: The fact that we're seeing ambiguous errors under normal circumstances is a problem. I don't think we should paper over it by applying this blindly. Instead, we should try to eliminate such sources of ambiguity (e.g. #52566).