Improves metric and error handling for history

[davidporter-id-au]

This was originally added (and not working) with #5438 and this followup corrects it and adds some actual metrics tests to ensure such a miss doesn't happen again.

The driver of this change, to reiterate was two things:

• More concretely, there are some times of invalid data are annoyingly difficult to track down because it lacks runID information. Obviously the oncall can dig around in the DB for the workflow and guess, but it's operationally quite a lot of work in a fast-moving environment. Some problems with invalid workflows without a current runID in this state blocked a preproduction environment for quite a while.
• Zooming out, a goal @Groxx and others have had, is to make errors able to be much richer by wrapping them. However, this requires more than case-switching on types in order to convey more useful information such as the stacktrace and debug info. However, to do so requires any logic which does type or equality matching on errors to start properly using errors.Is/As. This is a small part of that initiative (albeit with a few bumps)

[taylanisikdemir]

I am not familiar with this mock library. where is the validation happening? AFAICT td.expectation is marking some method calls with once but when is it ensured those calls happened exactly once and no other calls?

[davidporter-id-au]

I don't love the mocking library either, I'd like to get rid of it in favour of our normal gomock one, but it's what's already here, so I didn't try and change it.

    --- FAIL: TestCorrectUseOfErrorHandling/uncategorized_error (0.00s)
panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("IncCounter").Return(...) first, or remove the IncCounter() call.
	This method was unexpected:
		IncCounter(int)
		0: 1
	panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("IncCounter").Return(...) first, or remove the IncCounter() call.
	This method was unexpected:
		IncCounter(int)
		0: 1

You're right there's nothing ensuring there's a counter call, let me add an assertion on that.

[davidporter-id-au]

Added scope.Mock.AssertCalled below to ensure that completely ommiting a call to counter is not missed