Latest watchers do not pick up withdrawal finalization

[eternauta1337]

This happens when we upgrade @eth-optimism/watcher to @eth-optimism/core-utils as can be seen in this PR https://github.com/Synthetixio/synthetix/pull/1321/files

The result is that the watcher "hangs" when detecting withdrawal finalization, even though it seems to pick up deposit finalization. This can be seen as a timeout of 10m in the previously mentioned CI run https://app.circleci.com/pipelines/github/Synthetixio/synthetix/7427/workflows/fcf5e4f3-527f-4fe8-8e78-4f5ce376ec9d/jobs/69921

To reproduce locally, check out the Synthetix repo at the PR above, start the ops tool at the specified commit, then run:
npx hardhat test:integration:dual --compile --deploy

This is how we setup the watcher: https://github.com/Synthetixio/synthetix/pull/1321/files#diff-0ec74ac23703db180e935294b0032c83811c218e87dacfdf89637dca21d4e352L70-L80

And how we use it: https://github.com/Synthetixio/synthetix/blob/develop/test/integration/utils/watchers.js

[eternauta1337]

@smartcontracts ah found the issue and fixed it here: #1121

Our integration tests no longer hand and all pass with this mini change directly implemented in our node_modules.

[eternauta1337]

@smartcontracts it seems like the fix proposed in #1121 allows some L2>L1 withdrawal's get picked up, but it still times out on some situations.

[K-Ho]

Confirmed that I was able to reproduce locally

1) Start up the local network
2)  npx hardhat test:integration:dual --compile --deploy on branch https://github.com/Synthetixio/synthetix/pull/1343
3) npx hardhat test:integration:dual to re-rerun the tests
4) npx hardhat test:integration:dual --deploy to re-run tests with clean state

Withdrawal succeeds on first few tries, but then npx hardhat test:integration:dual eventually stops having withdrawals relayed.

[K-Ho]

Looks like batch submitter is failing with logs

{"level":30,"time":1624315102382,"name":"oe:batch_submitter:state_chain","address":"0x70997970C51812dc3A010C7d01b50e0d17dc79C8","ether":"4999.841759976","msg":"Checked balance"}
{"level":30,"time":1624315102382,"name":"oe:batch_submitter:state_chain","l2ChainId":"0x1a4","batchSubmitterAddress":"0x70997970C51812dc3A010C7d01b50e0d17dc79C8","msg":"Readying to submit next batch..."}
{"level":30,"time":1624315102382,"name":"oe:batch_submitter:state_chain","msg":"Getting batch start and end for state batch submitter..."}
{"level":30,"time":1624315102399,"name":"oe:batch_submitter:state_chain","startBlock":158,"msg":"Retrieved start block number from SCC"}
{"level":30,"time":1624315102412,"name":"oe:batch_submitter:state_chain","totalElements":158,"msg":"Retrieved total elements from CTC"}
{"level":30,"time":1624315102412,"name":"oe:batch_submitter:state_chain","msg":"No state commitments to submit. Skipping batch submission..."}
{"level":30,"time":1624315102723,"name":"oe:batch_submitter:tx_chain","address":"0x70997970C51812dc3A010C7d01b50e0d17dc79C8","ether":"4999.841759976","msg":"Checked balance"}
{"level":30,"time":1624315102723,"name":"oe:batch_submitter:tx_chain","l2ChainId":"0x1a4","batchSubmitterAddress":"0x70997970C51812dc3A010C7d01b50e0d17dc79C8","msg":"Readying to submit next batch..."}
{"level":30,"time":1624315102723,"name":"oe:batch_submitter:tx_chain","msg":"Getting batch start and end for transaction batch submitter..."}
{"level":30,"time":1624315102737,"name":"oe:batch_submitter:tx_chain","startBlock":158,"msg":"Retrieved start block number from CTC"}
{"level":30,"time":1624315102740,"name":"oe:batch_submitter:tx_chain","endBlock":180,"msg":"Retrieved end block number from L2 sequencer"}
{"level":50,"time":1624315102838,"name":"oe:batch_submitter:tx_chain","extra":{"idx":4,"ele":{"stateRoot":"0x3784f6c52f6330c3d17a02a458adb982a8d70a6470bcf527ea019d493e0f86de","timestamp":1624315442,"blockNumber":509,"isSequencerTx":false}},"msg":"Timestamp monotonicity violated! Element"}
{"level":50,"time":1624315102838,"extra":{"message":"TypeError: (intermediate value) is not iterable","stack":"TypeError: (intermediate value) is not iterable\n    at TransactionBatchSubmitter._submitBatch (/opt/optimism/packages/batch-submitter/src/batch-submitter/tx-batch-submitter.ts:234:9)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at loop (/opt/optimism/packages/batch-submitter/src/exec/run-batch-submitter.ts:442:9)"},"msg":"Unhandled exception during batch submission"}
{"level":30,"time":1624315102838,"msg":"Retrying..."}

[smartcontracts]

@K-Ho @ajsantander this is a pretty old issue. Is this still relevant?

[smartcontracts]

Closing this out, I assume this is fixed