Handle timestamps that are ahead of the local browser time gracefully

[frol]

Problem

Time getting out of sync in the network causes weird artifacts on Explorer UI.

Most of the blocks in testnet these days are produced "in the future". Sometimes, it is several seconds, but quite often I see blocks created 2 minutes ahead of time served by time synchronization services.

A single block producer causes the following block producers to produce blocks "the same second" (I believe we ensure strictly monotonical increasing property for the block timestamp, so the block producers that have properly synced time just fall into the trap of a single node that produces the block "in the future" and thus produce the blocks where timestamp is just +1 nanosecond from the previous block):

Block #103430762 timestamp is October 25, 2022 at 8:52:56am
Block #103430763 timestamp is October 25, 2022 at 8:53:55am
and then 13 more blocks have timestamp October 25, 2022 at 8:53:55am - Block #103430776
Block #103430777 is again pushes the blockchain into the future at the timestamp October 25, 2022 at 8:54:13am

How did we discover this problem?

I was alerted by the odd fact that the time reported on the https://explorer.testnet.near.org/blocks page was going backward and blocks were reported to be produced in the same second.

What are we planning to do about it?

To prevent time going backward, I suggest we use the maximum of the local browser time and the most recent block timestamp as our baseline for "current time".

What are we not planning to do about it?

We cannot rely either on the local browser time, as users might have all sorts of issues with their local time.
We cannot rely on the network having properly synced clocks, as we already observed.
Thus, we should have our own strategy for how to report the time to the users in a non-confusing way.

[telezhnaya]

@frol do you know how "regular" this situation may be? I mean the situation when next block timestamp differs from the previous one with only 1 nanosecond
Was it the once-in-a-lifetime exception or it's fully legal blockchain behaviour?
We all know that it may happen again, but anyway, I need the probability 🙂

[telezhnaya]

It's not common, but we meet such cases from time to time. Here are some testnet examples:

1634390031607488593, 1634390031607488594
1641709023449906142..1641709023449906145
1641792023700895915..1641792023700895918
1641845021278521128..1641845021278521131

I think there are much more, we just haven't noticed that before

[frol]

I want to link this conversation with the nearcore team here: https://near.zulipchat.com/#narrow/stream/297873-pagoda.2Fnode/topic/testnet.20blocks.20are.20produced.20in.20.22future.22/near/305969301

we allow 'up to 2 minutes' in the future.
We could change that - but this would be a protocol change.
https://github.com/near/nearcore/blob/63f07683306f3141b7df54c9981fc9664c825d8e/chain/chain/src/chain.rs#L1169