Specs for protocol upgrades ahead of implementation

[anorth]

The go-filecoin were planning to write code for handling protocol upgrades real soon now. Our motivation for doing so is that this is a complex and high-risk item, and the more experience and less time pressure we gain by doing it sooner will help us reach the best result.

However, this is an area that is complex, critical to long term network health, has broad reach, and also has prior art in other blockchains, of which the implementation team may not be especially familiar. If we proceed, the process would likely involve us completing some tech designs, and then proposing the results for the spec. I fear this has a reasonably high chance of a poor result because the resulting ideas and concepts may (a) not be the best informed, leading to a poor end result, and (b) diverge significantly from those already thought about, discussed etc but not yet documented, leading to significant churn and disagreement as we attempt to converge.

I'm therefore raising the possibility that we should wait for the protocol development and spec teams to go deep and establish a foundation. Involvement and interaction with the implementation team will still be critical, but we may be better off with the spec leadership coming from, say, @whyrusleeping. The go-filecoin team can still do some work in anticipation of clear upgrade protocols, but without a shared understanding of the eventual spec, such work is risky.

Some existing spec issues in this domain include #132, #197, #198, #199, but the scope is much greater than just those issues, including handling changes to encoding, block and message structures or semantics, consensus rules, existing actors, new actors, state encoding. @ZenGround0 is currently gathering an understanding of the full problem.

If this idea gains acceptance as a preferable path, I would request that protocol upgrades be a high priority item for speccing, for the reasons in the opening paragraph.

cc @mishmosh @acruikshank

[gnunicorn]

I'd like to add the question/consideration of forkless-upgrades into the mix - the idea to have the an on-chain-indicator (decided upon by the network) regarding when a new version switches live automatically and make it part of the protocol to adhere to that indicator. Meaning that if you reach said block from which the version goes live, but don't speak said version, even if you created a new block, it would be of the wrong version and rejected by the network (even if they didn't run the new version yet either).

Such a mechanism would make "forking the chain off mainline" an active act, that needs coordination (could still be done) rather than something that could happen on accident through "just forgetting to upgrade". I think this is one of the most powerful changes in the way substrate thinks about blockchain upgrades (/ governance).

[whyrusleeping]

@gnunicorn it seems like in that case, all participants would still have to have updated to the software that contains that trigger. Am I missing something?

@anorth (as discussed in our chat) since we are explicitly opting away from doing any fancy governance stuff for mainnet launch (the number of 'new' things we're doing is already uncomfortably high) any network upgrades will simply follow the precedent set by existing crypto networks: When a change needs to happen, software is pushed out that will switch to the new rules at an agreed upon block height, the new software has some way of signaling that its on the new version and intends to switch (this will be something people could obviously lie about) so we can see if the fork will work or not (network splits in filecoin will make miners very unhappy, they lose much more by being on the wrong side than bitcoin or ethereum miners do).

As far as the spec goes, the only thing I can think of would be the signaling mechanism. I've put some thought into that, and think that a generic method for users to post bits of info on-chain would be more useful than a specific non-binding mechanism for this. To that end, I have #178 that would allow accounts to have arbitrary key value storage. We could use this and set a convention that miners signal what their software versions are.

[gnunicorn]

@gnunicorn it seems like in that case, all participants would still have to have updated to the software that contains that trigger. Am I missing something?

Depends how you implemented it. In substrate, for example, we ship a runtime engine and handle these upgrades as wasm-blob changes on the chain. Meaning that if the upgrade is to runtime essentials only (not network or consensus), then you could read the code from chain and just run that. No need to upgrade in 80%+ of cases - your node will probably be slower compared to a native implementation, but you will not fork out.

Secondly, we provide a distinction between consensus upgrades and authoring-upgrades: on the later you can't author a block anymore that would be considered valid from that block on, but you'd very likely still be able to validate blocks and thus wouldn't for of. This is only relevant to a smaller minority of nodes, that actually author and we've played with that in a bigger network yet, but it sounds like a reasonable trade-off: because with that the majority doesn't necesarily have to upgrade but can run their old code, while a smaller minority of highly committed actors has a huge incentive to upgrade as quickly as they can.

---

that said, I don't see this being possible if you upgrade network or core-consensus things: you'd inevetiably fork out, who isn't speaking the same language as you are...

[anorth]

FYI @acruikshank is proposing a network name and protocol version in this design doc. The go-filecoin team are going to implement something like this soon so that we can launch an upgradable network.