ECIP-1029: Include Uncles in Total Difficulty Calculation

[sorpaas]

(Rendered)

Abstract

This proposes to include uncle difficulty in total difficulty calcuation after FORK_BLOCK. New client will use a different Ethereum Wire Protocol version number to broadcast the total difficulty calculated using the new formula. As a result, old clients will continue to function if soft-forked clients are the majority.

Background

Ethereum is advertised to use GHOST protocol for best chain selection. The calculation is done using "total difficulty". The client will always select the chain with larger total difficulty. However, while uncles are included in blocks and miners are rewarded to include them, they do not participate in the calculation of total difficulty thus do not participate in the best chain selection. This deflicts the goal of GHOST protocol, which should always select the chain in which, including stale blocks, is the heaviest.

Analysis by Gervais et al. has shown that, due to the above fact, Ethereum currently resembles a similar Proof of Work algorithm as Bitcoin. Besides, due to its fast block time (around 10 seconds) and growing block size, it is having a much larger orphan rate compared with other Proof of Work cryptocurrencies like Bitcoin, Litcoin and Dogecoin.

[elaineo]

Is the purpose of modifying the difficulty function for the sake of best-chain selection, or in changing how the difficulty is updated? In either case, I don't see a clear benefit to the difficulty change.

It's worth noting that ETH/ETC has had problems with miners optimizing for uncle mining.

[sorpaas]

@elaineo It is for best-chain selection.

This only changes how total difficulty is calculated. Block difficulty is unaffected.

[sorpaas]

Also, this ECIP only dealt with the GHOST protocol part of the uncles (to get us a "properly implemented" version of the GHOST protocol, rather than keeping the sort-of design flaw). Mining rewards of them are not so relevant here. So uncle mining is a separate issue.

[sorpaas]

There's a PR created on go-ethereum (ethereumproject/go-ethereum#334) that implements this change.

[elaineo]

@sorpaas isn't this a hard-forking change? what are the plans for roll-out?

[sorpaas]

@elaineo This is a soft-fork. Note that the protocol version number is changed from 63 to 70. A client broadcast its best block and new blocks available in version 70 under the new total difficulty scheme, while still uses the old total difficulty scheme in version 63.

Total difficulty is not written in the blockchain.

It is expected that when the majority soft-forked, best blocks selected by the new total difficulty scheme will also be the best block selected by the old total difficulty scheme.

To roll out this ECIP soft-fork we need to agree on a FORK_BLOCK number. This can use a hard-coded block number, or ECIP-1022, ECIP-1023, ECIP-1024 or its variants.

[elaineo]

Well... that's not quite a soft fork. If a sub-majority of miners adopt this change, the chain that they continue mining may not be perceived as the heaviest chain by the other nodes, in which case the non-updated nodes will either ignore that chain, or include the newly mined blocks as uncles in the non-forked chain. Am I thinking through this correctly?

Also, I haven't looked at the back tests for this. How much does the difficulty total differ after uncles are included?

[elaineo]

(although, if this update can be combined with the ECIP1017 HF which is already scheduled, then there will be little risk of a split)

[sorpaas]

@elaineo If a super-majority of miners adopt this change, then the chain they continue to mine will be perceived as the heaviest chain by old nodes.

Let the total amount of difficulty (including uncles), be T. We have around 6% orphan rate. So old clients will see the newest block having total difficulty of (1-0.06)T. Because the super majority of miners has adopted this change, it would be highly unlikely that any uncles would have higher total difficulty even under the old scheme.

Even if it does, let's say old clients spin off to an orphan. Because the super majority of miners are mining the block that is considered heaviest for the subtree, this orphan will only be mined by a super-minority of miners, and thus it cannot compete, even under the old total difficulty scheme. So in a really short time (around few more blocks mined), all old clients will switch to the heaviest tree (the new total difficulty scheme with GHOST) and it will be the same as the heaviest single chain (the old total difficulty scheme).

So it's a soft-fork, if my logic is correct.

[sorpaas]

Hmm yes @elaineo if only a sub-majority of miners adopt this change, then the security of the blockchain can indeed not be preserved. But here I think soft-fork assumes that at least a majority of miners adopt this change. correct?

[sorpaas]

Yes. Soft fork all requires majority of miners to change if the security of the blockchain can be preserved. See https://en.bitcoin.it/wiki/Softfork

[whilei]

In which case it seems to me that if we go ahead with GHOST/ECIP1029, it may be best to include it in (or near) the anticipated Monetary Policy/ECIP1017 fork, planned for block 5m.

@sorpaas I'm unclear on why we'd need a new protocol version (as proposed 70) in the case that the difficulty calculations would change as a result of a soft fork?

[sorpaas]

@whilei In my opinion this soft-fork can be carried out along side all our planned hard forks. There's no conflicts to it. So I think we can do it at any block, given we reached agreement. The only thing we need to ensure is that when this ECIP is activated, it has majority of miner support. So if we could get something like ECIP-1022 with this, that would be great. Note that users are not really affected by this soft-fork -- their clients, if not updated, will still function correctly. This ECIP is mainly about reducing orphan rates before it goes insane.

The reason that we need a new protocol version 70 is because it is better to make this a soft-fork. If we continue to use the existing protocol version 63 and change its total difficulty meaning, then all clients will need to be updated -- that will be a hard-fork. Create a new protocol version 70 that uses the new meaning of total difficulty and retain 63's old meaning would mean backward compatibility so users are not affected.

[sorpaas]

Just to clarify some more about protocol version 70 and 63. Soft-forked clients will ALWAYS use 70's total difficulty to negotiate best chain selection. That means, occasionally, 70's total difficulty and 63's total difficulty might select different block for a best chain. In this case, soft-forked clients take 70's one and mine on top of that. Because soft-forked miners are the majority, they will always build on top of the best tree rather than the best single chain. So 70's total difficulty and 63's total difficulty will quickly come to agreement about the best chain again. This is why old clients will continue to function without any problem.

[elaineo]

I don't agree with your use of the term "soft fork", but it looks like we're on the same page here in requiring a majority of clients to upgrade for a safe fork. Non-mining client nodes will still need to update since they participate in broadcast/transmission, and not updating could mean they face frequent block reorgs.

[sorpaas]

@elaineo From my understanding from https://en.bitcoin.it/wiki/Softfork, to qualify as a soft-fork instead of a hard-fork, those are the criteria needed:

1. The existing blockchain structure is unchanged, or only a subset of it is valid. Thus it is backward compatible.
2. Only miners (requiring a majority of them) need to upgrade. Validators (normal users) are not affected (although they face some chances for block reorgs, they are still fully-functional).

You can also argue that non-updating clients for Bitcoin's Segwit have larger chance for block reorgs, as they can accept blocks that are considered invalid by the majority.

Another point I want to make is that if a super-majority of miners adopted this change at the activation block, non-updating clients would only face at most equal reorganization probability as it currently has before the soft-fork. The current chance for reorganization is just the orphan rate -- a client currently has 6% of chance needing to abandon what it's considered the best block and switch to another. If this ECIP has super-majority support from miners, then best tree orphans are always about the same amount of the best single chain orphans (although the best single chain algorithm might occasionally consider a orphan as the best block while the majority would consider another, this would quickly get resolved just as one more block is mined). As a result, the reorg probability for the old clients are still the orphan rate.

You are right for the point that non-updating clients wouldn't be able to participate in version 70's block broadcast but only version 63's. However, if most of the clients still keep version 63 compatible, I don't think there will be a huge problem if they decide not to update.

[elaineo]

@sorpaas Soft-fork means backwards-compatible, so blocks/transactions generated by non-updated nodes are still valid. Bitcoin recently had a user-activated soft fork (UASF, or BIP148 ): Non-updated nodes still generate valid blocks, even if the blocks don't employ the P2WPKH transactions. As far as non-updated nodes are concerned, these just look like anyone-can-spend transactions.

I don't quite understand how changing the protocol version (70 vs 63) enables backward compatibility vs keeping the same protocol version (63) ?

[sorpaas]

@elaineo Protocol version 63 and 70 is in the network layer. After soft-fork, 63's total difficulty still uses the old scheme for both new and old clients, so they are able to communicate with each other. Old clients can use 63 to reach consensus with the whole network.

Keeping the same protocol version would mean that 63's total difficulty meaning is changed, and thus is not backward compatible -- old clients cannot agree with the network about the total difficulty.

[elaineo]

okay got it, thanks! :)

[sorpaas]

A note that Ethereum has EIP100 (ethereum/EIPs#100). Another issue that is related to uncles and difficulty. However, EIP100 requires a hard-fork. So let's address it in another ECIP and don't interference with this one.