Phoenix Hard-fork to occur at the same block as Aztlan - to "fix it"

[bobsummerwill]

Discussion for ECIP-1078 and ECIP-1086.

[meowsbits]

Just a note that EIP1283 ( + EIP1884) + EIP1706 =~ EIP2200. Aztlan does not enable 1884.

[TheEnthusiasticAs]

I would lable this as "Aztlán Hardfork" project.

[soc1c]

#293

[meowsbits]

Rel ECIP1086.

So as I understand it, this spec "allows" the incorrect EIP2200 specification as-is (SLOAD.constantGas=200), and will effectively do so on the Mordor and Kotti testnets for the span between the respective Aztlan and Phoenix activations.

This prevents the EIP2200 fix (EIP2200.constantGas=800, eg. etclabscore/core-geth#144) from breaking the testnets by changing (actually correcting) the gas price without doing so in a hard fork. (Kind of a weird concept: the correction would break the incorrect network implementations; this proposal effectively delays the correction indefinitely on these affected networks).

[meowsbits]

Rel ECIP1086.

What if we were to also provide an activation number for the disabling of this allowance, especially with a value below the Phoenix-Fix (ECIP-1078) activation.

Pros:

• Allow us to test correct EIP2200 implementations and outcomes (both for logical soundness, and benchmarking live gas metering as it will be comparable to EIP1283+EIP1708).

Cons:

• Aside from being a comparative reference, this configuration won't directly correspond to any Mainnet configuration, and could thus be seen as overkill.

[soc1c]

This prevents the EIP2200 fix (EIP2200.constantGas=800, eg. etclabscore/core-geth#144) from breaking the testnets by changing (actually correcting) the gas price without doing so in a hard fork. (Kind of a weird concept: the correction would break the incorrect network implementations; this proposal effectively delays the correction indefinitely on these affected networks).

Exactly. Fun fact, such a thing used to happen on Ethereum Foundation mainnet and it was decided to rather specify the bug than rolling back the network.

[bobsummerwill]

For the papertrail, some discussion from ETC Discord ...

Bob: ECIP-1086 makes a lot of sense to me and has my support.

soc1c: besu and geth already have PRs
not sure who would implement this in Parity

Bob: @soc1c So that would likely have to be ChainSafe. I am happy for ETC Cooperative to pay for that work, if there is engineering capacity available for that, @GregTheGreek?

GregTheGreek: @bobsummerwill We can definitely do it

Donald: seems the name should be changed to "ChainSave"

Cody: We control the validators on kotti seems straight forward. There shouldn't be any expectations that testnets wont be reset periodically.
I agree ecip1086 solves the problem, but it is a testnet problem. We found a problem with an implementation and know how to fix it. We can just pick a block number and redo
Resetting the testnet sets expectations

Bob: And that is the plan, right, @DontPanicBurns?
And ECIP-1086 just the specification of that plan to coordinate that switchover between the different clients.
We can argue that there is no need for the ECIP - that we can just agree to make those changes and switch over - just by the clients making that change themselves - but the specification gives us the papertrail and coordination.
I don't see any downside to pushing that ECIP through the process as well as doing the work within the clients.
We do include testnet activation blocks within the ECIP process too, which you could argue does not need that same degree of coordination.
But we have all the same coordination problems with the shared, public testnets as we do with the mainnet. Just not the same economic impact.

Cody: That is not the plan as proposed in 1086
We are doing a testnet only fork. This will not be the same fork that will happen on mainnet

Bob: Right.

Cody: It needs to be a clean test that mirrors exactly what will happen on mainnet when aztlan happens
We are not doing a one two punch on mainnet, we should not do a one two punch on testnet

Bob: So that test is not possible on Mordor and Kotti anymore.
ECIP-1086 is about getting those back in consensus so they can be used.

To do what you are suggesting would need additional new testnets to be created.
As happened with Nazgul for Atlantis.

Cody: Why is it not possible

Bob: And maybe that should happen too, yes.
"Why is it not possible"

Not possible without resetting which might screw end-users up in a way which we cannot quantify.

Cody: It is a testnet
We are the enduser

Bob: But "we" (ie. the group talking here) are not all the users of Kotti.

Cody: Expectation set
Everything on the testnet can be trivially resubmitted
If it isnt trivial, it should be running on mainnet for a fee
We dont know what happens when all the eips/ecips are activated at the same time

Bob: You are describing a desirable world, but that world is not the world we are currently living in. We do not and cannot know the consequences onto end users of resetting the testnets.

There was no clear expectation set that testnets may be reset at will.

"We dont know what happens when all the eips/ecips are activated at the same time"

Sounds like we need some more (temporary) testnets as well as Kotti and Mordor.

Cody: No we need to reset the screed up fork in the yestnet
Or throw the other two in the garbage

Bob: Which would do the atomic Aztlan+Phoenix activation at the same block.
We can dry run that transition multiple times on those testnets.

Same as for the Keccak256 tests. That Ethash to Keccak256 transition is something which you want to dry-run (for all of the stack including clients, mining software, exchanges, wallets) multiple times.

Cody: Kotti is that testnet that should be restarted frequently.

Bob: Maybe.

But that is a change of expectations which would need to be worked through the ECIP process itself, build human consensus, etc.

Cody: Or make an un fork block
O done see any change
Is there an ecip creating the kotti network?
Or anything stating a general view of "why testnets exist"
4 people need to update their nodes to change anything on kotti
Or maybe 5 I forget how many validator we have
Its equally a pain in my ass to update my node to reset the testnet as it is to update my node to do forks

Bob: "Is there an ecip creating the kotti network? Or anything stating a general view of "why testnets exist"

There should be.
Because at some point it became an implicit assumption that any ETC mainnet consensus change should first make its way through the ETC testnets. First, I guess, only assuming Classic Morden, at some point adding Kotti to the mix, and then with Agharta including Mordor instead of Classic Morden.

There was an ECIP for defining Clique, but the creation of Gorli and Kotti happened outside of the ECIP process. @soc1c would be more aware of that history than me.
ECIP-1000 and eventually ECIP-0001 should probably define our expectations about testnets, and our specific expectations for Mordor and Kotti during the phase where we are "dry-running" hardforks.

Even if we have not explicitly specified it as a requirement, I think it exceedingly unlikely that any HF would be Accepted if it did not include activation blocks for both Mordor and Kotti, with the expectation that the ECIPs included in that HF would be tested on both testnets prior to mainnet activation.
@GregTheGreek @isaac @soc1c Would be exceedingly useful for you guys to speak up at this stage, either here on the Discord or better yet on the Github issues and PRs.
Or better yet, @DontPanicBurns, for you and me to take this discussion there too.

Greg: Theres honestly too much noise at this point
its becoming increasignly difficult to monitor and help this channel
I believe i approved @soc1c pr, it logically makes sense
To respond to @DontPanicBurns i completely disagree with you. Rolling back a testnet is not the right way to go about things. Patching all the way. Treat your testnets like its your mainnet.
HF dev is the same as traditional software dev. Your stages:
Dev -> Staging -> Prod

In blockchain world how isee it:
Local or personal networks -> Testnets -> Mainnet
Traditionally speaking, if you broke staging and had to revert, its a failure.

Husain: only 4 different people other than you (Bob/Tala/Cody/Donald ) have been contributing since the 6th of Feb...
and only in the last few days has this been active with Donald/Bob discussing ECIP0001 and now the testnet roll back issue
how on earth is it becoming increasingly difficult to monitor this channel?

Cody: Traditionally speaking, if you broke staging and had to revert, its a failure.

Staging was broke it revealed the error, we need to revert
I approved the ecip because it is proper ally formated

[realcodywburns]

My Tldr of above is we have ill defined, at least imho, whom the testnet is designed to test for. If it is for mainet hf deployment testing, reset it. If it is a platform for app developers to run production parallel, revert the bad fork, then implement as you would on mainnet.

For me, I care more about publicly proving that hard fork features are non breaking than supporting an app graveyard

[bobsummerwill]

I think the essential problem, @realcodywburns, is that these public, well-known, hopefully fairly reliable testnets are used for both purposes.

There is a general expectation of decent availability for Mordor and Kotti, and there is an expectation of continuity. If you have something deployed on these testnets and it gets nuked, that is really rather inconvenient.

I think that perhaps what the universe is telling us is that we need more public networks than we have.

There is no need whatsoever for a long-running public testnet for transition testing. I would propose that, like Nazgul, new ad-hoc, single-purpose testnets would better serve that need, and indeed Whiteblock Genesis might be a good host environment for such.

Those would reboot on some fixed cadence - starting with the current protocol, and then upgrading at some fairly early block. So we could dry-run the transition many, many times.

[bobsummerwill]

Having said that, such a plan for "parallel testnets which are PURELY for testing protocol transitions" sound like a new thing to me, which needs a new ECIP, and coordination in the coming months.

For the here and now, we need to get Mordor and Kotti back in consensus, and given that we have merged fixes for all clients which get back to consensus at SLOAD=800, please could we focus on the matter in hand there.

What is needed to roll that out? @soc1c? @meowsbits? @GregTheGreek?

[GregTheGreek]

It would be nice to unfuck everything and get closer to what a mainnet hf might look like for kotti & mordor #293 achieves that - IMO.

I think it would be nice to do another temporary testnet like we did for Atlantis @soc1c, so we can actually test the double activation that mainnet would experience.

[realcodywburns]

Not ideal but I see the point.

[soc1c]

testnets shall be considered public good for application developers. not breaking them should be our primary goal. not reverting them should be our secondary.

the fix proposed in ecip 1086 is of low impact and just validates the status quo. once the draft is accepted, we can move it straight to final as it's already live on testnets.

[meowsbits]

testnets shall be considered public good for application developers. not breaking them should be our primary goal.

I disagree here; I see them as a tool, not a service. I also see breaking them as our primary goal (along with incidental confidence when we fail to), and have no feelings of passion when I imaging running debug.setHead on a testnet client.

[tzdybal]

IMO:

• as name suggests, test networks are for testing. Test failed - on testnet, not mainnet - this is great!
• we should somehow document this (client's wiki, or something)
• we should fix the clients and move along; rolling back testnets seems perfectly fine for me.

Having separate ECIP and separate fork, and separate configuration in clients (for forks), just to "fix" testnets seems not practical. Resetting seems way more pragmatic.

KISS.

[meowsbits]

Just an update from the development side at ETC Core.

We've just renamed and migrated etclabscore/multi-geth to etclabscore/core-geth, and now we want to make a first release which'll include the rename for network client advertisements.

To that end, we're also needing to manage which ECIPs we include.

I'm going to merge the patch for ECIP1086 which will prevent the testnets from breaking with a fixed EIP2200 implementation. This isn't because I'm trying to strongarm the "Patch it" side of the "Patch it" vs. "Roll it back" debate; but, instead because these approaches are not incompatible from the perspective of the protocol provider (despite having patched the testnets to not be broken, we can still roll them back to whatever, whenever... if that's what we wind up wanting to do -- and then if rollback is decided on, we can simply unplug the ECIP1086 spec from the client as "useless code").

Stay tuned.

[bobsummerwill]

From @sorpaas on OpenEthereum Discord:

By the way I’m considering clarifying this in EIP-2200 — ethereum/EIPs#2514 That will mean reverting #11474. That won’t affect Ethereum at all so shouldn’t be a big deal.

So my read on this (and please do confirm, Wei), is that EIP-2200 reused a variable in a way which worked "by mistake" for ETH, but caused the confusion which is at the root of the consensus split on Mordor and Kotti. Though that consensus split is only currently manifesting for Hyperledger Besu clients.

If the confusion was all just caused by the reused variable in EIP-2200 then the resolution is simple - making Hyperledger Besu match the Parity-Ethereum and Geth/Multi-Geth behaviours and doing a release. The patches to Parity-Ethereum, Geth and Multi-Geth would be reverted, your errata to EIP-2200 merged, and matching code changes to clarify made to each of the clients (though these could not affect consensus - just clarity of implementation of the specification).

@soc1c @GregTheGreek @meowsbits ^^

[bobsummerwill]

OK - chatted with @GregTheGreek. So his understanding is that while the EIP-2200 naming was confusing, this rewording on the EIP would not affect consensus, so we still need the fixes.

And so, my understanding of next steps here, as communicated to Greg and he confirmed his agreement were ...

So we proceed with Afri's ECIP which clarifies the testnet consensus.
We proceed with getting point releases of Parity-Ethereum and CoreGeth.
We upgrade the validators on Kotti and we mine our way back into consensus on Mordor.
Then that is Aztlan "done" (caveat finding further issues).
The point-release from CoreGeth would also include Phoenix support.
You would merge a PR against Hyperledger Besu activating Phoenix support too, and we would need to get a point-release of that too. Maybe that makes 1.4.0?
And then you have a new SOW to generate for me to sign which is to add Phoenix support to OpenEthereum.
And if Wei does his clarification, and then matching "cleans" are done in all the clients that is nice, but not consensus affecting.

[soc1c]

perfect.

[meowsbits]

I have a proposal for what to do with the testnets. It's a compromise. #295

[meowsbits]

Here's a doc I drafted trying to make sense of things. Please feel welcome to comment and fork -- would like to make sure the story is represented at least approaching correctly.

https://gist.github.com/meowsbits/ddf54e7cb225e7f89b3ed0b7d92b8a37

[soc1c]

let's activate 1884+2200 with phoenix.

[meowsbits]

lol

[bobsummerwill]

"let's activate 1884+2200 with phoenix."

Yeah-no, @soc1c.

Thanks so much for write-up, @meowsbits. That is very helpful.

[bobsummerwill]

From Discord ...

Bob: My brain has been stewing and I think I have a better and simpler plan.

In the last 24 hours it has become evident that 2200 is utterly broken. That is fine. It is only ever going to be deployed on the Kotti and Mordor testnets. Geth and Parity are in consensus. Besu is the only one out of consensus.

So we:

• Make Besu match (200 pricing)
• That is Atzlan in the bag.
• Undo the “fixes” in Geth, Multi-Geth and Parity. That was a complete waste of time.
• I ditch the PR again 2200 I just made. Again a waste of time. We don’t try to fix 2200. It is screwed.
• We push releases for all clients with the 1078 Phoenix changes.
• We all get on with our lives.
• No Testnet rollback required.
• No EIP/ECIP changes required

Yaz: does that mean 1884 is included?

Bob:

No
It means Aztlan is weird (known) but only hits testnets.
And Aztlan+Phoenix is equivalent to Istanbul - 1884 repricings, but without the broken 2200 code, which gets disabled as part of Phoenix.
If we had our wits about us Aztlan+Phoenix would have activated at the same block on the testnets too. All this crap is the result of the early activation of Aztlan before we were ready with “the overlay”

The only downside I see to my plan is that we never rectify 2200 in the spec. But that makes no practical difference. The EIPs/ECIPs exist to coordinate and keep the clients in consensus, but they are not mandatory. If we can stay in consensus even if the spec is broken then that is fine anyway.

Maybe ETH Core Devs care and some wording will go through, but we do not need to be gated on that.

And we do not NEED to change the Final text. Clarifying would be good for other future client implementations, but is not urgent.

EIP-2200 can be cleaned up at our leisure. The clients can be cleaned up to make that flow clearer at our leisure. But the existing consensus would be “blessed”. And most importantly we are not waiting on anybody.

Greg:
I kinda said this before in november or whenever. But the underlying mechanics for implementing 1884 and 2200 were due to historical data from ETH. We don't have the same volume on ETC, although it is a security vulnerability (at scale).

so playing around with 1884 & 2200 are a double edged sword

Bob:

“so playing around with 1884 & 2200 are a double edged sword”

Yeah. I think Phoenix scope is honestly fine. We know we have that TODO to we do it carefully within EVM versioning at the next HF. Or maybe even unconditionally at that point. But we do it intentionally, and in a data driven way.

[bobsummerwill]

Issue tracking remaining potential state trie DOS vector to resolve next fork:

#227

[soc1c]

@bobsummerwill "no" or "over my dead body"? There are quite a couple voices calling for 1884...

[bobsummerwill]

I think that we would need another Core Developers call if we wanted to reintroduce 1884, because that is a substantial change in plan. We very intentionally rejected it in November.

I understand that it would be simple to include 1884, but I think it would also be ideologically weak and I don't think it is necessary.

All the pain is coming from 2200 and if we just stick with existing consensus then all that pain goes away too.

[soc1c]

Agreed.

[bobsummerwill]

Also, I think there is real merit in avoiding testnet resets.
If nothing else, it is just a pain to coordinate and risks breaking testnet users.
Where my plan above is something we can just do right now.
And is aligned with the Aztlan+Phoenix plan of record.
If there wasn't the mess with 200/800 then we would have finished that work a long time ago.
It's been a huge distraction, especially considering that 2200 will never hit ETC mainnet anyway.

[soc1c]

@bobsummerwill #298

[soc1c]

let's finally pull aztlan #297

[q9f]

The ECIP-1088 review period ends tomorrow. Please raise your voice now if there is anything we would need to address. Thanks.

[meowsbits]

@q9f Can you give use a reminder summary of the current state of review, like original concerns - if any - and their resolution, the state of the testnets, which tests and/or heuristics are passing, and what will/would happen tomorrow?

[q9f]

Concerns

They are very well summed up by Wei here.

State of the Testnets

Mordor activated Phoenix on March 9th, 2020, block 999_983. All clients ignored the fork block pretty much as the state is not much populated and no transactions were executed. The major issue was not the fork itself but recovering the testnet and reorganizing from the much more heavy Agharta/Aztlan chain that already activated ECIP-1061. Once this was done and mining stabilized, I added additional Parity, Geth, and Besu bootnodes to ease discovery for new clients to sync with.

At this point, I even used old Classic Geth instances to stabilize the network because they had a much better discovery and they didn't crash during mining on low memory machines. Once everything was stable and the Phoenix chain had the higher total difficulty, I used ChainSafe's Anemone to test all opcodes live on the Mordor Phoenix testnet. As expected, the Classic Geth client is now forked off and I started forgetting that it even exists. All other clients, namely Besu, Geth, and Parity are in consensus:

Mordor Besu Phoenix:

2020-03-30 18:29:33.979+02:00 | EthScheduler-Workers-2 | INFO  | BlockPropagationManager | Imported #1,145,174 / 0 tx / 0 om / 0 (0.0%) gas / (0x9ee72b032718739858ebe9bd15832044684bf72d98ade75eccea383a30c4ddcd) in 0.001s.
2020-03-30 18:29:41.100+02:00 | EthScheduler-Workers-1 | INFO  | BlockPropagationManager | Imported #1,145,175 / 0 tx / 0 om / 0 (0.0%) gas / (0x5942d7b038428bf310446bb4f190603893cf90418a7cc2b08fa597a35df5ca19) in 0.001s.
2020-03-30 18:30:03.338+02:00 | EthScheduler-Workers-3 | INFO  | BlockPropagationManager | Imported #1,145,176 / 0 tx / 0 om / 0 (0.0%) gas / (0x87c560f3c6202b87727bde029b8d0f46e7af8998fb3bcca4a2cc4b43ee314cb1) in 0.007s.
2020-03-30 18:30:07.702+02:00 | EthScheduler-Workers-1 | INFO  | BlockPropagationManager | Imported #1,145,177 / 0 tx / 0 om / 0 (0.0%) gas / (0x26afc3cf74077b3cc979c46b522678195366b14c502add6d9c5c73d68879d8a8) in 0.009s.

Mordor Geth Phoenix:

INFO [03-30|18:29:33.473] Imported new chain segment               blocks=1  txs=0 mgas=0.000 elapsed=46.931ms  mgasps=0.000 number=1145174 hash=9ee72b…c4ddcd dirty=439.76KiB
INFO [03-30|18:29:38.498] Imported new chain segment               blocks=1  txs=0 mgas=0.000 elapsed=14.739ms  mgasps=0.000 number=1145175 hash=5942d7…f5ca19 dirty=439.76KiB
INFO [03-30|18:30:00.803] Imported new chain segment               blocks=1  txs=0 mgas=0.000 elapsed=21.899ms  mgasps=0.000 number=1145176 hash=87c560…314cb1 dirty=439.67KiB
INFO [03-30|18:30:06.386] Imported new chain segment               blocks=1  txs=0 mgas=0.000 elapsed=16.702ms  mgasps=0.000 number=1145177 hash=26afc3…79d8a8 dirty=439.83KiB

Mordor Parity Phoenix:

2020-03-30 18:29:33  Imported #1145174 0x9ee7…ddcd (0 txs, 0.00 Mgas, 9 ms, 0.53 KiB) + another 1 block(s) containing 0 tx(s)
2020-03-30 18:29:38  Imported #1145175 0x5942…ca19 (0 txs, 0.00 Mgas, 13 ms, 0.53 KiB)
2020-03-30 18:29:51    14/128 peers   7 MiB chain 62 MiB db 0 bytes queue 33 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2020-03-30 18:30:00  Imported #1145176 0x87c5…4cb1 (0 txs, 0.00 Mgas, 14 ms, 0.53 KiB)
2020-03-30 18:30:06  Imported #1145177 0x26af…d8a8 (0 txs, 0.00 Mgas, 22 ms, 0.52 KiB)

I encourage everyone to use Mordor, send transactions, deploy and execute contracts. Or just mine it to keep the chain running for others.

Kotti is lagging a bit behind for various reasons. Firstly, it required more coordination in the community to align all the authorities. While I was able to mainly recover Mordor on my own with a miner and a special phoenix config, we were basically waiting for client releases of Geth and Parity to happen to have a stable upgrade path for the authorities. This, however, never happened as Parity didn't do any release in the last three months.

So at some point, we were just handing out chain configurations to clients or encouraging Geth validators to switch from Multi Geth to Core Geth which had a release tagged already. This went well as all validators were very responsive and we had the recovered chain going soon after a couple of days.

However, there was this sync-loop bug in Geth that kept these nodes trying to reorganize to the old Agharta/Aztlan chain, see etclabscore/core-geth#75

This was a pain for Kotti validators because if more than three validators ended up in this sync loop at the same time, the network was stuck and this happened quite a couple of times. There are patches in place now to stabilize the authorities and for a couple of days, this network is running fairly stable.

It's not on Phoenix yet, still on Agharta. ETA: April 13, 2020, if the validators do their job correctly ;)

Kotti sees much more activity in general, the state is much bigger and the chain is much busier. I'm looking forward to seeing Phoenix activated on Kotti and run additional tests on that network, too.

Morden finally died.

[bobsummerwill]

From ETC Discord:

@gitr0n1n asked "What is the thought/logic behind using the same gas metrics as the ETH network?"

That is something which ETC has always done. I don't think it is really a valid criticism. Prior to the launch of the Ethereum network, the gas pricings were "best guesses". Some of those guesses were really bad, to the degree where they become DOS vectors or serious performance imbalances and were changed in hard-forks.

That is exactly what is happening here for 1884. Are the costs perfectly tuned for ETC? Probably not. Would it be worthwhile doing a deep analysis of ETC transactions and coming up with an ETC-tuned set of gas prices? Probably, yes. But the repricings in 1884 are part of a set of consistent prices within ETH. "Doing the same as ETH" has always been the starting point for ETC.

This is a "Whataboutism".

When we start intentionally pulling away from ETH it would be entirely sensible (within versioning discipline) to have an ETC-specific gas schedule based on some expensive and deep research into the ETC chain distinct from the ETH chain. That is not now.

[bobsummerwill]

Michael Joseph:

Do we have any progress on a solution to the potential CHAINID issue for ECIP-1088? If I understand correctly, the Phoenix fork will prevent backward compatibility for the loser of the prospective battle that ensues if we have another dramatic fork (to the tune of the DAO attack) which divides the community. Not that we won't be able to make it through, somehow, but if we are anticipating this issue already, is there no way to be proactive?

@bobsummerwill:
The whole concept of Chain ID is based on "Simple replay attack protection" (https://eips.ethereum.org/EIPS/eip-155). Note the word simple. The scheme only works if different chains use different IDs. If there is a fight over IDs then the scheme breaks down at every level.

This ECIP is specifically just exposing that existing information as an opcode, because it is useful. If we want a more robust scheme for replay protection than that is something different again.