Compilation still slow with via-ir

[marktoda]

Was chatting with Georgios about this and ran some benchmarks / tests on https://github.com/uniswap/universal-router. Posting here to hopefully help debug this

Repo:

• solc 0.8.17
• via-ir = true
• optimizer runs = 1,000,000
• forge-std version: 1.1.0 33d4895

Cold build with hardhat: 16s

Cold build with forge: 45s
Removing only test/: 23s
Removing only script/: 30s
Removing test/ and script/: 7s
Removing forge-std and test/ and script: 6.5s

With the hunch that console.log or console2.log was causing this, i removed those from forge-std and all callsites: 40s

So, something in forge-std is causing the slow compilation and it's not console. I can maybe go one-by-one and test further later but wanted to post here in case other folks have time to TAL. 45s obviously not terrible but 3x worse than hardhat and bit of a drag on development

[mds1]

Just want to confirm my understanding and make sure we're on the same page:

• When building with hardhat, the 16s build time is just the contracts/ dir, so the equivalent comparison of foundry vs. hardhat build times is the 7s build time when removing test/ and script/
• Then, compiling scripts takes 16s (23 total minus 7s base)
• And compiling tests takes 23s (30s total minus 7s base).
• This makes sense, since you cited 45s as the full build time, and 7 + 16 + 23 = 46s is very close

There was speculation in #207 that the slow via-ir build times might be related to something in foundry/svm-rs, as opposed to generic solc slowness, which is why I want to confirm that hardhat's 16s time is not including the foundry scripts/tests.

@marktoda Do you mind trying a few other benchmarks and sharing the results? (I tested these changes in forge-std directly using the CompilationTest contract and include the results I saw below)

1. Try compiling with optimizerSteps = '' and see if that impacts build times, see here for how to set this correctly. This disables all via-ir optimizer steps and should reduce build time significantly. If this speeds up build times, you can have separate test and release foundry profiles (i.e. with and without optimizer steps)
   • When I make this change in forge-std and compile with via-ir, compilation goes from 60s to 5s (@leonardoalt pretty sure I configured this improperly last time and told you it did not affect compilation time, sorry)
   • (Side bar: @mattsse maybe this should be renamed to optimizer_steps instead of optimizerSteps for consistency with other config names? Same for other related solc flags)
2. Comment out the _constructor method in StdCheats.sol, which was the problem section originally last time. If this reduces build time, perhaps we change to the solution from Fix long compile times when using via-ir #209 which lazily ran that method on the first usage and just take the small UX hit
   • When I make this change in forge-std, compilation goes from 60s to 4s
3. Try the combination of the first two
   • When I make this change in forge-std, compilation goes from 60s to 3s

If you see similar results, there's two options to consider.

1. Make no changes, and just better document that the via-ir optimizers are enabled by default and suggest different test/release profiles
2. Implement Fix long compile times when using via-ir #209 or something similar for lazy initialization at the small UX expense so users don't need to setup the two profiles

I'd be curious to hear your thoughts on each.

I think the solc team is working on improving via-ir build times too, so hopefully this becomes less of an issue in future versions, but it makes sense the optimizer will always be slower than no optimizer

[ZeroEkkusu]

there's two options to consider

I say the latter option, if that ends up being the case. Don’t want to require additional setup from users - want the library to be as easy to use as possible.

Good analysis, @marktoda, @mds1.

[gakonst]

@marktoda WDYT re Matt's response above?

[marktoda]

Thanks for the analysis here! running these now:

1. optimizerSteps = '': 8s
2. commenting out _constructor: 18s

I think having a test profile with no optimizer steps is a good option, hadn't considered that before. A bit of a hassle / chance of forgetting to set to release mode and accidentally releasing unoptimized code though

IMO the lazy initialization is worth it for a 2x speedup, though I'm not sure I fully understand the UX issue

[marktoda]

Oh, other problem with the test/release profiles is that we heavily weigh gas snapshots in PR reviews. These snapshots would now be much less reflective of real gas if not using optimizer when generating them

[gakonst]

IMO the lazy initialization is worth it for a 2x speedup, though I'm not sure I fully understand the UX issue

Not a big one, it's just instead of calling x you call x() s.t. it does the lazy evaluation.

[mds1]

Ok great. So let’s move to lazy initialization and document the optimizer suggestion in the book as well. I'm traveling this week, @ZeroEkkusu do you think you can take the lead on making these changes?

chance of forgetting to set to release mode and accidentally releasing unoptimized code though

Agreed, the way I like to handle this is keep the default profile to have the release settings, and use that in CI, then a lite profile for local dev and testing. I think this makes it less likely to forget to set release mode, since you would need to explicitly enable the wrong profile during deployment.

Regarding snapshots, @gakonst maybe we bump the priority of foundry-rs/foundry#2056 and discuss options/UX there, and make them easier to integrate into CI?

[gakonst]

Haven't thought about snapshots at all, let's move conversation there. @ZeroEkkusu if you can take point in the forge-std optimizations that would be great. And we can follow-up with the release mode discussion in a separate issue, if @mds1 can take point there with a braindump.

[mds1]

@gakonst re lite vs. release mode, just created foundry-rs/book#709

[zeroknots]

Im on forge-std 1.5 and compile times on a rather simple code base are still over 35 seconds.

optimizer = true
via_ir = true
optimizer_runs = 1000000
optimizerSteps = ''

[⠔] Compiling 63 files with 0.8.17
[⠘] Solc 0.8.17 finished in 29.33s

[mds1]

@zeroknots Can you open a new issue with steps to reproduce? And can you benchmark the time to only compile the source files, compared to both source + test, to make sure the slowness is caused by forge-std?

[zeroknots]

@zeroknots Can you open a new issue with steps to reproduce?

Done :) Hope this is helpful to triage.
#321