-
-
Notifications
You must be signed in to change notification settings - Fork 346
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Pre compile zinc compiler interface for common Scala versions (#2424)
While we still the to keep the dynamic codepath necessary to compile stuff at runtime, at least for the common case of existing Scala versions we can pre-compile it, improving performance and removing some of the confusing log messages about compiling stuff that the user didn't write. This only affects the first load, but first impressions matter. Doesn't benefit Scala 3, but there's half the community is still on Scala 2, and Mill's own `build.sc` is still on Scala 2 Before, we can see that a clean `run` ends up compiling the compiler interface twice (once to compile `build.sc`, once to compile the user's `ScalaModule`) taking ~14s on my laptop ``` Compiling compiler interface... [info] compiling 1 Scala source to /Users/lihaoyi/Github/mill/example/basic/1-hello-world/out/mill-build/compile.dest/classes ... [info] done compiling [34/48] compile Compiling compiler interface... 8 warnings [info] compiling 1 Scala source to /Users/lihaoyi/Github/mill/example/basic/1-hello-world/out/compile.dest/classes ... [warn] 1 deprecation (since 2.13.0); re-run with -deprecation for details [warn] one warning found [info] done compiling [48/48] run <h1>hello</h1> /Users/lihaoyi/Github/mill/target/mill-release -i run --text hello 47.58s user 1.70s system 343% cpu 14.345 total ``` After, we compile the compiler interface zero times, with a clean `run` taking ~7s on my laptop ``` [build.sc] [40/47] compile [info] compiling 1 Scala source to /Users/lihaoyi/Github/mill/example/basic/1-hello-world/out/mill-build/compile.dest/classes ... [info] done compiling [34/48] compile [info] compiling 1 Scala source to /Users/lihaoyi/Github/mill/example/basic/1-hello-world/out/compile.dest/classes ... [warn] 1 deprecation (since 2.13.0); re-run with -deprecation for details [warn] one warning found [info] done compiling [48/48] run <h1>hello</h1> /Users/lihaoyi/Github/mill/target/mill-release -i run --text hello 19.27s user 0.85s system 297% cpu 6.768 total ``` ## Implementation I added cross modules `bridge[_]` for all supported Scala versions. These more or less follow what `ZincWorkerModule#scalaCompilerBridgeJar` already does on-the-fly: download the respective source jar from Maven Central and compile it using the respective Scala version. Note that the `META-INF` metadata is necessary for Zinc to properly pick these up, and so I manually copy the folder from the unzipped source jar into the `resources` so it can get propagated to the final jar. To avoid the slowness of compiling every bridge version during local development, we separate the `bridge[_]` publishing from the rest of the build, only enabled via the `MILL_BUILD_COMPILER_BRIDGES=true` environment variable, and published under a separate version. We expect to bump the `bridge[_]` version rarely, and want to avoid redundantly re-publishing them every Mill release. ## Testing Most of the existing unit tests use the three scala versions that we pre-compile in development mode, though some of them are left on other versions for legacy reasons e.g. they depend on a specific version of Scala to be compatible with a specific version of semanticDB or scala-native, and those continue to use the old download-compile-on-the-fly code path. I also added a new test case `mill.scalalib.HelloWorldTests.compile.nonPreCompiledBridge` that together with `.fromScratch` specifically exercises the code paths for compiled/not-compiled compiler bridges, asserting that the compiler bridge is compiled in the versions it should be compiled for (i.e. those that are not pre-compiled) and not compiled for versions it should not be compiled for (i.e. those that *are* pre-compiled)
- Loading branch information
Showing
48 changed files
with
401 additions
and
195 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| name: Publish Bridges | ||
|
|
||
| # Manually-triggered github action to publish mill-scala-compiler-bridges jars, | ||
| # since those do not change frequently enough to be worth including in the main | ||
| # publishing workflow that runs every Mill version | ||
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
| jobs: | ||
| publish-bridges: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| concurrency: publish-sonatype-${{ github.sha }} | ||
|
|
||
| env: | ||
| SONATYPE_PGP_SECRET: ${{ secrets.SONATYPE_PGP_SECRET }} | ||
| SONATYPE_USERNAME: ${{ secrets.SONATYPE_DEPLOY_USER }} | ||
| SONATYPE_PASSWORD: ${{ secrets.SONATYPE_DEPLOY_PASSWORD }} | ||
| SONATYPE_PGP_PASSWORD: ${{ secrets.SONATYPE_PGP_PASSWORD }} | ||
| LANG: "en_US.UTF-8" | ||
| LC_MESSAGES: "en_US.UTF-8" | ||
| LC_ALL: "en_US.UTF-8" | ||
| MILL_BUILD_COMPILER_BRIDGES: "true" | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - uses: actions/setup-java@v3 | ||
| with: | ||
| java-version: 8 | ||
| distribution: temurin | ||
|
|
||
| - run: ci/release-bridge-maven.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -eu | ||
|
|
||
| echo $SONATYPE_PGP_SECRET | base64 --decode > gpg_key | ||
|
|
||
| gpg --import --no-tty --batch --yes gpg_key | ||
|
|
||
| rm gpg_key | ||
|
|
||
| # Build all artifacts | ||
| ./mill -i __.publishArtifacts | ||
|
|
||
| export MILL_BUILD_COMPILER_BRIDGES=true | ||
|
|
||
| # Publish all artifacts | ||
| ./mill -i \ | ||
| mill.scalalib.PublishModule/publishAll \ | ||
| --sonatypeCreds $SONATYPE_USERNAME:$SONATYPE_PASSWORD \ | ||
| --gpgArgs --passphrase=$SONATYPE_PGP_PASSWORD,--no-tty,--pinentry-mode,loopback,--batch,--yes,-a,-b \ | ||
| --publishArtifacts bridges.__.publishArtifacts \ | ||
| --readTimeout 3600000 \ | ||
| --awaitTimeout 3600000 \ | ||
| --release true \ | ||
| --signed true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.