Initial steps to generate TASTy for the 2.13 library #17526

nicolasstucki · 2023-05-17T09:50:46Z

The aim is to have a project that compiles the 2.13 library using Scala 3 to generate the TASTy. This will require a special compilation mode to align with Scala 2 semantics. Then in a later step, we can package the TASTy files in a JAR that can be loaded with the class file JAR of the Scala 2 standard library.

`stdlib-bootstrapped` project

This PR changes the purpose of stdlib-bootstrapped. Now the project compiles the Scala 2.13 library (only) sources using -Yscala2-stdlib. With this flag, the compiler will generate code that aligns with the Scala 2 version of the library. The main purpose is to have TASTy that contains signatures that align with the Scala 2 library bytecode.

Under -Yscala2-stdlib we currently

change the signature of the case class unapply methods,
do not emit mirrors,
and inline definitions case class _N.

We add MiMa tests to this project to have a better view of the differences between the Scala 2 generated bytecode and the one generated in this PR. The bytecode differences are a useful guide of differences between the TASTy of the library and how applications will link to it.

[skip community_build]

project/Build.scala

smarter · 2023-05-22T09:57:09Z

compiler/src/dotty/tools/dotc/ast/Desugar.scala

@@ -735,13 +735,26 @@ object desugar {
              .withMods(appMods) :: Nil
          }
        val unapplyMeth = {
+          def scala2LibCompatUnapplyRhs(unapplyParamName: Name) =
+            assert(arity <= Definitions.MaxTupleArity, "Unexpected case class with tuple larger than 22: "+ cdef.show)
+            val optionApply = Select(Ident(nme.scala), "Option".toTermName)


Use scalaDot to get a fully-qualified path.

smarter · 2023-05-22T10:01:38Z

-Yscala2-stdlib should try to set the Scala2x flag on all classes, in particular because this affects erasure: https://github.com/lampepfl/dotty/blob/c205a89c20c3c9d687edfac7fdc5cdec7091d9f6/compiler/src/dotty/tools/dotc/core/TypeErasure.scala#L26-L35

nicolasstucki · 2023-05-22T11:15:33Z

-Yscala2-stdlib should try to set the Scala2x flag on all classes, in particular because this affects erasure:

Yes, this is something that must be done. I did not include it in this first step because setting this flags causes other issues while compiling the library.

smarter · 2023-05-22T11:36:52Z

project/Build.scala

+        IO.delete(classDir / "META-INF")
+        // TODO What to do with library.properties, LICENSE, NOTICE, rootdoc.txt? Keep? Modify?
+
+        // Copy .tasty files from stdlib-2_13-bootstrapped into classDir


The usual way to add files to the output jar is via mappings: https://www.scala-sbt.org/1.x/docs/Howto-Package.html#Modify+the+contents+of+the+package, this is likely to be more robust for interactive compilation etc.

Now we use mappings to filter the .tasty files when packaging stdlib-2_13-bootstrapped.

smarter · 2023-05-22T11:37:40Z

.github/workflows/ci.yaml

@@ -134,7 +134,7 @@ jobs:

      - name: Cmd Tests
        run: |
-          ./project/scripts/sbt ";dist/pack; scala3-bootstrapped/compile; scala3-bootstrapped/test ;sbt-test/scripted scala2-compat/* ;stdlib-bootstrapped/test:run ;stdlib-bootstrapped-tasty-tests/test; scala3-compiler-bootstrapped/scala3CompilerCoursierTest:test"
+          ./project/scripts/sbt ";dist/pack; scala3-bootstrapped/compile; scala3-bootstrapped/test ;sbt-test/scripted scala2-compat/* ;stdlib-bootstrapped/test:run ;stdlib-2_13-bootstrapped/test:run ;stdlib-2_13-tasty-bundle/packageBin ;stdlib-bootstrapped-tasty-tests/test; scala3-compiler-bootstrapped/scala3CompilerCoursierTest:test"


Could we delete stdlib-bootstrapped? If we have both the community build stdlib and stdlib-2_13-bootstrapped then I don't think it serves a purpose anymore

Can we merge stdlib-2_13-bootstrapped and stdlib-2_13-tasty-bundle in one project? This might be easier to do when using mappings in the build as I propose in my other comment.

Could we delete stdlib-bootstrapped? If we have both the community build stdlib and stdlib-2_13-bootstrapped then I don't think it serves a purpose anymore

It still serves the purpose of checking that we can compile the stdlib using Scala 3 semantics/signatures (without -Yscala2-stdlib). It its also a way to check that we are able to compile all Scala 2 and Scala 3 library sources together. This is not something we need to be able to do right now. It could prevent us from breaking something we might need in the long term.

Can we merge stdlib-2_13-bootstrapped and stdlib-2_13-tasty-bundle in one project? This might be easier to do when using mappings in the build as I propose in my other comment.

I removed the bundle project. Now we package only the .tasty files in stdlib-2_13-bootstrapped.

It still serves the purpose of checking that we can compile the stdlib using Scala 3 semantics/signatures (without -Yscala2-stdlib). It its also a way to check that we are able to compile all Scala 2 and Scala 3 library sources together.

This is also checked by the stdlib213 community-build project (it relies on https://github.com/scala/scala/blob/2.13.x/project/DottySupport.scala).

OK, then I guess we can remove it and only keep one.

I rebased and cleaned up the git history of the changes in the build. Now we only modify the definition of stdlib-bootstrapped.

Scala 2 does not generate the methods `_N`. This is solved by inlining those calls, which removes the calls to those methods and the definitions of those methods in the bytecode.

This adds a temporary workaround (`stdlib-bootstrapped-legacy`) to be able to generate documentation for the full standard library. In the future, the documentation will be generated from `stdlib-bootstrapped` and `scala3-library-bootstrapped` combined. Fixes #17964 Fixes #17621 Partially reverts #17526

nicolasstucki force-pushed the bootstrap-stdlib branch 2 times, most recently from a47d0bc to 19a029b Compare May 19, 2023 08:20

nicolasstucki marked this pull request as ready for review May 19, 2023 14:52

nicolasstucki requested a review from smarter May 19, 2023 14:52

nicolasstucki assigned smarter May 19, 2023

smarter reviewed May 22, 2023

View reviewed changes

project/Build.scala Outdated Show resolved Hide resolved

smarter reviewed May 22, 2023

View reviewed changes

nicolasstucki force-pushed the bootstrap-stdlib branch 2 times, most recently from d68d5dc to 52bf33e Compare May 23, 2023 05:54

nicolasstucki requested a review from smarter May 23, 2023 05:55

nicolasstucki force-pushed the bootstrap-stdlib branch from 50d4524 to 24472ff Compare May 23, 2023 08:25

nicolasstucki added 2 commits May 23, 2023 11:24

Only compile 2.13 library in stdlib-bootstrapped

f7d664d

Allow overwriting sources in stdlib-bootstrapped

6c8ae7b

nicolasstucki force-pushed the bootstrap-stdlib branch from 24472ff to e6ea8cf Compare May 23, 2023 09:28

nicolasstucki added 5 commits May 23, 2023 11:37

Add MiMa to stdlib-2_13-bootstraped

c475bd6

Compile stdlib-bootstrapped with -Yscala2-stdlib

85e4f95

Align case class unapply under -Yscala2-stdlib

d4a8dfa

Do not add mirrors under -Yscala2-stdlib

1d1f0d7

Inline _N case class product members under -Yscala2-stdlib

8c58cbf

Scala 2 does not generate the methods `_N`. This is solved by inlining those calls, which removes the calls to those methods and the definitions of those methods in the bytecode.

nicolasstucki force-pushed the bootstrap-stdlib branch from e6ea8cf to 8c58cbf Compare May 23, 2023 09:38

smarter approved these changes May 24, 2023

View reviewed changes

nicolasstucki merged commit d077bbd into scala:main May 25, 2023

nicolasstucki deleted the bootstrap-stdlib branch May 25, 2023 08:09

Florian3k mentioned this pull request Jun 13, 2023

Nightly docs are missing Scala 3 parts of standard library #17964

Closed

nicolasstucki mentioned this pull request Jun 21, 2023

Create legacy Scala 2/3 stdlib TASTy #18025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial steps to generate TASTy for the 2.13 library #17526

Initial steps to generate TASTy for the 2.13 library #17526

nicolasstucki commented May 17, 2023 •

edited

Loading

smarter May 22, 2023

smarter commented May 22, 2023

nicolasstucki commented May 22, 2023

smarter May 22, 2023

nicolasstucki May 22, 2023

smarter May 22, 2023

nicolasstucki May 22, 2023

smarter May 22, 2023

nicolasstucki May 23, 2023

nicolasstucki May 23, 2023

Initial steps to generate TASTy for the 2.13 library #17526

Initial steps to generate TASTy for the 2.13 library #17526

Conversation

nicolasstucki commented May 17, 2023 • edited Loading

stdlib-bootstrapped project

Choose a reason for hiding this comment

smarter commented May 22, 2023

nicolasstucki commented May 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasstucki commented May 17, 2023 •

edited

Loading

`stdlib-bootstrapped` project