Distribute releases as .deb and .msi

[bjornregnell]

Expectation

There are not yet any links to packages under "Other resources" on
https://www.scala-lang.org/download/ for deb and msi to the new awesome Scala 3.0.0

It would be good to know how the naming standard of the links will be when it is eventually up; direct links are published from elsewhere, e.g. in coming course material etc.

Will it be something like this?
https://downloads.lightbend.com/scala/3.0.0/scala-3.0.0.deb

[bjornregnell]

A reason I'm asking is that I need to give links to deb packages to our sysadmin of our university computers that will be upgraded soon.

[SethTisue]

(It won't be lightbend.com, but I don't know if it's been otherwise discussed.)

[smarter]

There's no process in place currently for generating .deb, the released artifacts are at https://github.com/lampepfl/dotty/releases and only includes tarballs currently. I assume our sbt-assembly settings could be tweaked to also generate .debs but we need someone to investigate that.

[bjornregnell]

I guess deb and msi would be good as probably quite a few have gotten used to rely on them over the years and it might be good to reduce the gap between scala2 and scala3. I know our sysadmins prefer .deb over .tar.gz, but if deb don't exist they'll have to manually place the stuff in the file system, which is also doable but more work. (We have a lot of strange stuff installed on our student lab computer's linux installations...)

So, the download link is currently https://github.com/lampepfl/dotty/releases/download/3.0.0/scala3-3.0.0.tar.gz
@smarter Can we rely on that the link will have the same form for 3.x.y for the foreseable future as in
https://github.com/lampepfl/dotty/releases/download/3.x.y/scala3-3.x.y.tar.gz
with scala3 (and not just scala). And will the dotty part of the url be kept or will it be called scala (or scala3) instead?

[smarter]

I think this link format should keep working yeah, even if we rename the repo it should become a redirect.

[bjornregnell]

ok thanks @smarter good to now.

It seems as sbt-native-packager can do deb-packaging with some settings ceremony and installing some linux stuff like dpkg-deb etc:
https://github.com/sbt/sbt-native-packager
https://www.scala-sbt.org/sbt-native-packager/formats/debian.html

[bjornregnell]

@SethTisue How is Scala 2 packages deb msi build currently? Is that process easy to more or less copy-paste for Scala 3?

[SethTisue]

How is Scala 2 packages deb msi build currently? Is that process easy to more or less copy-paste for Scala 3?

all that stuff is over in https://github.com/scala/scala-dist (and https://github.com/scala/scala-dist-smoketest). it uses (a now probably rather outdated) version of sbt-native-packager

scala-dist has been largely untouched for some years now, and I was never super familiar with it in the first place (since we've barely needed to touch it since Scala 2.11 days), so I'm not really sure what the difficulty level would be of adapting it to Scala 3, but offhand I wouldn't expect there to be any fundamental barrier. most of the actual functionality comes from sbt-native-packager.

[bjornregnell]

Thanks for pointers. I had a look and there is some complexity in it above my head, but I guess much of that is due to docker and CI and platform specific stuff. Perhaps the actual packages are easy to generate, but it seems as if one VM is needed per platform etc.

[bjornregnell]

@smarter Since 22 days ago winget has support for 2.13.5. I guess msi-packaging is needed for winget to support also Scala 3. So solving this issue would also help windows user that opt in for the new windows package manager (I guess quite a few of my students on windows will want that). Is there any progress or plans for this?
https://github.com/microsoft/winget-pkgs/tree/master/manifests/s/Scala/scala
https://github.com/microsoft/winget-pkgs/blob/master/manifests/s/Scala/scala/2.13.5/Scala.scala.yaml
https://docs.microsoft.com/en-us/windows/package-manager/package/manifest?tabs=minschema%2Cversion-example

[smarter]

Is there any progress or plans for this?

No one is assigned on this which means no one is working or planning to work on this and it's open for taking by anyone interested.

[bjornregnell]

@smarter (or who is responsible for the packaging of scala 3? perhaps @anatoliykmetyuk who is writing the release blogs)
I was able to build a deb package after doing these preliminary additions (should probably go elsewhere in the dotty build?):
in build.sbt:

//added to try native packaging: should perhaps go elsewhere (?)
//https://www.scala-sbt.org/sbt-native-packager/formats/debian.html#requirements
enablePlugins(JavaAppPackaging)
enablePlugins(DebianPlugin)
debianPackageDependencies := Seq("java8-runtime-headless")
maintainer := "[email protected]"  // Who should be the person? must be a person?
packageSummary := "scala3"  // should include the right version etc
packageDescription := """More information here: https://www.scala-lang.org/"""

in project/plugins.sbt

//to enable native packaging: 
//https://github.com/sbt/sbt-native-packager
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.6")

The changes are available here: bjornregnell@c8e9d6d
The above changes made me able to do:

sbt:scala3> debian:packageBin
[warn] sbt 0.13 shell syntax is deprecated; use slash syntax instead: Debian / packageBin
[info] Wrote /home/bjornr/git/hub/bjornregnell/dotty/target/scala-3.0.0/scala3_3-3.0.1-RC1-bin-SNAPSHOT-nonbootstrapped.pom
[info] Wrote /home/bjornr/git/hub/bjornregnell/dotty/library/target/scala-3.0.0/scala3-library_3-3.0.1-RC1-bin-SNAPSHOT-nonbootstrapped.pom
[info] Wrote /home/bjornr/git/hub/bjornregnell/dotty/tasty/target/scala-3.0.0/tasty-core_3-3.0.1-RC1-bin-SNAPSHOT-nonbootstrapped.pom
[info] Wrote /home/bjornr/git/hub/bjornregnell/dotty/interfaces/target/scala3-interfaces-3.0.1-RC1-bin-SNAPSHOT.pom
[info] Wrote /home/bjornr/git/hub/bjornregnell/dotty/compiler/target/scala-3.0.0/scala3-compiler_3-3.0.1-RC1-bin-SNAPSHOT-nonbootstrapped.pom
[warn] You have no main class in your project. No start script will be generated.
[warn] You have no main class in your project. No start script will be generated.
[info] Building debian package with native implementation
[info] dpkg-deb: building package 'scala3' in '../scala3_3.0.1-RC1-bin-SNAPSHOT-nonbootstrapped_all.deb'.
[success] Total time: 2 s, completed May 30, 2021, 3:48:35 PM
sbt:scala3> 

What is left (at least...) if this is a viable route forward:

• check if the deb package is correct (whatever that means)
• Try it on a windows box to see if .msi can be created and test if it is correct (whatever that means)
• integrate this approach in the dotty build in the "proper" way (difficult for me to understand where because of build complexity)
• integrate this in the CI/CD perhaps dockerize to enable package testing etc (I have no clue on what is needed there)
• Update https://www.scala-lang.org/download/scala3.html
• Add manifest to https://github.com/microsoft/winget-pkgs/blob/master/manifests/s/Scala/scala/
• ...

More inspiration for settings of sbt-native-packager can be found here: https://github.com/scala/scala-dist/tree/2.13.x/project in the Wix.scala Unix.scala and ScalaDist.scala files.

[smarter]

should probably go elsewhere in the dotty build?

Probably in the dist project since that's what's used to generate tarball (dist/packArchive): https://github.com/lampepfl/dotty/blob/master/project/Build.scala#L1717-L1718 /cc @liufengyun

integrate this in the CI/CD perhaps dockerize to enable package testing etc (I have no clue on what is needed there)

We run dist/packArchive and upload it to github in the publish_release job, so that's where we'd have to run whatever extra commands are needed to generate the .deb and .msi: https://github.com/lampepfl/dotty/blob/56abade12dadc0519eb149d53fb88e97fcd31c67/.github/workflows/ci.yaml#L541-L588

[bjornregnell]

@liufengyun @smarter I have pushed some more settings here but the deb-package is still produced in target and not in dist:
bjornregnell@d5d4503
Let me know if I can help more (e.g. I'm happy to test a deb archive and an msi file on my machines); I tried to fiddle with the build architecture but I'm still in the dark on how the build is structured and how to get settings and plugins to be working with the dist sub project. And I have no clue about wix. Anyway, the CI-stuff is better done by someone with admin rights etc.

[liufengyun]

We are currently using sbt-pack. It was a choice several years ago. I think it makes sense to switch to sbt-native-packager, as it supports more features.

Anyway, the CI-stuff is better done by someone with admin rights etc.

@anatoliykmetyuk Could you please extend a hand?

[bjornregnell]

See also coursier/coursier#2058
An .msi installer and a .deb package would be really helpful to regain the scala 2 user experience in terms of installation. Esp. since cs does not work for runing scala 3 in terminal (currently).

[bjornregnell]

The current situation on scala-land.org/download is more problematic than it seems; it actually does not say how to run a compiled app. I think this i pretty important and a severe regression from Scala 2 getting-started-instructions. @smarter Should I open a new issue on that matter, as packaging and absence of "how to run" are separate things?

[smarter]

On https://github.com/scala/scala-lang/ yes

[bjornregnell]

Ok thanks, I'll discuss there if the existing issue scala/scala-lang#1248 should be a more general/specific/split/rephrased to make the importance of a runner for Scala 3 more explicit. I don't think this is "just" a coursier issue, but a degradation from Scala 2 and a "hygiene factor" so to speak.

[bjornregnell]

See also #12551

[bjornregnell]

Any progress on this? My windows-bound students that can't get WSL going run into the brick-wall, so they need an .msi... Let me know if I can contribute somehow.

[michelou]

Any progress on this? My windows-bound students that can't get WSL going run into the brick-wall, so they need an .msi... Let me know if I can contribute somehow.

Starting with version 3.0.2 of Scala 3 (see pull #13006) a Windows user can just unzip the distribution archive and use the batch files scala.bat, scalac.bat, etc.. (just as sbt.bat, mvn.cmd, gradle.bat, ant.bat, etc..). Your students then have to take care of the two environment variables %JAVA_HOME% and %PATH%.

[bjornregnell]

Thanks @michelou for the workaround and for contributing the bat-magic !! ❤️
It still would be very good with a .msi install that sets the path and java home for pure beginners, as with Scala 2. Any progress on that or anything there that I can do to contribute?

[michelou]

@bjornregnell In the past I've developed several Windows installers using NSIS (e.g. RS MINERVE) but WiX 3 looks more adequate to your/our needs since

• it generates a MSI file instead of a EXE file
• it uses XML files (.wxs, .wxi and .wxl) instead of a proprietary (low-level) language
• it benefits from the support of Microsoft (including Visual Studio integration)

I'm experimenting with small examples and will eventually submit a PR if things go well.

[bjornregnell]

and will eventually submit a PR if things go well.

That would be really great! @anatoliykmetyuk @SethTisue It would be cool if this can be integrated with the CI-pipeline that produces the releases? It seems like the sbt native packager delegates to WIX, so perhaps that's the way to go?
https://www.scala-sbt.org/sbt-native-packager/formats/windows.html

[bjornregnell]

FYI, @Husenap has made a custom bat-script just to help us survive the current situation in my scala course with a majority of students on windows, available here:
https://github.com/lunduniversity/introprog/blob/master/tools/install-scala.bat
@michelou

[michelou]

@bjornregnell I saw it; do you expect any feedback from me ?!
PS. Using the GitHub link directly would be better : install-scala.bat.

[bjornregnell]

Yes thanks, I should have used that link. :) I just changed my comment above now to that explicit link.
Sorry for not being explicit about expected feedback while tagging you -- I just wanted to show our work-around if you or others can benefit from that. Perhaps if you can share info on how it goes with .msi-installation-file for Scala 3 and if it is slated for 3.1.0 or if there are any showstoppers so far etc?

[bjornregnell]

@anatoliykmetyuk I have tested this .msi from @michelou 's repo https://github.com/michelou/wix-examples/releases/tag/scala3-3.1.0.msi and it works like a charm :)
Could this work be used as a basis for an official msi for the scala-lang.org release page for windows, so that there is an easy alternative for those on windows, esp beginners who don't know what a path is and just wants a single click official msi install?

[bjornregnell]

It could e.g. for now be linked under "Other ways to install Scala" on https://www.scala-lang.org/download/scala3.html

[anatoliykmetyuk]

/cc @prolativ

[bjornregnell]

Now also API docs included by @michelou here: michelou/wix-examples#2