What to do about the Scala 3-specific artifact?

[sjrd]

We merged support for Scala 3 in #450. The question is now: what do we do about the Scala 3-specific artifacts of scalajs-dom?

If we release a version 1.2.0 with a new artifacts for _sjs1_3, we can create problems in the binary ecosystem:

• There can be a library A compiled against the _2.13 artifact of scalajs-dom 1.1.0
• And a library a B compiled against the _3 artifact of scalajs-dom 1.2.0

Mixing A and B in the same project will cause both scalajs-dom_2.13 v1.1.0 and scalajs-dom_3 v1.2.0 to be on the classpath. This can cause serious issues.

There are two solutions, IMO:

• Not release the _3 artifact. After all, it does not bring any actual value compared to the _2.13 artifact at the moment. Users will still have to declare their dependency on scalajs-dom using .cross(CrossVersion.for3Use2_13), as they currently do.
• Bump to scalajs-dom 2.0.0, clearly advertising that there is binary conflict with the previous versions of scalajs-dom.

@gzm0 Any opinion?

Anyone else?

[Daenyth]

I'd say that anything normal developers run into that advises them to use for3Use2_13 is undesirable. Bumping the version seems fine.

[gzm0]

Maybe I'm missing something here, but is anything of this specific to scala-js-dom? Feels like the argument above applies to any Scala library that is Scala 2 and 3 compatible.

[sjrd]

Yes, it applies to any Scala library that

• Had a period where its 2.13 artifact was used by Scala 3
• Now wants to publish a real artifact for Scala 3

It seems most library maintainers just pretend that there is no issue. They release a _3 artifact in a minor (or even patch) release.

[armanbilge]

See also: https://gist.github.com/djspiewak/b8f2b4547951442102488964bc351cf9#5-dont-rely-on-withdottycompat

[sjrd]

Daniel Spiewak's advice is harsh precisely because library maintainers have been doing willy-nilly releases of _3 artifacts in random versions. His advice does not hold if we correctly acknowledge and deal with the issue.

[armanbilge]

I do see your point, but as far as I can tell (in my limited experience) this is just deferring an inevitably necessary change which risks increasing downstream pain.

[SethTisue]

Might as well rip the bandaid off IMO

I guess I see for3Use2_13 as temporary, transitional, experimental stuff where as soon as it enters the picture, buyer beware.

So the best thing is to make it unnecessary, one library at a time, and do it as soon as possible.

(I'm agreeing with Arman.)

[sjrd]

So, apparently I am getting flagged down to death here with my controversial opinions, so I guess we'll just bump it to 2.0.0.

[larsrh]

because library maintainers have been doing willy-nilly releases of _3 artifacts in random versions

Speaking for Typelevel, nothing could be further from the truth.

[gzm0]

Huh? That is not how I interpret #451 (comment). From the link:

In other words, treat this like a "normal" breaking Scala update. Try to forget that withDottyCompat exists, because for most intents and purposes, it doesn't exist.

The way I understand this: 2.13 -> 3 is no different than 2.12 -> 2.13. So simply publish _3 for the first version you support it and be done.

FWIW: It seems that bumping to 2.0.0 only improves the situation you describe marginally: The dependency resolver will flag it, but it will still not work.

Another way of looking at this, isn't really the library that introduces a _2.13 / _3 dependency (without appropriate flagging, I do not know how the whole dotty compatibility system works) the culprit?

[sjrd]

The way I understand this: 2.13 -> 3 is no different than 2.12 -> 2.13. So simply publish _3 for the first version you support it and be done.

It's different because using 2.12 from 2.13 simply never worked, never works and will never work. However, it is perfectly possible to use 2.13 from 3 and conversely.

FWIW: It seems that bumping to 2.0.0 only improves the situation you describe marginally: The dependency resolver will flag it, but it will still not work.

Yes, clearly. It's a major version bump to make it clear that it will not work. It doesn't solve anything.

Another way of looking at this, isn't really the library that introduces a _2.13 / _3 dependency (without appropriate flagging, I do not know how the whole dotty compatibility system works) the culprit?

Only if we say that publishing anything that uses for3Use2_13 or for2_13Use3 is a reason for a library to be blamed. If we allow libraries to be published like that, then it was in its right to do so.

It's possible that the consensus among maintainers was that publishing such a library is bad. If that is the case, then we can start publishing for _3 in a minor version, clearly.

[SethTisue]

So, apparently I am getting flagged down to death here with my controversial opinions

Well, we still love you anyway :-)

Perhaps we can re-conceive/re-market “willy-nilly” as, rather, “freedom” 🇺🇸 and/or “diversity” 🏳️‍🌈

[Daenyth]

Only if we say that publishing anything that uses for3Use2_13 or for2_13Use3 is a reason for a library to be blamed.

The advice I've seen is that library maintainers should definitely not use those in published jars, and they exist only for application developers who don't need to care about compatibility

[djspiewak]

However, it is perfectly possible to use 2.13 from 3 and conversely.

To an extent, yes, but not in all cases everywhere. And this is really the rub. It's very difficult across an ecosystem to build on a patchwork of libraries, some cross-published, some not cross-published. The only avenue which works consistently for everything is precisely to cross-publish, so that's what we chose. If we could have made the for3Use2_13 avenue work universally, that's what we would have picked and my advice gist would have looked very different.

The specific problem in this issue is a great example of something that wasn't foreseen beyond the catch-all "we've never done this before and we're probably going to run into something unforeseen." This problem is going to happen for any project which started out saying "use for3Use2_13" and later needs to shift to a full cross-publication. The compatibility issue here does not stem from the cross-publication of the _3 artifacts, with or without a minor version bump. The issue stems from the fact that a capability (being able to use the 2.13 artifacts) was previously guaranteed which is now being rescinded.

It is for this reason that Cats was able to get away with adding functionality in a minor version in the form of newly-published _3 artifacts, whereas scala-js-dom is now in a bit of a bind because you're trying to remove functionality (specifically, for3Use2_13), which necessitates a major version bump.

Not trying to pile on here, just trying to call out the object lesson. I think this is also a bit of a warning to any other projects which 1) care as deeply as you do about compatibility, and 2) have been to date telling users to use for3Use2_13. Each and every one of them will hit the same problem in the end when they themselves upgrade to Scala 3. They should all plan on doing that along with a major version bump.

If a project wants to avoid the major bump, then they need to handle this in a strictly additive fashion, which means full cross-publication from the very start.

[sjrd]

Only if we say that publishing anything that uses for3Use2_13 or for2_13Use3 is a reason for a library to be blamed.

The advice I've seen is that library maintainers should definitely not use those in published jars, and they exist only for application developers who don't need to care about compatibility

At least scalajs-react and Laminar (through raquo/scala-dom-types), two very popular Scala.js libraries, have published _sjs1_3 artifacts that depend on scalajs-dom_sjs1_2.13.

---

@djspiewak

Each and every one of them will hit the same problem in the end when they themselves upgrade to Scala 3. They should all plan on doing that along with a major version bump.

So IIUC, you agree that if we should bump to 2.0.0 if we start publishing scalajs-dom_sjs1_3 artifacts now, is that right?

If we do bump to 2.0.0, there's a number of sweeping cleanups that I'd like to do in this library, notably change all the config objects to use the

var configField: js.UndefOr[Int] = js.undefined

idiom. So the bump wouldn't completely be for nothing.

[djspiewak]

At least scalajs-react and Laminar (through raquo/scala-dom-types), two very popular Scala.js libraries, have published _sjs1_3 artifacts that depend on scalajs-dom_sjs1_2.13.

That's unfortunate, since they and their ecosystem will feel the pain of what happens here quite acutely.

So IIUC, you agree that if we should bump to 2.0.0 if we start publishing scalajs-dom_sjs1_3 artifacts now, is that right?

I don't see any alternative. In a sense this is like removing a signature from the ABI (in this case, the signature is "the entire for3Use2_13 artifact"). The only way to really do that is to communicate it as such, using a major version increment.

[lihaoyi]

For all the libraries in com-lihaoyi, we published separate _3 artifacts for Scala3, even for the ones without macros where technically the 2.13 artifacts could be reused (e.g. requests, os-lib)

Cross publishing is wasteful and clunky, but it's simple enough and has worked for the past 10 years and 3 scala version upgrades. Re-using binaries is a clever trick, but as others have pointed out it introduces all sorts of messiness when you have a mixed dependency graph.

While we could probably figure out some way to make mixed 2.13/3.x dependency graphs work well in all cases, the way cross publishing has conventionally worked in the Scala ecosystem definitely feels to me like it "ain't broke", so easier to just keep doing what we've always been doing

[armanbilge]

That's unfortunate, since they and their ecosystem will feel the pain of what happens here quite acutely.

Especially so, considering that the Scala 3 cross-build was such a trivial PR :(

[raquo]

At least scalajs-react and Laminar (through raquo/scala-dom-types), two very popular Scala.js libraries, have published _sjs1_3 artifacts that depend on scalajs-dom_sjs1_2.13

So uh... what was the proper procedure to safely cross-publish raquo/scala-dom-types for both Scala 2 and Scala 3, given that my dependency (scala-js-dom) was not published for Scala 3?

[armanbilge]

@raquo to make a PR like #450 and (politely) bug @sjrd until he published 😅

[raquo]

I mean... I'm told that I did something wrong by publishing scala-dom-types for Scala 3 with a dependency on the 2.13 version of scala-js-dom. But, I don't see a different way to do it, yet it is implied that such a way existed, so I'd like to know what the correct solution was (aside from just waiting for scala-js-dom to be published for Scala 3, which was for a long time blocked by some issue in dottydoc).

[SethTisue]

I'm told that I did something wrong by publishing scala-dom-types for Scala 3 with a dependency on the 2.13 version

You had only bad options. (Namely: do what you did, or wait.)

[armanbilge]

I noticed that @japgolly recently released scalajs-react 2.0.0-RC2. If it's not too late, it seems like it could be a big win if scala-js-dom can release 2.0.0 with Scala 3 artifacts before scalajs-react 2.0.0 finalizes. This will help consolidate some of the aforementioned pain.

[sjrd]

I am on vacation and will come back to this issue in a couple of weeks.

Please also read my earlier comment about sweeping cleanups.

[japgolly]

That's unfortunate, since they and their ecosystem will feel the pain of what happens here quite acutely.

Sorry what is this looming catastrophe you're talking about? I'm happy to hold off releasing scalajs-react 2.0.0 and do it properly.

[japgolly]

I am on vacation and will come back to this issue in a couple of weeks.

@sjrd It seems to me that you might have too much on your plate, mate. (And I'm gonna remind you here again: you still haven't replied to my email. Wth? Has anyone reading this ever gotten a response from the Scala Centre in a professional capacity?) Anyway, back to the immediate point, I'm starting to think you might be stretched too thin. You do so much awesome work but this library always seems to be left behind. The last release was nearly a year ago, PRs have been queued up for ages with no attention. It's really frustrating as a downstream user of such a fundamental library, to be kept waiting for so long, so regularly. How about sharing some of the scala-js-dom admin with a few community members? We could get all the PRs merged, proper Scala 3 support and publish a 2.0 in a few days. I'm happy to do that work if that's what it takes. I'm sure there'll be others happy to help as well (please make yourself known).

[armanbilge]

@japgolly thanks for chiming in and your attention to this matter. I'm also happy to help make 2.0 happen.

[sjrd]

How about sharing some of the scala-js-dom admin with a few community members? We could get all the PRs merged, proper Scala 3 support and publish a 2.0 in a few days. I'm happy to do that work if that's what it takes. I'm sure there'll be others happy to help as well (please make yourself known).

If there are people willing to take over scalajs-dom, it is quite possible to fork it, publish it on your own, and then claim for it to be the true one. I am more than happy to discontinue this repo in favor of a better maintained one, since I don't have expertise with working with the DOM. This wouldn't be the first: scala-java-time is a prime example of success of this model: it has completely superseded scalajs-java-time that we used to maintain.

[japgolly]

No problem and fair enough. I'll get the ball rolling tomorrow('ish) by forking and creating a new Github organisation just for this (so that it's ownership isn't tied to me directly) and we can go from there.

[lihaoyi]

To be honest, I'd say that the ideal Github organization for this library already exists: @scala-js. I don't think it makes sense to go create a new one. This begs the question: what is the @scala-js org for? Is it strictly for @sjrd working on the compiler? Or should it include Java/JS standard library support as well?

scala-js-dom is basically the facade for the de-facto JS standard library for people using Scala.js in the browser (probably most of them). So if @scala-js should hold support for the standard libraries, then having scala-js-dom be shared ownership under the scala-js org makes more sense than having it shuffle between the personal accounts of whoever happens to be maintaining it at the time, or some other odd @scala-js-but-not-really-scala-js organization.

And if we wanted to share ownership of @scala-js libraries with anyone, then @japgolly seems a good candidate as someone who maintains important OSS Scala.js projects and has been heavily invested in the ecosystem since early on (7 years now, his first email to the list was 3 July 2014!).

TBH I'd prefer that scala-java-time were under @scala-js as well, but that it started off with multiple competing forks where it wasn't clear which (if any) would win, so it made sense to keep them as forks. This does not apply to scala-js-dom, which has been stable and under @scala-js org since forever. Both scala-java-time and scala-js-dom are basically parts of the Java/JS standard libraries

[smarter]

Mixing A and B in the same project will cause both scalajs-dom_2.13 v1.1.0 and scalajs-dom_3 v1.2.0 to be on the classpath. This can cause serious issues.

sbt >= 1.5.5 will error out if it detects both foo_2.13 and foo_3 dependencies in one project (previous versions of sbt didn't error out with _3 specifically due to a bug: sbt/librarymanagement#383). I don't know what mill does but if it isn't already it should definitely add a check to prevent mixing libraries with different cross-suffixes, because even in a Scala 2 only world, one can end up depending on both foo_2.13 and some-java-library that depends on foo_2.12. All this to say that I don't think that cross-publishing with _3 requires a version bump by itself.

[japgolly]

I've let this situation simmer in my head and after some more thought, I've decided it's best to retract my offer (for now). I don't think setting up a new org for this is a right way to go. @lihaoyi, wonderfully as always, has articulated some of my concerns so see above. The remaining consideration I have, is the impact on the community. If there are two independent versions of sjs-dom then:

• it's inevitably going to cause pain when a user ends up with both on the classpath, especially considering how widespread of a transitive dependency this library is. Most OSS projects don't have the luxury of dedicated full-time maintenance so it will be yet another typical-Scala-community-migration that will go on for a year, with people being blocked by dependencies for months. I don't know about everyone else but I'm sick to death of what feels like a never-ending amount of maintenance churn on Scala libraries. I justified the necessity and have painted it in a good and worthy light for years, but as of this year I'm done and I don't want to create more of this burden for everyone else. In this case it's just not worth the cost.
• should we change the package name in the fork? If yes, everyone in the community needs to change all of their imports, and alternative versions will be able to coexist (which adds more bloat to output JS sizes which are already huge). If not, then the above problem of both libs on the classpath becomes a dice roll.

The alternative is just to add a few maintainers to the scala-js-dom repo and to automate releasing via Github Actions. That to me seems to be better tradeoff for everyone. There's no need to add people to the scalajs org, individuals can be added to specific projects so if you're understandably worried @sjrd about handing over keys to bigger repos, that's very easily avoidable. Are there any other concerns to consider?

[armanbilge]

+1 to @lihaoyi and @japgolly. FWIW, my understanding was that @japgolly was initially offering to become a maintainer for this repo, but at @sjrd's recommendation was open to forking instead.

However, regarding the change in package name: this actually could be a very good thing. Given the unfortunate situation where a number of libraries currently depend on scala-js-dom via .cross(CrossVersion.for3Use2_13) and @smarter's comment that sbt will (correctly) prevent the mixing of the _2.13 and _3 artifacts, as far as I can tell keeping the artifact/package name would cause the most churn and breakage. I.e. there is absolutely no way to use library A depending on scala-js-dom_2.13 and library B depending on scala-js-dom_3 in the same project. As far as I can see, the only way to avoid this would be an artifact/package name change, to allow the new scala 3 artifacts to co-exist with the old. This way the JS size bloat could act as an incentive to make the change without completely breaking things.

[japgolly]

@sjrd Please make a decision. We're being held up by this downstream.

[SethTisue]

I am not a Scala.js or scala-js-dom user and thus I am mostly an outsider to this discussion, but I will offer my perspective anyway.

Basically, I agree with Haoyi. (And I also agree with David's points about not creating unnecessary disruption for downstream users and maintainers.)

I think we should consider it normal for there to exist projects which remain under the Scala organization's umbrella but which are primarily community-maintained, rather than primarily maintained by actual Scala Center, LAMP, or Lightbend staff.

Take scala-xml for example. It has now been primarily community-maintained for some years now, but it is still under scala/ on GitHub and is still published to Sonatype under org.scala-lang. Over the years, community maintainers are likely to come and go. The Scala organization anoints new maintainers and recruits new maintainers if they are needed. We don't seek to interfere with ordinary, everyday decisions about the code and or about how the project is run, but we do retain ultimate ownership and veto power over the most important decisions.

I think that model makes sense for scala-js-dom. It makes sense for the Scala organization to retain ownership:

• because the project originated here
• because it's foundational for the Scala.js front-end ecosystem
• because the community trusts the Scala organization and feels most comfortable relying on projects the organization blesses and shepherds for the long term
• because having a single blessed, "official" solution prevents undesirable community fragmentation, because sometimes it is more valuable to have a single preferred solution than to have multiple competing solutions, which can be highly confusing for users, especially when they are just learning Scala or just beginning to adopt Scala at their organizations

There is a downside to the Scala org retaining ownership, namely: it can discourage community maintainers from being fully invested and involved, they since don't fully "own" the project they are maintaining. But this has to be weighed against the benefit to community maintainers, who might also be pleased and proud to be working on something which is officially blessed and has some level of protection from pointless competition/fragmentation.

In short, I hope a way forward can be found for this library to stay under the Scala org's umbrella, rather then being entirely cut loose.

[aappddeevv]

$0.02:

I just started working on upgrading my published libs and immediately hit this issue. I think I understand the issues although not perfectly.

Since 1.1.0 has not really changed in awhile and people have their workarounds/fixes for various aspects they care about, I know I would be personally OK (this opinion represents just myself) with freezing 1.1.0 and shifting all work to scala3 and publishing under 2.0.0+.

If there is limited bandwidth by any maintainer, I think having something cleaner and easier to maintain that represents the active path is better as it offers the best opportunity for maintaining the change velocity. The DOM will continue to change and this is a foundational library from my perspective. I think I read above that maintaining 2.x and 3.x and various cross versions would be more work for a maintainer. And I think keeping it under this org is a good idea. Adding a maintainer to just this repo should be possible.

[sjrd]

Given the discussion above, here is what I propose:

1. Keep this repo, in the scala-js org
2. Bump to 2.0.0-SNAPSHOT (i.e., merge Bump to 2.0.0-SNAPSHOT. #452)
3. Hand over the keys to this repo to whomever wants to be a maintainer
4. Perform some cleanups that are long due:
   • Change config object traits from native to non-native JS trait with js.undefined members (see my comment above)
   • Move dom.raw.* to dom.*, as those raw things are quite ugly, and they show up in error messages because stuff in dom.* are type aliases to them, not the other way around
   • Move dom.experimental.* to dom.*: things become non-experimental at some point. If they are in experimental.*, they have to stay there forever, for compatibility, so eventually things become nonsensical
   • Get rid of most of the ext.* stuff, which are actually bad practice (it's full of unconstrained implicit conversions); keep the Seq API over NodeList and such, but move them to their respective companion objects, so that they are automatically available.
5. Set up tag-based publishing, and publish 2.0.0
6. Let this repo be managed by the new maintainers

Does that sound like a reasonable plan?

[lihaoyi]

That does sound like a reasonable plan

I like the cleanup for the dom.raw/dom.experimental/dom.ext stuff. Some of that code made sense to me 7 years ago, but it definitely wouldn't pass review for code written today.

[sjrd]

Here is a first wave of cleanups: #458. Feel free to review it.

For those who would like to become maintainers, please say so and take a stab at reviewing the PR. Perhaps @armanbilge @japgolly ?

[armanbilge]

Thanks @sjrd. I think that I'm too new to Scala.js to be qualified to be a maintainer, but I am eager to help out. I think @japgolly is a great candidate if he has the bandwidth available :)

[japgolly]

Thanks @armanbilge . Yeah I don't really have the bandwidth but I keep signing up for things somehow so I'll do my best :) @sjrd np, send me an invite and I'll review and merge.

Also if anyone else can join me as a maintainer I'd really appreciate it. Good to both share the burden and provide different perspectives.

[armanbilge]

@japgolly If you'll have me, I can be a deputy maintainer 😆

[japgolly]

@armanbilge Thank you and it'd be my pleasure! We're all in this together 😄

[armanbilge]

Also maybe @raquo? It seems they have an important stake in the library as well.

[sjrd]

@japgolly

Thanks @armanbilge . Yeah I don't really have the bandwidth but I keep signing up for things somehow so I'll do my best :) @sjrd np, send me an invite and I'll review and merge.

Also if anyone else can join me as a maintainer I'd really appreciate it. Good to both share the burden and provide different perspectives.

@armanbilge

@japgolly If you'll have me, I can be a deputy maintainer 😆

Invitations sent.

[aappddeevv]

I know this is closed now, but @japgolly and @armanbilge if you find that you are unable to get lift off on this, please reach out to me.

[armanbilge]

@aappddeevv thanks for the offer!! If you have a moment to review PRs and/or create PRs to address any "help wanted" issue that would be much appreciated! :)