Tracking issue- move from deps to toolchains

[ittaiz]

Description:
We'd like to move from using the deps/repositories pattern and bind to toolchains.
A great deal of the work has been done in #399 #530 and finally in #544 but we only handled scala itself and left the other parts of the repo untouched. This means that using a different proto version or specs2 is still difficult.
Motivation:
For one bind is deprecated, it causes issues with labels (strict-deps/unused-deps) and also the configurability roadmap (which toolchains belongs to) specifies that one should be able to run bazel with more than one toolchain configured (so we will be able to have a single bazel build cross building scala versions for example).
Additional motivation surfaced in #769 where unclarity on which dependencies actually come from which features of rules_scala led to a user asking to split up the repo. If we had separate distinct toolchains this view would have been easier.

• proto
• scrooge
• specs2
• scala_test
• tut
• jmh
• scalafmt

[smparkes]

FYI, more than anything:

We're looking at what it will take to be able to compile with multiple versions of scala in the same build. We only need to support different major versions, we don't see the need to support different minor versions.

We need to be able to build the same (for some definition of "same") library/binary against different versions of scala and also different versions of dependencies. Doing something like external templating of WORKSPACE and/or BUILD files is a very last resort for us.

We end up defining BUILD/.bzl pairs where targets are defined in macros that take an environment parameter and create targets for that environment. We then call that macro once for each environment. We implement this a bit like scala does with version suffixes. This lets us build the "same" target e.g., //package/:some_library_{env} for different runtime environment (i.e., classpaths) where we don't have control over all the dependences.

The macros handle all the muck around scala version suffixes and also clean up some of the junk that we, along with @johnynek, found annoying with rules_jvm_external. We write deps that look like

        "@{environment}//com.fasterxml.jackson.module:jackson-module-scala_{scala}",

and use macro code to do the conversion to labels. We do find the pinning/version resolution in rules_jvm_external/coursier really important.

The last step for this effort is supporting multiple scala compiler/runtime versions in the same build. Working on that now ... though I'm only now getting familiar with toolchains so it's early. I'd like to do this part "right" but not sure I know enough to get it right (and for our use, we don't necessarily need this to be "right" as long as we don't have to make incompatible changes with master: "right" would be nice but not sure the effort levels.)

[ittaiz]

I understand.
I should say that this is not the main motivation for our move and so I don't want rules_scala to codify a pattern of solving this when Bazel are lined up to solving it in a clear way in the next months (definitely in 2020).
Having said that it seems that if you're willing to live with two different bazel invocations (one per version) then you might already have this ability right now with toolchains.
Have you tried it on a vanilla target without all of the above mentioned toolchain gaps?

[smparkes]

Understood.

Thanks for the comment: I haven't kept up with what the bazel folks have been working on.
https://bazel.build/roadmaps/configuration.html and more precisely
bazelbuild/bazel#6519?

Helps to know that other folks are looking for the right pattern: makes it a bit easier to live with something a bit more hack-ish knowing there's a better solution coming up that we should be able to move to when it lands.

Having said that it seems that if you're willing to live with two different bazel invocations (one per version) then you might already have this ability right now with toolchains.

I looked at this but didn't find a solution ... but there are a lot of different ways to approach this so I'm not at all confident I considered the right alternatives.

The blocker for us for multiple invocations has been using different versions of deps for different platforms. Since platform is not available until analysis, it appears we would have to go way deeper into rule implementation and I'm not sure I'm particularly competent there. The macro approach (for environment in environments.items(): scala_library(...)) feels like a hack but so far has been handling things "okay". I'll spend some time on the roadmap to see what others are doing (feature flags, etc.)

Not sure there's anything actionable at this point for others (but happy for input). If things go well, I'll put up an example of what we've done and see what others think/use as a place for folks to share thoughts. Maybe folks at that point can point me in a better direction and I can iterate from there.

Definitely looking to collaborate with others that are facing the similar issues ...

[ittaiz]

I asked on bazel slack about this and got a few pointers namely
Configuration transitions and configuration flags.
I also think that indeed cracking the repository rule and scala version is a hard nut as you can see from our discussions on #544

[smparkes]

See also bazelbuild/bazel#7572

[smparkes]

I actually made progress on this. I wasn't sure I was going to be able to but it's actually come out fairly well (at least enough that I can live with it.)

Example: https://github.com/smparkes/multi-scala-example

I'll put more comments in the PR. This is only a PR within my fork: if there's interest in pursuing it, I can switch the PR to here and we can chat there.

[JamieRu]

@smparkes does this allow having multiple minor versions as well (e.g. using both 2.12.0 and 2.12.5 simultaneously)?

[smparkes]

I hadn't planned on that. It's not a need for us.

It would be a moderately simple extension. It would require suffixing everything with the complete version,e.g., my_library_2_12_5 which is non-standard and a bit intrusive to others. That might be mitigatable with automatically created aliases.

See #962.

[Jamie5]

Having this would be useful in adding tests for #867 and I suspect useful in other situations as well. So would be great if we can get that.

Should future discussion go in #962 ?

[liucijus]

Hey, here are my ideas how to introduce toolchains for dependencies: #1067