-
Notifications
You must be signed in to change notification settings - Fork 59
Maintenance
This is partly for community build maintainers on the Scala team, and partly for project authors interested in having their projects in the community build and keeping the version included current.
The Troubleshooting a failure page is more specifically targeted at project authors and external contributors. (It overlaps somewhat with this page.)
You'll want to be familiar with the basics of our CI infrastructure, as documented at https://github.com/scala/scala-jenkins-infra/blob/main/doc/overview.md. Most community build maintenance tasks only involve making commits in the scala/community-build repo, but some types of improvements may require scala/scala-jenkins-infra changes.
You'll need an account on our Jenkins instance (https://scala-ci.typesafe.com) with enough power to start jobs, cancel jobs, and edit configs. (Permanent config changes are made through scala-jenkins-infra, but it's sometimes helpful to make temporary changes on the fly.)
Getting set up to make scala-jenkins-infra is harder; you might wait and see if you need it. See the doc in that repo.
dbuild is the meta-build-tool that is the foundation of the community build.
You'll need at least a basic grounding in what dbuild is about; so at least skim through the dbuild manual at https://lightbend-labs.github.io/dbuild/ . If it isn't clear 1) why something like dbuild is needed for this, and 2) basically how dbuild does its job, talk to someone on the Scala team about it.
Don't forget to login into Jenkins, or you won't see all of the options that are available. Jenkins is exceptionally eager to log you out again, so prepare to find yourself clicking that "login" link a lot. (Help wanted: find out why that is and fix it?)
There are multiple jobs, entries in a matrix of Scala versions (2.12/2.13) and JDK versions (8/11/17).
All the builds run nightly. When you're actively maintaining the builds, you'll be triggering a lot of runs manually. The "rebuild last" button is handy for this.
Each build has parameters, with an appropriate set of defaults. When manually triggering a build, you'll often want to change some of the following:
-
repo_user: if you're testing a change in your fork of the community-build repo, change this fromscalato your own GitHub user name -
repo_ref: your change in your fork is presumably in a feature branch, so put the branch name here -
version: this is a Scala version number such as2.13.2-bin-abcd123(nightly Scala) or2.13.2-bin-abcd123-SNAPSHOT(Scala from PR validation)- dbuild caches results from previous runs of the community build, so when testing changes, your first choice will to be use a version number for which a recent fully-green build exists.
Manually aborting a community build through the Jenkins UI doesn't lose any completed and cached builds; so if you have additional changes you want to test, it's OK to abort an existing run and start a new one with your additional changes.
As mentioned already, there are multiple builds by Scala version and JDK version.
Changes are merged from 2.12.x -> 2.13.x. Since the 2.12 build is pretty much frozen these days, this will normally just be things like sbt version bumps and improvements to the various scripts. There isn't any schedule or automation for the merging, it's just something we do when we remember to.
Particular builds are assigned to particular Jenkins workers ("behemoth" 1, 2, or 3) to get the greatest advantage of dbuild's caching.
You'll want to look over the issues in https://github.com/scala/community-build/issues to get an idea of what shortcomings currently exist. The community build is capable of absorbing a practically unlimited amount of time and effort if you let it, so it's often necessary to make cost/benefit tradeoffs between getting things right and getting them "good enough".
In an ideal world:
- every significant Lightbend and Scala Center library would be included
- every popular third-party open-source Scala library would be included
- and perhaps also some important Scala codebases that aren't themselves libraries
- every included project would be tracked from a recent yet stable branch
- every time we forked a project to make it work, we would notify the maintainers and make sure they incorporate the needed changes, so we can eliminate the fork and go back to tracking a branch
- every included project's test suite would be run
and so on. In practice, counterexamples are abundant.
Regardless, as long as we include lots of projects, with lots of code, in reasonably recent versions, we're doing pretty well. We're not trying to provide the entire Scala community with CI, we're mainly just trying to QA the Scala compiler and standard library.
Every so often, check to see if we're using the latest Scala and the latest sbt to do extractions and builds, and bump the versions if not.
To bump the Scala version, you only need to edit nightly.properties.
For sbt, use git grep to find all the places that need to be updated. The main one is in community.conf, but it's also good to use the latest version in the other places.
All projects are built from fixed SHAs.
Every so often, we move them all to new SHAs, by using the latest commit on the branch or tag specified in the first line of each proj/*.conf (and core/*.conf) file.
How often? More often is good because we catch problems earlier. Less often is good because it avoids unnecessary effort and churn. What's a good compromise? Monthly or so, probably.
To advance all the SHAs, run ./advance. There is a high likelihood something will break, so you want to do this on a branch (and probably a pull request), run Jenkins on the branch, and make sure everything is green before merging the changes.
Note that the ./advance script can also be a run for one or more more specific target names, e.g. ./advance akka playframework lagom, that kind of thing. This is useful in various circumstances, but it's especially common if you want to use a different branch or tag. Editing the URL in the comment at the top of a project's config file doesn't affect dbuild directly; you have to ./advance myproject to make the change take effect.
What typically goes wrong with the community build?
Keep an occasional (weekly?) eye on the free space numbers at https://scala-ci.typesafe.com/computer/. (You need to be logged in to Jenkins to see the numbers. Use "Launch slave agent" to fire up the behemoths if needed.)
It is normal for disk usage to creep up slowly over time, and eventually a behemoth will need cache clearing, but if we're running out of disk space more than occasionally, the root cause should be found and fixed.
Sometimes a behemoth will run out of inodes before it runs out of space. (If you weren't expecting that, it can be very puzzling indeed!) You can check inode usage with df -i.
As you'll see in the URLs referenced in proj/*.conf, we actively track current development branches of many projects, by running the advance script to move to newer SHAs. Advancing the SHAs is very likely to cause failures, so we normally make a PR for the new SHAs and make sure it's green before merging the changes.
In response, you can either:
- advance the SHA but add something to its configuration to address the problem (e.g. by disabling a subproject, changing the build settings, etc.)
- freeze at an older rev (perhaps also forking to make a needed change)
- when depending on an old rev it's better to depend on a version tag than a raw SHA
- if you fork, include in your comment what the fork point was and why; be nice to future you and future maintainers
We encourage upstream maintainers not to leave intermittently failing tests in their suites, but it happens sometimes. Try to report the problem upstream.
If you must disable tests, try to do it just for an offending subproject, if there is one, rather than the whole build.
extra.run-tests: false disables not only running tests, but also compiling them. If there are test failures, instead first try extra.test-tasks: ["compile"], so we at least compile the tests.
There are examples in the build of how to disable only individual test sources, or how to compile-but-not-run the tests in a certain subproject.
Some people's builds are a grab bag of stable vs unstable, official vs experimental stuff. If a failure can be isolated to a particular subproject, and if other people don't depend on it, you can just skip it (see examples in common.conf).
If a particular project regresses often, it may be worth contacting the project maintainer and asking (or just looking for yourself on GitHub) if there's a less volatile branch we could track. We're not trying to provide the entire Scala community with CI, after all.
e.g., they might be using -Xfatal-warnings. It's rarely worth the effort to maintain that high a standard in the context of the community build. There are examples in proj/*.conf of how to disable unwanted compiler flags.
or the build might be picky about Scala version numbers and get confused by the unusual/unexpected scalaVersion and scalaBinaryVersion values that are used in the community build.
or the build might be sensitive to sbt version. normally we use the latest sbt version to build everything, but that might fail for certain projects. proj/*.conf has examples of how to force a lower sbt version for a certain project
If a dependency in a project can't be found, perhaps it's coming from a custom resolvers and you'll need to add a resolver to the Artifactory config on scala-ci. (If you do this, add an include rule for the resolver that only even checks the resolver for artifacts matching a certain pattern, rather than checking it for **/*; otherwise you'll slow the build down.)
if you see an error message like "org.scalacheck#scalacheck from scalacheck and scalacheck, both visible in space “default”” a typical cause is that you have an upgraded to a version of a library that has added Scala.js support, and dbuild is seeing both JVM and/or JS and/or Native artifacts. usual fix: override extra.projects and/or extra.exclude, e.g. extra.exclude ["*JS", "*Native"].
It's often possible to resolve a version incompatibility by forking a project and patching its source a little bit. If that happens, try to contribute the change upstream, so your fork won't be needed anymore.
A core space (core/*.conf) contains projects that are fully-versioned, usually compiler plugins.
Then the rest of the projects are in proj/*.conf and each one gets its own space, to prevent any crosstalk between them.
Run the advance script and PR the changes it makes, and test them before merging. It's hard to
know how often makes sense, but before a Scala release seems like one time it really ought to be done.
This can really save you a lot of time. See Local runs.
https://github.com/scalacommunitybuild was created so we don't end up using forks scattered around various personal accounts. We don't want to use forks that might be lost, and we don't want to depend on commits that might disappear, either.
(In the future we might use the organization for freezing, too, not just forking. Anytime we freeze either an individual project, or freeze all of the projects at release time, there is a danger of the commit we froze at disppearing from GitHub later. If we only froze to commits stored in the scalacommunitybuild organization, that danger would go away. But in practice, we've hardly ever had trouble with this.)
They gradually become out of date, since advance doesn't advance them.
It's not your job to troubleshoot every failure yourself. Open a ticket, paste in relevant section of the error log (with a link to the full log on Jenkins), @-mention the maintainer, and ask for their help. It's their code, so they'll probably spot the problem faster, and usually maintainers are happy to be included in the community build and happy to help.
Dale and Eugene (sbt wizards), Toni (dbuild author), etc. They know more about dbuild, sbt, resolvers, Artifactory, etc than you ever will.
The community build is important, but it is rarely urgent. It only becomes urgent when a Scala release is impending.
The rest of the time, it's fine if things move slowly: because you're waiting for a project maintainer, because you're waiting for slow builds to finish, because a change needs multiple rounds of testing to get right, etc. It's common for this to drag on for days and weeks. It's fine. The community build is a moving target.
When you fix an issue, leave clues for future-you by including the error message in the commit comment. And if an error message seems mysterious, look for it in git log, a previous maintainer might have seen it before and left clues for you.
Every Jenkins run begins with extracting the dependencies of all of the projects, even if none of them need to be rebuilt. This takes a few minutes, which is too long and sit and watch, so inevitably you'll start working on something else. And actually getting feedback on the change you made might take 30 minutes, or an hour, or 10+ hours if everything has to be rebuilt! By the time you get back to it, it's easy to forget what you were even doing.
It's tempting to maintain mental flow by keeping a tight watch on your builds, but doing so is a huge waste of time. It's better to always write down everything you're thinking and doing as you go, so it's never too costly to set the community build aside while you wait for results. Get something else done, and once your results are in, if you have good notes on where you were, it won't be that hard to reconstruct your mental state and figure out what your next move is.
It's basically the same advice as "leave bread crumbs", but you're leaving the crumbs for yourself in 10 minutes or 1 hour or 1 day, not just some other sad sack months or years later.
Don't just tackle one open ticket, considering tackling 3 or 4 at once, at least if you're reasonably confident the changes are independent and won't interfere with each other. A test run can take a long time, so why not maximize the amount of information you get from each run?
Experts only! This is definitely bad advice at least some of the time.
Just because a project was in at one time doesn't mean we need to keep it forever. Keep cost/benefit in mind.
The same goes for questions of whether to run tests, what subprojects to include, and so forth.