How to deal with CI

How to deal with CI

The CI (Continuous Integration, aka. checks) results can be seen at the bottom of each PRs by clicking to the "details" link. Each bullet point is a different log that you can read. The error(s) encountered are usually around the end of the log.

To reproduce a test, the commands to call can be seen at the beginning of each log files. This requires docker for Linux. Not that for FreeBSD and macOS the Dockerfile is only an approximation as those platform can't run as a guest on Docker.

The CI currently does:

• a lint check on the opam file
• for each packages added or modified in a significant way, check if the package builds and its tests run correctly:
  • on all reasonably recent OCaml compilers
  • on different Linux distributions, FreeBSD and macOS
  • on different architectures (arm64, ppc64le, s390x, x86_32, arm32)
  • with different versions of opam (2.0, 2.1, 2.2)
  • with different options unabled in the compiler (flambda, frame-pointer, bytecode-only, naked-pointers, …)
• check if each packages builds with the oldest versions of the dependencies available

When modifying older packages, errors that existed before the PR can be ignored if the cause of the error stay the same. When adding new packages though, ideally the goal should be to have everything succeed. However some errors might occur that are to be ignored for the moment, such as:

• missing system binaries from a distribution or missing system configuration. Using x-ci-accept-failures (see annex for more details) can help ignore those failures if the targetted platforms require something that can be dealt with manually but not in CI. Using the available field can also help if a certain platform will never work (e.g. uses x86_64 assembly, …)

Unexpected errors can occur. Such errors are inherently unpredictible but a couple of examples can be:

• temporary network failures or broken build machine. If this happens, please open a ticket at ocaml/infrastructure
• packages that were broken before and went unnoticed. Contributions are most welcome for this type of errors.

If you're noticing errors that look geniuine but never noticed before on your own CI it could be because:

• your CI doesn't test the oldest versions for the dependencies (aka. lower-bounds) of your packages (check the FAQ to know how to check this locally)
• your CI doesn't test building only one packages at a time. This is common for dune users who have several packages in one repository. The typical dune workflow is to use dune build. However testing dune build -p usually reveals issues in the assignment of dune rules (by default rules belong to every packages unless (package ...) is given). Typical error will occur in tests where a test will be ran in all packages but dependencies become cycles.
• we're testing more platforms (freebsd, macos, s390x, …). If you do not have access to that platform or access to a service providing this platform, don't hesitate to ask the opam-repository maintainers for help.
• we have a different opam setup: opam-repo-ci has the sandbox unable for both build and tests, which might not be the case in your own CI (e.g. if you’re using docker). Errors are typical here if you’re using the network (e.g. during the tests)

If you have a feature request for the CI, feel free to open a ticket or a PR at ocurrent/opam-repo-ci

Annex

• x-ci-accept-failures is a custom extra field used in opam-repository. The syntax for this field is the following: x-ci-accept-failures: [ "<platform>" ... ], where <platform> is a certain platform tested by opam-repo-ci. The list of platforms is found under "Operating Systems" in the generated documentation for tested platforms. You can also find a platform you want to ignore from the logs of a CI job, by looking for lines similar to if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"opensuse-tumbleweed\"". In this case, opensuse-tumbleweed` is the platform.