CI with Github Actions and conda

[rth]

Ref #1688 (comment)

[rth]

Tests in Chrome generally work, in Firefox there still some issue with selenium setup that needs fixing.

[rth]

The latest issue here, is that with conda (without Docker), running,

make src/js/pyproxy.gen.js

produces a file with C code, instead of Javascript, that Typescript then unsurprisingly fails to parse. I can reproduce this locally as well. So,

src/js/pyproxy.gen.js : src/core/pyproxy.* src/core/*.h
        # We can't input pyproxy.js directly because CC will be unhappy about the file
        # extension. Instead cat it and have CC read from stdin.
        # -E : Only apply prepreocessor
        # -C : Leave comments alone (this allows them to be preserved in typescript
        #      definition files, rollup will strip them out)
        # -P : Don't put in macro debug info
        # -imacros pyproxy.c : include all of the macros definitions from pyproxy.c
        rm -f $@
        echo "// This file is generated by applying the C preprocessor to core/pyproxy.js" >> $@
        echo "// It uses the macros defined in core/pyproxy.c" >> $@
        echo "// Do not edit it directly!" >> $@
        cat $< | $(CC) -E -C -P -imacros src/core/pyproxy.c $(MAIN_MODULE_CFLAGS) -Isrc/core/ - >> $@

is not working as expected there.

[ryanking13]

it's currently impossible to restart a single job (Add ability to re-run single jobs actions/runner#432, [2]). Which because jobs still occasionally fail, I think, is a deal breaker for us.

Seems like this will be resolved at 2021 Q4: github/roadmap#271

[rth]

This should be ready for a review. So far this adds a Linux setup in CI to build core + numpy outside of Docker with conda and corresponding tests in Firefox. The setup is very similar to the one we have in CI. The only change outside of CI is

Even though it's a bit redundant the goal is to make sure,

• we can build outside of Docker. Well there are various flavors of Linux but this adds another example of setup in addition to docker.
• also when CI fails erratically in CircleCI this would give another reference point to verify if it's due to CircleCI or code changes.
• I'll open a follow up PR to build for MacOS, ideally mostly with the same setup.

The motivation for conda is to be able to install host python of the required version, other non python dependencies with reasonably recent versions (e.g. patch on MacOS) and be somewhat cross-platform in terms of how dependencies are installed. Also ref #795
I'll update the docs once MacOS build works.

@ryanking13 Would you be interested in reviewing?

[ryanking13]

@ryanking13 Would you be interested in reviewing?

Yes I am!

[ryanking13]

Thanks for your effort! My comments are:

[ryanking13]

How about matching retention-days of core-build and core-build-all?
And also, how about keeping them for a longer period? It looks like we can keep them in maximum 90 days with no disadvantage.

[rth]

Actually I just realized that we don't really need core-build-all: it's a left over from a previous config where there were 2 build steps. For debugging we can add other smaller artifacts (e.g. package logs) in the future but for now I'll just remove core-build-all since it's quite slow.

Also changed core-build retention duration to 60 day following your suggestion. We could put to 90, but there is likely also no need, and it does use storage somewhere, even if we don't pay for it. As far as I remember it's 30 days in CircleCI, and we rarely have an issue with it.

[ryanking13]

I think we can remove many files in emsdk to reduce archive size, which are not actually build outputs.

For example:

• emsdk/emsdk/upstream/emscripten/node_modules
• emsdk/emsdk/node

[rth]

Indeed. For now I removed all of core-build-all.

[ryanking13]

It looks like if we build numpy but not build joblib, the test fails at test_joblib.py

I don't think invoking tests in test_joblib.py are what we expect in this case. Maybe some parameter is misused in pytest?

[rth]

Indeed. Well the test name test_joblib_numpy_pickle in test_numpy.py is OK, but my pytest selection query was not specific enough. Fixed it.

[rth]

Thanks a lot for the review @ryanking13!