CICD: build does not work because of dependency conflicts

[tloubrieu-jpl]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

For a few week the github action do not work anymore because of dependency conflict issues.

However, the project builts properly in a new virtual environment.

🕵️ Expected behavior

I expected the github actions to run successfully in less than 30 minutes.

📜 To Reproduce

No response

🖥 Environment Info

No response

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

This ticket and NASA-PDS/roundup-action#109 are both a result of not using a new virtual environment anymore in the github actions.

[nutjob4life]

@tloubrieu-jpl please accept my apologies. I read the title:

CICD: build does not work because of dependency conflicts

and fixed that. But I didn't notice the acceptance criterion:

I expected the github actions to run successfully in less than 30 minutes.

I shall get to work on that.

[nutjob4life]

Okay, a lot to unpack here. First, even if "the project builts properly in a new virtual environment", that's no guarantee it'll work in GitHub Actions. The two environments are vastly different: different versions of Python 3.9 and package dependencies, different operating systems (macOS vs Linux vs Windows), and now thanks to the rise of Apple Silicon, even different CPUs.

What we strive to do is compromise between the CI/CD environment and developers' desktops. That's what github-actions-base really encapsulates: a compromise between the desired development context and what GitHub Actions can reasonably provide for free.

So, executive summary: this ticket cannot be resolved.

Read on for the gory details.

Package Dependencies

The doi-service has three problematic dependencies:

• lxml >= 4.5
• numpy >= 1.18
• pandas ~= 1.4.4

These all slow down the build because these packages in particular take a long time to construct. By changing these dependencies to:

• lxml == 4.6.3
• numpy == 1.21.2

we satisfy the the previous setup.cfg requirements while taking advantage of these packages that are present in github-actions-base.

It also has two other speed-bumps:

• tox without a version pin
• virtualenv as an implied dependency but also without a version pin

Adding these pins:

• tox == 3.28.0
• virtualenv == 20.8.1

we can further speed up the build.

That brings us to pandas. According to this issue, the doi-service needs pandas in the 1.4.4+ version range. The github-actions-base provides pandas == 1.3.4. We've got two ways to speed this up:

1. Upgrade github-actions-base to pandas == 1.4.4 (the latest in the 1.4.x range). However, this would require us to check every other Python package within the PDS-EN that's using Pandas to see if this version is compatible. Maybe this is okay, though? I haven't looked extensively, but it seems maybe only doi-service is using pandas right now.
2. Downgrade doi-service to pandas == 1.3.4. I ran pytest using both pandas-1.4.4 and pandas-1.3.4 and got the same number of failures, errors, warnings, and passes, so maybe this is okay.

I'd like @jimmie to chime in as he originally submitted #369 and @alexdunnjpl who provided the fix to it. What's important about 1.4? Is 1.3 okay?

Code Issues

Once we address the above, there are lingering issues in the main branch even today. I guess this is what we get for not having a routine Roundup running here!

Specifically, there were lint errors in these files: update.py, input_util.py, config_parser.py, contributors_util_test.py, doi_validator_tes.py.

And in util.py there was

from collections import Iterable

which doesn't exist; it should be

from typing import Iterable

After fixing all these, we can finally turn it over to the Roundup.

The Speed Limit

With all this in place, the free GitHub Actions runner can complete the Roundup in about 45 minutes, which exceeds the acceptance criterion for this ticket.

We could put even more of doi-service dependencies into the github-actions-base to try and speed things up. Aside from lxml, numpy, and pandas, doi-service has 32 other dependencies, from flask==1.1.2 to jinja2==3.0.1 (a troublemaker in the past) to requests>=2.23 and more. Putting those 32 dependencies into github-actions-base would speed up the build by perhaps 7–9 minutes, which puts the total build time to 36–38 minutes, which still doesn't satisfy the acceptance criterion.

However, doing so means it freezes the versions of 36 packages for the entire NASA-PDS organization. Other packages could still supply newer versions, such as requests~=2.26, and that's okay—they'd lose the speed bonus of having a pre-cached package. Yet I'm still hesitant to put so many more sundry things into github-actions-base. Something just feels off about it.

The other time sink is the sheer size of the doi-service: it's 8600+ lines of Python and has an enormous test suite. Tox'ing, lint'ing, mypy-type-checking, flake8'ing, white-space-checking, shebanging, debug statement detection, black'ifying, etc., all takes a lot of time—and that's before we even get to the unit tests.

The only way I've found so far is to use a self-hosted GitHub Actions runner. Using such a runner I believe I can get 10 minutes or less.

Summary

Sorry, @tloubrieu-jpl: cannot fix without self-hosted runner.

[tloubrieu-jpl]

Let's downgrade pandas to 1.3.4