Problem: compile-requirements doesn't work in Docker makefile

[ross-spencer]

Expected behaviour

The Archivematica development environment makes it as easy as possible for developers to complete their tasks.

Current behaviour

$ make compile-requirements will return an error with the pip-compile command, and there are related issues discussed in #765 as well.

The error snapshot looks as follows:

docker-compose run --workdir /src/MCPServer/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm  --no-deps --user=root --entrypoint make archivematica-mcp-server all
pip-compile -v  --output-file base.txt base.in
Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 5, in <module>
    from piptools.scripts.compile import cli
  File "/usr/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 11, in <module>
    from .._compat import install_req_from_line, parse_requirements
  File "/usr/local/lib/python2.7/site-packages/piptools/_compat/__init__.py", line 7, in <module>
    from .pip_compat import (
  File "/usr/local/lib/python2.7/site-packages/piptools/_compat/pip_compat.py", line 32, in <module>
    is_file_url = do_import("download", "is_file_url")
  File "/usr/local/lib/python2.7/site-packages/piptools/_compat/pip_compat.py", line 24, in do_import
    return getattr(imported, package)
AttributeError: 'module' object has no attribute 'is_file_url'
Makefile:11: recipe for target 'base.txt' failed
make: *** [base.txt] Error 1
Makefile:115: recipe for target 'compile-requirements' failed
make: *** [compile-requirements] Error 2

Steps to reproduce

As above.

Your environment (version of Archivematica, operating system, other relevant details)

Docker-compose => qa/1.x

Workaround

• Exec into the container that you're updating the requirements for.
• Run make all from the requirements folder for the src.
• Exit the container and assign 0644 permissions to the user on all the requirements files. See: comment.

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged (if applicable)
• Details about this issue have been added to the release notes (if applicable)

[ross-spencer]

NB. pip tools is pinned at 3.7.0 and at least 4.3.0 is available and resolves the issue. To update this globally requires updating the requirements files in the Archivematica and Storage Service projects.

[ross-spencer]

Related to #1025

[ross-spencer]

• Related note for compiling messages: here.
• Internal Slack: here.

[replaceafill]

I verified the make compile-requirements rule using fresh containers on artefactual-labs/am@28b2790 and found a few problems:

• The MCPClient command is broken because the base docker image install the newest pip release. In my case this is 20.0.2. Support for pip >= 20.0 was added to pip-tools until version 4.4.

• The SS command is broken because pip-tools was only updated in local.in and not in test.in which is what the Dockerfile uses.

• These problems are hard to spot because the make all rule in each requirements directory will do nothing if the *.txt files have already been built saying that there's nothing to do:

  replaceafill@eluk:~/dev/archivematica/am/compose$ make compile-requirements 
  docker-compose run --workdir /src/MCPServer/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm  --no-deps --user=root --entrypoint make archivematica-mcp-server all
  make: Nothing to be done for 'all'.
  docker-compose run --workdir /src/MCPClient/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm --no-deps --user=root --entrypoint make archivematica-mcp-client all
  make: Nothing to be done for 'all'.
  docker-compose run --workdir /src/dashboard/src/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm --no-deps --user=root --entrypoint make archivematica-dashboard all
  make: Nothing to be done for 'all'.
  docker-compose run --workdir /src/requirements --volume "tmpfs:/pip-cache:rw" -e XDG_CACHE_HOME=/pip-cache --rm  --no-deps --user=root --entrypoint make archivematica-storage-service all
  make: Nothing to be done for 'all'.

  Since these *.txt files are version controlled this is almost always the case. An alternative would be to include make clean calls in the compile-requirements rule for each component before the make all call.

[replaceafill]

We decided to tackle this after the 1.11 release.

[ross-spencer]

There are some notes that will help guide this here: #984 (comment) the current PR, uses sudo which we'd like to avoid. I do have a preference toward compile-am-requirements and compile-ss-requirements as recipes. As well as the potential to run clean before each run as well as Douglas noted. I think we're almost there with this part of Archivematica and understanding how it works during development. E.g. compared to when I first wrote this issue #765 and a few mishaps along the way, e.g. when we haven't compiled requirements for a new feature that needed it, cascading changes (confusingly!) to the next PR down the road.

[jraddaoui]

@replaceafill just pointed me to this issue, I've partially addressed it in:

artefactual/archivematica#1605
artefactual/archivematica-storage-service#531
artefactual-labs/am#94

I didn't notice that the SS local and test requirements don't extend each other and that there is another pip-tools in the test requirements that has not been upgraded.

[tw4l]

Following @ross-spencer's recent work in #1253 and PR artefactual/archivematica-storage-service#537, I had expected this issue to be resolved (not being aware of some of the discussion above in this issue), but I'm still unable to build the Storage Service dependencies using the Docker Compose Makefile command compile-requirements. I think this is because Docker is using the test requirements rather than local, and as per @jraddaoui's comment above, the test requirements don't extend local and are still pinned to pip-tools==3.7.0.

[ross-spencer]

@tw4l #1258 is a different issue I'm afraid, so I wasn't expecting this to be resolved via that.

My workaround is always to:

1. exec into the storage service container: sudo docker-compose exec --user=root archivematica-storage-service bash
2. navigate to cd ../requirements
3. update pip-tools pip install -U pip-tools
4. run make

It looks like we have a milestone on this issue for 1.12.0. I can look at this and complete the trail of consciousness on this issue at some point soon. Hopefully the workaround is okay for you, Note: You'll need to change the permissions before adding and commuting the changes

It was just unfortunate timing the way this wasn't completed the first time I tried to fix this all up.

[tw4l]

@ross-spencer Agreed, and to be clear I don't think that #1258 should have fixed this! Perhaps I should have said "I naively expected this issue to be resolved" 😆

In the meantime I've used a similar workaround to what you've described to get around this so I don't mean to describe this as a blocker for me, but it would be great to get it fixed properly, and I'm glad to see the 1.12 milestone.

[sevein]

Both compile-requirements-am and compile-requirements-ss work nicely!