add makefile to requirements

[badGarnet]

This PR resolves #1294 by adding a Makefile to compile requirements. This makefile respects the dependencies between file and will compile them in order. E.g., extra-*.txt will be compiled after base.txt is updated.

Test locally by simply running make pip-compile or cd requirements && make clean && make all

[cragwolfe]

why this is better than the existing solution? in the off chance that the existing make pip-compile results in inconsistent dependencies, this would get caught by https://github.com/Unstructured-IO/unstructured/blob/b710baf/.github/workflows/ci.yml#L70 .

but as long as pins are being kept in constraints.in, this really shouldn't happen in the first place.

[badGarnet]

why this is better than the existing solution? in the off chance that the existing make pip-compile results in inconsistent dependencies, this would get caught by https://github.com/Unstructured-IO/unstructured/blob/b710baf/.github/workflows/ci.yml#L70 .

but as long as pins are being kept in constraints.in, this really shouldn't happen in the first place.

The current method relies on base.txt is compiled first because of alphabetical order. The other extras that depend on base.txt are able to get updated base.txt because of naming. Additionally if we have more layered dependencies like dev.txt we may not be able to rely on the file names.

Makefile is designed to handle compilation dependency so I would argue that is the safest way to ensure pip-compile runs correctly. Now I do concede that with current requirements this may seem a bit over engineering.

[badGarnet]

probably not needed for now

[badGarnet]

reopen this PR because recently we have encountered a case where dev.in was compiled before test.txt was compiled fresh, and results in incompatible versions of dependencies that fails ci: https://github.com/Unstructured-IO/unstructured/actions/runs/6421527738/job/17436375115; https://github.com/Unstructured-IO/unstructured/actions/runs/6421723279/job/17436965312

[rbiseck3]

Got this:

❯ make pip-compile
scripts/pip-compile.sh: line 11: MAKE[1]:: command not found
cp: requirements/build.txt: No such file or directory
make: *** [pip-compile] Error 1

[rbiseck3]

Can use pushd/popd to navigate to/from the requirements dir. Has some pleasantries: Navigating with popd

[qued]

It looks like everything in the ingest subdirectory of requirements is being skipped.

[qued]

Also there used to be logic to copy the build.txt file in requirements/ into the docs/ folder overwriting docs/requirements.txt. In other words, the file build.txt is really the requirements file for sphinx builds. I think that logic is missing from what we are currently doing as well.

[badGarnet]

Also there used to be logic to copy the build.txt file in requirements/ into the docs/ folder overwriting docs/requirements.txt. In other words, the file build.txt is really the requirements file for sphinx builds. I think that logic is missing from what we are currently doing as well.

updated makefile to work on subdirectories and added back the copy line