Complete pip-tools setup

[greshilov]

What do these changes do?

Solve problems with inconsistent approaches to installing dependencies in Makefile.

Are there changes in behavior for the user?

No.

Related issue number

Related PR: #5470.

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes
• If you provide code modification, please add yourself to CONTRIBUTORS.txt
  • The format is <Name> <Surname>.
  • Please keep alphabetical order, the file is sorted by names.
• Add a new news fragment into the CHANGES folder
  • name it <issue_id>.<type> for example (588.bugfix)
  • if you don't have an issue_id change it to the pr id after creating the pr
  • ensure type is one of the following:
    • .feature: Signifying a new feature.
    • .bugfix: Signifying a bug fix.
    • .doc: Signifying a documentation improvement.
    • .removal: Signifying a deprecation or removal of public API.
    • .misc: A ticket has been closed, but it is not of interest to users.
  • Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

[greshilov]

@webknjaz I've marked this PR with WIP because I'm not sure about pip-sync usage. It seemed logical to use it since we moved to pip-compile, what do you think?

[codecov]

Codecov Report

Merging #5486 (7cbe6fd) into master (8c82ba1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5486   +/-   ##
=======================================
  Coverage   97.17%   97.17%           
=======================================
  Files          41       41           
  Lines        8857     8857           
  Branches     1424     1424           
=======================================
  Hits         8607     8607           
  Misses        133      133           
  Partials      117      117           

Flag	Coverage Δ
unit	97.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c82ba1...7cbe6fd. Read the comment docs.

[greshilov]

Looks like some tests are failing because of new warnings introduced in 77d0366.

[webknjaz]

I don't use pip-sync and it's a good idea to make sure that this all works with vanilla pip.

Also, for dependabot to work properly, we need to turn all the direct dependency files into .in. The pip-tools setup is incomplete and it needs polishing first before we even consider integrating extra things.
Take a look at pypa/warehouse — they have a good example of the file pairs for dependabot.

[greshilov]

Summarizing the changes:

1. Files doc-spelling,lint,cython,dev are split into in,txt pairs. All of them are used as a direct requirement somewhere (lint in workflow script e.t.c.).
   Note the order, lint file must be always compiled before dev.

2. Unfortunately we can't use -r dev.in -c dev.txt anymore, because of the new pip resolver implementation. That's not a problem, I guess. pip-compile is designed to produce ready-to-go .txt dependencies.

3. I added the .update-pip step before all installation steps.

[webknjaz]

• Unfortunately we can't use -r dev.in -c dev.txt anymore, because of the new pip resolver implementation. That's not a problem, I guess. pip-compile is designed to produce ready-to-go .txt dependencies.

Why is that? What exactly causes problems?

[greshilov]

• Unfortunately we can't use -r dev.in -c dev.txt anymore, because of the new pip resolver implementation. That's not a problem, I guess. pip-compile is designed to produce ready-to-go .txt dependencies.

Why is that? What exactly causes problems?

We have a requirement pyjwt[crypto]==2.0.0, which is declared with the extra dependency [crypto]. pip-compile carefully includes this requirement in dev.txt, but new pip resolver rules strictly prohibit any extras in a constraint file.

Generally speaking any dependency with extra will break the approach with -r dev.in -c dev.txt :(
Read more about the reasons behind this behavior here.

[webknjaz]

@greshilov sounds like pip-tools should be fixed to generate a proper constraints file...

[webknjaz]

Meanwhile, we'll have to use -r req.txt syntax and probably generate separate requirements for different envs

[greshilov]

Meanwhile, we'll have to use -r req.txt syntax and probably generate separate requirements for different envs

This PR is alright for now then? I also wanted to suggest adding hashes to lock files, like in pypa/warehouse and mozilla/ichnaea.

[webknjaz]

This PR is alright for now then?

Alright. Let's merge it then, but we still need to raise this issue in the pip-tools repo, I think.
Also, I suppose you'll backport this, right?

I also wanted to suggest adding hashes to lock files, like in pypa/warehouse and mozilla/ichnaea.

That's a great idea! I do this in all of my projects by default.

[webknjaz]

Urgh... This is not a bugfix but a misc change because it's not a user-facing fix but something that would only affect the core devs and contributors.

[greshilov]

Sorry, my bad!

[webknjaz]

It's not a big deal, I guess this can be fixed by a follow-up PR with git mv.