Inconsistent performance between GitHub CI and local

[vsoch]

Hi there! We're using black for snakemake (I love it so far!) but I'm running into an issue where, despite having the same version of black installed via conda in GitHub CI versus locally, the local version reports that all is well, the GitHub CI version reports that one file needs to be reformatted.
Locally:

$ black --check snakemake tests/*.py
All done! ✨ 🍰 ✨
51 files would be left unchanged.
$ black --version
black, version 19.3b0

and in GitHub CI

Check formatting6s
##[error]Process completed with exit code 1.
Run export PATH="/usr/share/miniconda/bin:$PATH"
  export PATH="/usr/share/miniconda/bin:$PATH"
  source activate black
  black --check snakemake tests/*.py
  shell: /bin/bash -e {0}
would reformat /home/runner/work/snakemake/snakemake/snakemake/dag.py
All done! 💥 💔 💥
1 file would be reformatted, 50 files would be left unchanged.
##[error]Process completed with exit code 1.

The version of black there is also the same

 black              pkgs/main/noarch::black-19.3b0-py_0

Further, locally it tells me I need to reformat other files (which the remote doesn't suggest)

would reformat /home/vanessa/Documents/Dropbox/Code/Python/snakemake/snakemake/remote/FTP.py
would reformat /home/vanessa/Documents/Dropbox/Code/Python/snakemake/snakemake/remote/HTTP.py
Oh no! 💥 💔 💥
2 files would be reformatted, 49 files would be left unchanged.

Do you have suggestions for what we might further try, or ideas for why it's inconsistent? The testing step is here (not sure if you can see it) and the PR in question is here. Thanks!

Expected behavior A clear and concise description of what you expected to happen.

The local and remote black formatting commands would produce the same result.

Environment (please complete the following information):

• Version: black, version 19.3b0
• OS and Python version: Linux/Python 3.7.3 locally, Linux/Python 3.7.5 remote.

Does this bug also happen on master? To answer this, you have two options:

Yes I tried this, same result locally. Thanks for your help!

[olirice]

I also ran into this problem. It turned out to be a a mac (local) vs debian (CI) issue

You can identify the offending lines by spinning up a docker container that matches your ci environment, mounting your local repo, and running black in the container. Example commands below:

# start container and mount local directory in /mycode
docker run -it --rm --name black -v $(pwd):/mycode -d python:3.6 /bin/bash

# open an interactive session in the container
docker exec -it /bin/bash

# navigate to where we mounted the volume
cd mycode

# install black
pip install black

# run black
black .

That will reproduce the changes you're getting on CI. Use git diff to see what changed & adjust to something that formats the same in both environments

[vsoch]

Interesting! I have ubuntu 18.04 locally (and at least that's the runner I specified in actions) so I don't think it would (at face value) be due to differences in operating system. This is a very nice sanity check, however, I can at least use a container to get a third opinion on the matter! :)

[JelleZijlstra]

I'd be interested to know what the differences in formatting are between the two versions; that might help us figure out where the problem is.

[vsoch]

You mean OS versions? For my bug report I installed the exact same version of the library, so something else must be different. But +1, knowing version differences is useful!

[JelleZijlstra]

Sorry, I meant what the change is that Black wants to make on your CI but not your local installation.

[vsoch]

Oh that’s a good idea! Is there a command to run that would show changes for a file (I’m using —check and it just shows the file names that would be changed).

[JelleZijlstra]

--diff

[olirice]

It was

Var_name = (
“Some long strong causing wrap” # comment
)

vs

Var_name = (
“Some long strong causing wrap”
) # comment

In my case. Using pre-commit, stable, language version 3.6

Sorry for formatting, working from phone.

[brenj]

I ran into this issue as well using the system black package on Ubuntu 20.04. There were wild differences between my mac version and ubuntu in terms of which files should be reformatted. For now I just worked around it by using the pip package version which should be consistent across platforms.

Example:

pip3 install black
python3 -m black -t py38 --check tests

This might have something to do with how ubuntu builds it's version of black and not an inconsistency problem on the part of black.

[yousra-rubikloud]

I'm running into the same problem, with an additional detail that the files it's complaining about in CI were not touched in this PR.

[JelleZijlstra]

@yousra-rubikloud we just had a release, are you sure that's not the reason for the difference you're seeing? I'd recommend pinning Black to an exact version whenever you're running it in CI.

[yousra-rubikloud]

@JelleZijlstra on the CI I have black-20.8b1 and locally I have black==19.10b0. Which version was the one with the fix?

[JelleZijlstra]

I don't think there is a fix for the issue originally reported here, but different versions of Black format things differently, so you should make sure you use the same version locally and in your CI, or you're bound to run into inconsistencies.

[schinckel]

but different versions of Black format things differently

That's not ideal. I'm in the process of moving a large project to black: I am okay with having one changeset that contains all of the changes black makes, but I'm not so sure about having these every time black updates something. Is it common to have significant changes like this?

Actually, looking deeper, it seems to me that 20.8b1 does not respect the line length directive. It's reflowing lines that are >88 chars, even though I have it set to 119.

[chebee7i]

but different versions of Black format things differently

That's not ideal.

What's the alternative? Black needs to have the ability to evolve. It needs to fix things too. You can pin versions if you need fixed behavior.

[ktmeaton]

I just encountered this issue with Github Actions, and solved it by fixing my version control of black.

1. I made sure my setup.py explicitly stated black==19.10b0
2. I made my linting CI workflow also install dependencies from setup.py.
3. I was using pre-commit, and explicitly set the revision in the configuration yaml:

  - repo: https://github.com/psf/black
    rev: 19.10b0

[vsoch]

@srinathganesh did you compare the version of black on your host vs. CI? That's usually the issue - see the above thread for examples to pin to a specific version.

[srinathganesh]

@srinathganesh did you compare the version of black on your host vs. CI? That's usually the issue - see the above thread for examples to pin to a specific version.

Deleted my comment since it was kind of misleading.

My problem was two things, one is version of black and second is my CI was running alpine while host was ubuntu, python-slim did the trick

[JelleZijlstra]

@synchronizing We can make progress on this if people who run into this look more closely into why it's happening. I strongly suspect it's usually a case of different Black versions being run in different places, but I'd be happy to be proven wrong if there's a way to reproduce the issue.

[JelleZijlstra]

Closing as there is no clear way to reproduce it. Most likely people running into this are actually running different versions of Black in different places.

[igor-petrik-invitae]

I am running into this and have confirmed that I am nominally running the same version of black both locally and in CI. I will post a minimum working example, once I deal with the issues this has caused.

[JunranY]

I ran into the same issue because I thought I was using the same version. It turns out that stating the version in tag is not enough. Without specifying it using version, it defaults to the latest release. check out their doc for ci.

[domoritz]

The behavior is very unexpected. @JunranY found the solution to the issue in our repo at cmudig/draco2#199.

[ichard26]

I'm alright with making the default Black version tied to the action version being used. For context version was introduced because the action didn't exist for a long time so tying black version to action version wouldn't work for version 19.10b0 for example. In hidesight, having the default being the action version keeping the version configuration option around as an escape hatch is the better solution. This will involve some complexity since commit SHAs aren't supported by the version code (but are by GHA) but there might be some pre-existing logic in scripts/diff_shades_gha_helper.py we could reuse.

[domoritz]

That would be awesome. Thank you for proving the GitHub action and black. It's an amazing tool to have!

[lifenautjoe]

Am I correct to assume that there's still no fix for this?

Seeing wild inconsistencies between local macOS and Github Actions runner.

[JelleZijlstra]

@lifenautjoe No reproducible issue has been reported. If you are seeing this, please open a new bug with instructions to reproduce it. Most likely you are running different versions of Black in different places. Note that the version of the GH Action is independent from the version of Black being used by it (cf. https://black.readthedocs.io/en/stable/integrations/github_actions.html).

[gosuto-inzasheru]

accepting - uses: psf/[email protected] in the yaml is very confusing if it doesnt actually run in that version.

comment by @JunranY fixed it for me; have to make the version explicit on a seperate line version: "21.9b0" to actually make it run in a custom version.

We recommend the use of the @stable tag, but per version tags also exist if you prefer that. Note that the action’s version you select is independent of the version of Black the action will use.
The version of Black the action will use can be configured via version. This can be any valid version specifier or just the version number if you want an exact version. The action defaults to the latest release available on PyPI. Only versions available from PyPI are supported, so no commit SHAs or branch names.

[ichard26]

I said I'd change this, but I never even opened an issue. I'll fix that.

[gonzaloetjo]

I'm having 2 github actions, running in exactly the same way, and one fails, and the other doesn't.