Replace black and blackdoc with ruff-format #9506

Armavica · 2024-09-17T01:15:27Z

Since ruff is already used for linting I figured we could also use it for formatting.
This allows to remove black as well as blackdoc which feels relatively slow on my machine.

Closes #xxxx
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

Armavica · 2024-09-17T01:24:18Z

Sorry, I didn't see the earlier attempt #8761.
It might still be interesting to see what changed in 7 months.

Armavica · 2024-09-17T02:08:48Z

It looks like the dust settled and the diff is much smaller now (+40 -71 compared to +513 -291 in #8761). But perhaps I am missing something?
In any case, please feel free to close my PR in favor of updating yours @etienneschalk.

max-sixty · 2024-09-25T23:04:04Z

Would this make sense to merge?

@etienneschalk if you see this and prefer to update #8761, let us know, otherwise I'm +1 to merge this in a couple of days...

max-sixty · 2024-09-25T23:06:39Z

xarray/core/dataarray.py

@@ -5558,7 +5558,6 @@ def map_blocks(
        ...     gb = da.groupby(groupby_type)
        ...     clim = gb.mean(dim="time")
        ...     return gb - clim
-        ...
        >>> time = xr.cftime_range("1990-01", "1992-01", freq="ME")


FWIW I really dislike this change, and find it makes the doctests much harder to scan. But a) I think it's maybe just me?, b) it's not enough to stop moving to ruff.

Unless there's some way to disable this? Even if we disabled doctest formatting in favor of retaining blackdoc (which doesn't have the same performance requirements as formatting the 7000 line python files)

I think it's maybe just me?

You're not alone. The sad thing is, this change is not consistent: we have a bunch of these in xarray.core.computation as well, which have not been removed (not sure why, though).

I'd be cautious about selectively enabling ruff format without docstrings: blackdoc relies on black for the formatting, so whenever ruff is (inevitably) inconsistent, we'll get inconsistent formatting between doctests and the normal code.

we'll get inconsistent formatting between doctests and the normal code.

Is this bad though? The differences are so so small, and if we can disable ruff from the docstrings then they're not going to step on each other?

probably not too much

OK, @Armavica if you're up for moving back to blackdoc for the docstrings, and reverting the docstring changes, then I'm +1 on merging; I'll merge unless anyone has objections

looks like ruff format does not format the docstring of apply_ufunc, at all. Not sure why that is, other functions within xarray.core.computation are formatted just fine.

If we're truly keeping blackdoc it might make sense to drop the pin for black (with all the reproducibility issues that would bring): there's no way to sync it with the black hook anymore, so we'd have to upgrade manually.

Ok, if I understand correctly the reason(s) for not using ruff on the docstrings are that:

we would rather it kept the empty ... after a function definition

it's actually inconsistent on this issue

Is that right? If so, would you like me to create a new issue or PR to record this, and eventually come back to it when ruff improves on this front?

yes to the first point, we'll probably have to ask the ruff team to add an option for that, as I don't see a lot of other projects that prefer the trailing ... after blocks – but I didn't ask around, so who knows. When they initially discussed this they seemed open to an option, so it shouldn't be too hard to convince them.

For the second point we'll probably have to figure out why ruff does not format the docstring of apply_ufunc, at all. blackdoc formats this correctly, so it can't be caused by a SyntaxError.

And to set the stakes: having ruff format the docstrings rather than blackdoc doesn't help us much — the main advantage of ruff is its speed vs. black, so saving a huge file doesn't take a few seconds. blackdoc doesn't have that much code to format and doesn't run on every save, so it's fine if it's slower.

Removing it would reduce one config, but that's fine to hold onto for a while.

headtr1ck

I personally like ruff a lot.

headtr1ck · 2024-10-09T08:28:41Z

pyproject.toml

@@ -239,9 +239,12 @@ extend-exclude = [
  "_typed_ops.pyi",
 ]

+[tool.ruff.format]
+docstring-code-format = true


That's not bad actually, nice

headtr1ck · 2024-10-09T08:29:21Z

.pre-commit-config.yaml

-  - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 24.8.0
-    hooks:
-      - id: black-jupyter


Is there something similar to Jupyter formatting in ruff as well?

I think that can be done using

[tool.ruff] # also check notebooks extend-include = ["*.ipynb"]

I think it is done by default: https://docs.astral.sh/ruff/configuration/#jupyter-notebook-discovery

headtr1ck · 2024-10-09T08:31:34Z

.pre-commit-config.yaml

-    rev: v0.3.9
-    hooks:
-      - id: blackdoc
-        exclude: "generate_aggregations.py"


We should probably add the generation scripts as exceptions to ruff as well (in the pyproject.toml?)

headtr1ck · 2024-10-09T08:35:23Z

pyproject.toml

 [tool.ruff.lint]
 # E402: module level import not at top of file
-# E501: line too long - let black worry about that
+# E501: line too long - let the formatter worry about that


Maybe this is obsolete when using ruff as a formatter as well?

It is not, ruff emits these warnings too

max-sixty · 2024-10-09T18:29:01Z

Would vote to exclude docstrings, restore blackdoc, and merge!

Armavica · 2024-10-16T09:33:37Z

Would vote to exclude docstrings, restore blackdoc

I just did this.

max-sixty · 2024-10-16T17:35:11Z

Great! Couple of merge conflicts and then will plan to merge!

* main: Fix multiple grouping with missing groups (pydata#9650) flox: Properly propagate multiindex (pydata#9649) Update Datatree html repr to indicate inheritance (pydata#9633) Re-implement map_over_datasets using group_subtrees (pydata#9636) fix zarr intersphinx (pydata#9652) Replace black and blackdoc with ruff-format (pydata#9506) Fix error and missing code cell in io.rst (pydata#9641) Support alternative names for the root node in DataTree.from_dict (pydata#9638) Updates to DataTree.equals and DataTree.identical (pydata#9627) DOC: Clarify error message in open_dataarray (pydata#9637) Add zip_subtrees for paired iteration over DataTrees (pydata#9623) Type check datatree tests (pydata#9632) Add missing `memo` argument to DataTree.__deepcopy__ (pydata#9631) Bug fixes for DataTree indexing and aggregation (pydata#9626) Add inherit=False option to DataTree.copy() (pydata#9628) docs(groupby): mention deprecation of `squeeze` kwarg (pydata#9625) Migration guide for users of old datatree repo (pydata#9598) Reimplement Datatree typed ops (pydata#9619)

* main: (63 commits) Add close() method to DataTree and use it to clean-up open files in tests (pydata#9651) Change URL for pydap test (pydata#9655) Fix multiple grouping with missing groups (pydata#9650) flox: Properly propagate multiindex (pydata#9649) Update Datatree html repr to indicate inheritance (pydata#9633) Re-implement map_over_datasets using group_subtrees (pydata#9636) fix zarr intersphinx (pydata#9652) Replace black and blackdoc with ruff-format (pydata#9506) Fix error and missing code cell in io.rst (pydata#9641) Support alternative names for the root node in DataTree.from_dict (pydata#9638) Updates to DataTree.equals and DataTree.identical (pydata#9627) DOC: Clarify error message in open_dataarray (pydata#9637) Add zip_subtrees for paired iteration over DataTrees (pydata#9623) Type check datatree tests (pydata#9632) Add missing `memo` argument to DataTree.__deepcopy__ (pydata#9631) Bug fixes for DataTree indexing and aggregation (pydata#9626) Add inherit=False option to DataTree.copy() (pydata#9628) docs(groupby): mention deprecation of `squeeze` kwarg (pydata#9625) Migration guide for users of old datatree repo (pydata#9598) Reimplement Datatree typed ops (pydata#9619) ...

Armavica force-pushed the ruff-format branch from f98980c to 07ddb58 Compare September 17, 2024 01:50

max-sixty reviewed Sep 25, 2024

View reviewed changes

Armavica force-pushed the ruff-format branch 3 times, most recently from 1b8ee42 to b4f8aa3 Compare September 27, 2024 15:24

headtr1ck approved these changes Oct 9, 2024

View reviewed changes

headtr1ck added the Automation Github bots, testing workflows, release automation label Oct 9, 2024

Armavica force-pushed the ruff-format branch 2 times, most recently from 8fd3798 to a97f313 Compare October 16, 2024 09:31

max-sixty added the plan to merge Final call for comments label Oct 16, 2024

Armavica added 3 commits October 17, 2024 19:33

Replace black with ruff-format

398bdc2

Fix formatting mistakes moving mypy comments

07d8f96

Replace black with ruff in the contributing guides

32d55fe

Armavica force-pushed the ruff-format branch from a97f313 to 32d55fe Compare October 17, 2024 17:34

max-sixty enabled auto-merge (squash) October 17, 2024 17:41

max-sixty merged commit b9780e7 into pydata:main Oct 17, 2024
28 checks passed

Armavica deleted the ruff-format branch October 18, 2024 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace black and blackdoc with ruff-format #9506

Replace black and blackdoc with ruff-format #9506

Armavica commented Sep 17, 2024 •

edited

Loading

Armavica commented Sep 17, 2024

Armavica commented Sep 17, 2024

max-sixty commented Sep 25, 2024

max-sixty Sep 25, 2024

keewis Sep 25, 2024

max-sixty Sep 25, 2024

keewis Sep 27, 2024

max-sixty Sep 27, 2024

keewis Sep 27, 2024

Armavica Sep 27, 2024

keewis Sep 27, 2024 •

edited

Loading

max-sixty Sep 27, 2024 •

edited

Loading

headtr1ck left a comment

headtr1ck Oct 9, 2024

headtr1ck Oct 9, 2024

mathause Oct 10, 2024

Armavica Oct 16, 2024

headtr1ck Oct 9, 2024 •

edited

Loading

headtr1ck Oct 9, 2024

Armavica Oct 16, 2024

max-sixty commented Oct 9, 2024

Armavica commented Oct 16, 2024

max-sixty commented Oct 16, 2024

Replace black and blackdoc with ruff-format #9506

Replace black and blackdoc with ruff-format #9506

Conversation

Armavica commented Sep 17, 2024 • edited Loading

Armavica commented Sep 17, 2024

Armavica commented Sep 17, 2024

max-sixty commented Sep 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keewis Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

max-sixty Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

headtr1ck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

headtr1ck Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

max-sixty commented Oct 9, 2024

Armavica commented Oct 16, 2024

max-sixty commented Oct 16, 2024

Armavica commented Sep 17, 2024 •

edited

Loading

keewis Sep 27, 2024 •

edited

Loading

max-sixty Sep 27, 2024 •

edited

Loading

headtr1ck Oct 9, 2024 •

edited

Loading