Enable integration with pre-commit (#698)

* Add .pre-commit-hooks.yaml * Use files instead of types * Rename main hook to jupytext * Remove extra hooks * Add default --from * Update docs * Add --add-untracked for pre-commit compatibility * Flip boolean * Update docs * Exit nonzero if a new file is added to the index with --add-untracked A pre-commit hook should fail if anything in the index is modified. By adding the new untracked file ourselves, we circumvent pre-commit, which means it won't have the chance to run other configured hooks against the new file. Exiting non-zero in this case ensures that the commit will fail, forcing the user to commit again. This time, the new file will be in the index from the start, so other hooks will run against it. * Remove accidental changes * Remove accidental changes * Clarify help * Strip for safety * Add tests for --add-untracked * Fix pre-commit hooks * Fix tests for windows * Add tests for pre-commit integration * Run pre-commit tests in ci * Retry running pre-commit tests in ci * Refactor to better align with pre-commit * Update tests for precommit * Update hook description Co-authored-by: Aaron Gokaslan <[email protected]> * Log output files that are untracked * Update examples * Fixed examples pre-commit documentation * Fix trailing whitespace * Ensure hooks runs in serial Prevents race condition during sync * Fix typo * Clarify log message * Convert test to --sync * args should be array * Ignore unmatched inputs also when using --sync * add notebook types following https://github.com/pre-commit/identify/blob/master/identify/extensions.py * Test & fix 'alert_untracked' in the --sync mode * Add .pre-commit-hooks.yaml * Use files instead of types * Rename main hook to jupytext * Remove extra hooks * Add default --from * Update docs * Add --add-untracked for pre-commit compatibility * Flip boolean * Update docs * Exit nonzero if a new file is added to the index with --add-untracked A pre-commit hook should fail if anything in the index is modified. By adding the new untracked file ourselves, we circumvent pre-commit, which means it won't have the chance to run other configured hooks against the new file. Exiting non-zero in this case ensures that the commit will fail, forcing the user to commit again. This time, the new file will be in the index from the start, so other hooks will run against it. * Remove accidental changes * Remove accidental changes * Clarify help * Strip for safety * Add tests for --add-untracked * Fix pre-commit hooks * Fix tests for windows * Add tests for pre-commit integration * Run pre-commit tests in ci * Retry running pre-commit tests in ci * Refactor to better align with pre-commit * Update tests for precommit * Update hook description Co-authored-by: Aaron Gokaslan <[email protected]> * Log output files that are untracked * Update examples * Fixed examples pre-commit documentation * Fix trailing whitespace * Ensure hooks runs in serial Prevents race condition during sync * Fix typo * Clarify log message * Convert test to --sync * args should be array * Remove types * Try calling pre_comit directly * When in doubt, return False * Fix exisiting file test * Only update the timestamp (not the content) of the text input file * Only run on notebooks * use arrays * Only run on notebooks * Handle pre-commit not being installed Relevant tests will be skipped in that case * Handle right exception Co-authored-by: Aaron Gokaslan <[email protected]> Co-authored-by: Aaron Gokaslan <[email protected]> Co-authored-by: Marc Wouts <[email protected]>
mwouts · Jan 21, 2021 · 41b7e9c · 41b7e9c
1 parent d2898da
commit 41b7e9c
Show file tree

Hide file tree

Showing 8 changed files with 334 additions and 47 deletions.
diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml
@@ -0,0 +1,8 @@
+- id: jupytext
+  name: jupytext
+  description: Runs jupytext on all notebooks and paired files.
+  language: python
+  entry: jupytext --ignore-unmatched --alert-untracked
+  pass_filenames: true
+  require_serial: true
+  types: [jupyter]
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -5,13 +5,14 @@ Jupytext ChangeLog
 ---------------
 
 **Changed**
-- Jupytext does not work properly with the new cell ids of the version 4.5 of `nbformat>=5.1.0` yet, so we added the requirement `nbformat<=5.0.8` (#715)
+- Jupytext does not work properly with the new cell ids of the version 4.5 of `nbformat>=5.1.0` yet, so we added the requirement `nbformat<=5.0.8` ([#715](https://github.com/mwouts/jupytext/issues/715))
+- `jupytext --sync` only updates the timestamp of the text file (not the file itself) when that file is the most recent ([#698](https://github.com/mwouts/jupytext/issues/698))
 
 **Fixed**
-- Indented magic commands are supported (#694)
+- Indented magic commands are supported ([#694](https://github.com/mwouts/jupytext/issues/694))
 
 **Added**
-- Added a test that ensures that `py:percent` scripts end with exactly one blank line (#682)
+- Added a test that ensures that `py:percent` scripts end with exactly one blank line ([#682](https://github.com/mwouts/jupytext/issues/682))
 
 
 1.9.1 (2021-01-06)

diff --git a/docs/using-pre-commit.md b/docs/using-pre-commit.md
@@ -27,35 +27,43 @@ Note that these hooks do not update the `.ipynb` notebook when you pull. Make su
 
 ## Using Jupytext with the pre-commit package manager
 
-Using Jupytext with the [pre-commit package manager](https://pre-commit.com/) is another option. You could add the following to your `.pre-commit-config.yaml` file:
-```
+Using Jupytext with the [pre-commit package manager](https://pre-commit.com/) is another option. You could add the following to your `.pre-commit-config.yaml` file to sync all staged notebooks:
+
+```yaml
 repos:
--   repo: local
+-   repo: https://github.com/mwouts/jupytext
+    rev: #CURRENT_TAG/COMMIT_HASH
     hooks:
     - id: jupytext
-      name: jupytext
-      entry: jupytext --to md
-      files: .ipynb
-      language: python
+      args: [--sync]
 ```
 
-Here is another `.pre-commit-config.yaml` example that uses the --pre-commit mode of Jupytext to convert all `.ipynb` notebooks to `py:light` representation and unstage the `.ipynb` files before committing.
+You can provide almost all command line arguments to Jupytext in pre-commit, for example to produce several kinds of output files:
+
+```yaml
+repos:
+-   repo: https://github.com/mwouts/jupytext
+    rev: #CURRENT_TAG/COMMIT_HASH
+    hooks:
+    - id: jupytext
+      args: [--from, ipynb, --to, py:light, --to, markdown]
 ```
+
+If you are combining Jupytext with other pre-commit hooks, you must ensure that all hooks will pass on any files you generate. For example, if you have a hook for using `black` to format all your python code, then you should use Jupytext's `--pipe` option to also format newly generated Python scripts before writing them:
+
+```yaml
 repos:
-  -
-    repo: local
+-   repo: https://github.com/mwouts/jupytext
+    rev: #CURRENT_TAG/COMMIT_HASH
     hooks:
-      -
-        id: jupytext
-        name: jupytext
-        entry: jupytext --from ipynb --to py:light --pre-commit
-        pass_filenames: false
-        language: python
-      -
-        id: unstage-ipynb
-        name: unstage-ipynb
-        entry: git reset HEAD **/*.ipynb
-        pass_filenames: false
-        language: system
+    - id: jupytext
+      args: [--sync, --pipe, black]
+      additional_dependencies:
+        - black==19.10b0 # Matches hook
 
+-   repo: https://github.com/psf/black
+    rev: 19.10b0
+    hooks:
+    - id: black
+      language_version: python3
 ```
diff --git a/jupytext/cli.py b/jupytext/cli.py
@@ -87,6 +87,18 @@ def parse_jupytext_args(args=None):
         "on the notebooks found in the git index, which have an "
         "extension that matches the (optional) --from argument.",
     )
+    parser.add_argument(
+        "--ignore-unmatched",
+        action="store_true",
+        help="Ignore passed filepaths that aren't in the source format "
+        "you are trying to convert from.",
+    )
+    parser.add_argument(
+        "--alert-untracked",
+        action="store_true",
+        help="Exit with a non-zero status if the output files are not "
+        "tracked in the git index",
+    )
     parser.add_argument(
         "--from",
         dest="input_format",
@@ -417,15 +429,29 @@ def jupytext_single_file(nb_file, args, log):
             msg += " Maybe you mean 'jupytext --sync {}' ?".format(args.set_formats)
         raise ValueError(msg)
 
-    nb_dest = args.output or (
-        None
-        if not args.output_format
-        else (
-            "-"
-            if nb_file == "-"
-            else full_path(base_path(nb_file, args.input_format), args.output_format)
-        )
-    )
+    nb_dest = None
+    if args.output:
+        nb_dest = args.output
+    elif args.output_format and nb_file == "-":
+        nb_dest = "-"
+    else:
+        try:
+            bp = base_path(nb_file, args.input_format)
+        except InconsistentPath:
+            if args.ignore_unmatched:
+                log(
+                    "[jupytext] Ignoring unmatched input path {}{}".format(
+                        nb_file,
+                        " for format {}".format(args.input_format)
+                        if args.input_format
+                        else "",
+                    )
+                )
+                return 0
+            else:
+                raise
+        if args.output_format:
+            nb_dest = full_path(bp, args.output_format)
 
     config = load_jupytext_config(os.path.abspath(nb_file))
 
@@ -688,18 +714,37 @@ def jupytext_single_file(nb_file, args, log):
         if modified:
             inputs_nb_file = outputs_nb_file = None
 
+        untracked_paths = []
+
         def write_function(path, fmt):
-            # Do not write the ipynb file if it was not modified
-            # But, always write text representations to make sure they are the most recent
-            if path == inputs_nb_file and path == outputs_nb_file:
+            if path == inputs_nb_file:
+                # We update the timestamp of the text file to make sure it remains more recent than the ipynb
+                if path != outputs_nb_file:
+                    log("[jupytext] Sync timestamp of '{}'".format(nb_file))
+                    os.utime(nb_file, None)
+
+                # We don't write the ipynb file if it was not modified
                 return
+
             log("[jupytext] Updating '{}'".format(path))
             write(notebook, path, fmt=fmt)
             if args.pre_commit:
                 system("git", "add", path)
+            if args.alert_untracked and is_untracked(path):
+                nonlocal untracked_paths
+                untracked_paths.append(path)
 
         formats = prepare_notebook_for_save(notebook, config, nb_file)
         write_pair(nb_file, formats, write_function)
+        if untracked_paths:
+            log(
+                "[jupytext] Output file {nb_dest} is not tracked in the git index, "
+                "add it to the index using 'git add' to fix this.".format(
+                    nb_dest=", ".join(untracked_paths)
+                )
+            )
+            return 1
+
     elif (
         os.path.isfile(nb_file)
         and nb_dest.endswith(".ipynb")
@@ -711,7 +756,14 @@ def write_function(path, fmt):
         log("[jupytext] Sync timestamp of '{}'".format(nb_file))
         os.utime(nb_file, None)
 
-    return 0
+    if args.alert_untracked and is_untracked(nb_dest):
+        log(
+            "[jupytext] Output file {nb_dest} is not tracked in the git index, "
+            "add it to the index using 'git add' to fix this.".format(nb_dest=nb_dest)
+        )
+        return 1
+    else:
+        return 0
 
 
 def notebooks_in_git_index(fmt):
@@ -731,6 +783,14 @@ def notebooks_in_git_index(fmt):
     return files
 
 
+def is_untracked(filepath):
+    """Check whether a file is untracked by the git index"""
+    if not filepath:
+        return False
+    output = system("git", "ls-files", filepath).strip()
+    return output == ""
+
+
 def print_paired_paths(nb_file, fmt):
     """Display the paired paths for this notebook"""
     notebook = read(nb_file, fmt=fmt)

diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -10,6 +10,7 @@ setuptools
 toml
 jupyterlab==3.0.0
 jupyter-packaging
+pre-commit
 
 # Python 2
 pathlib

diff --git a/tests/test_metadata_filter.py b/tests/test_metadata_filter.py
@@ -156,12 +156,11 @@ def test_default_config_has_priority_over_current_metadata(
 
     cfg_file = tmpdir.join("jupytext.toml")
     cfg_file.write(
-        """default_jupytext_formats = "ipynb,py:percent"
-default_cell_metadata_filter = "-some_metadata_key"
+        """default_cell_metadata_filter = "-some_metadata_key"
 """
     )
 
-    jupytext_cli([str(py_file), "--sync"])
+    jupytext_cli([str(py_file), "--to", "py"])
     assert (
         py_file.read()
         == """# %%