New-line problem on Windows

[jenstroeger]

First of all, thank you for a great tool!

While playing with it, I came across an obscure issue running on Github’s Windows runner: see for example this failing run. Now here’s the puzzling problem… all YAML files in question use \n line-endings, for example the .yamllint.yaml file:

python-package-template > cat .yamllint.yaml | hexdump -C
00000000  23 20 4d 6f 64 69 66 79  20 79 61 6d 6c 6c 69 6e  |# Modify yamllin|
00000010  74 27 73 20 64 65 66 61  75 6c 74 73 20 73 6f 20  |t's defaults so |
00000020  74 68 61 74 20 74 68 65  79 20 77 6f 72 6b 20 77  |that they work w|
00000030  65 6c 6c 20 61 6e 64 20  73 74 72 69 63 74 6c 79  |ell and strictly|
00000040  20 77 69 74 68 0a 23 20  72 75 61 6d 65 6c 27 73  | with.# ruamel's|
00000050  20 70 72 65 74 74 79 2d  66 6f 72 6d 61 74 74 69  | pretty-formatti|
00000060  6e 67 20 6f 66 20 74 68  65 20 59 41 4d 4c 20 66  |ng of the YAML f|
00000070  69 6c 65 73 2e 0a 23 0a  23 20 46 6f 72 20 6d 6f  |iles..#.# For mo|
00000080  72 65 20 64 65 74 61 69  6c 73 3a 20 68 74 74 70  |re details: http|
00000090  73 3a 2f 2f 79 61 6d 6c  6c 69 6e 74 2e 72 65 61  |s://yamllint.rea|
000000a0  64 74 68 65 64 6f 63 73  2e 69 6f 2f 65 6e 2f 73  |dthedocs.io/en/s|
000000b0  74 61 62 6c 65 2f 72 75  6c 65 73 2e 68 74 6d 6c  |table/rules.html|
000000c0  0a 23 0a 23 20 79 61 6d  6c 6c 69 6e 74 20 64 69  |.#.# yamllint di|
000000d0  73 61 62 6c 65 20 72 75  6c 65 3a 64 6f 63 75 6d  |sable rule:docum|
000000e0  65 6e 74 2d 73 74 61 72  74 20 72 75 6c 65 3a 64  |ent-start rule:d|
000000f0  6f 63 75 6d 65 6e 74 2d  65 6e 64 0a 0a 65 78 74  |ocument-end..ext|
00000100  65 6e 64 73 3a 20 64 65  66 61 75 6c 74 0a 72 75  |ends: default.ru|
00000110  6c 65 73 3a 0a 20 20 6c  69 6e 65 2d 6c 65 6e 67  |les:.  line-leng|
00000120  74 68 3a 0a 20 20 20 20  6d 61 78 3a 20 35 31 32  |th:.    max: 512|
00000130  0a 20 20 69 6e 64 65 6e  74 61 74 69 6f 6e 3a 0a  |.  indentation:.|
00000140  20 20 20 20 69 6e 64 65  6e 74 2d 73 65 71 75 65  |    indent-seque|
00000150  6e 63 65 73 3a 20 63 6f  6e 73 69 73 74 65 6e 74  |nces: consistent|
00000160  0a 20 20 63 6f 6d 6d 65  6e 74 73 3a 0a 20 20 20  |.  comments:.   |
00000170  20 6d 69 6e 2d 73 70 61  63 65 73 2d 66 72 6f 6d  | min-spaces-from|
00000180  2d 63 6f 6e 74 65 6e 74  3a 20 31 0a              |-content: 1.|
0000018c

By default, yamllint checks for UNIX-type line-endings (docs, src) which would be ok for these YAML files. Furthermore, YAML files are opened with their native line-endings preserved (docs):

yamllint/yamllint/cli.py

Lines 222 to 223 in 1d65ab6

	with open(file, newline='') as f:
	problems = linter.run(f, conf, filepath)

So by all means and purposes, that check should pass — but it doesn’t 🤔

After poking through the code I can’t spot an obvious issue. I’m also unable to reproduce this on Windows because I’m unsure how the GIthub runner executes Python: native Windows or via WSL.

Maybe this issue is related to issue #218 or #347, although I don’t quite see the connection. To aid debugging, I’m tempted to expand the error message

diff --git a/yamllint/rules/new_lines.py b/yamllint/rules/new_lines.py
index 4e3023e..8fb1174 100644
--- a/yamllint/rules/new_lines.py
+++ b/yamllint/rules/new_lines.py
@@ -53,7 +53,9 @@ def check(conf, line):
         newline_char = '\r\n'
 
     if line.start == 0 and len(line.buffer) > line.end:
-        if line.buffer[line.end:line.end + len(newline_char)] != newline_char:
+        actual_newline_char = line.buffer[line.end:line.end + len(newline_char)]
+        if actual_newline_char != newline_char:
             c = repr(newline_char).strip('\'')
+            actual_c = repr(actual_newline_char).strip('\'')
             yield LintProblem(1, line.end - line.start + 1,
-                              f'wrong new line character: expected {c}')
+                              f'wrong new line character: found {actual_c} expected {c}')

to get more details. Would you accept a PR?

[adrienverge]

Hello and thanks for your clear and comprehensible message.

Unfortunately, no: we won't accept a PR that changes yamllint output for a specific debugging use-case.

However, you can try building your own customized yamllint and use it or GitHub CI (I guess it's possible somehow).
You should also try to narrow down the problem, for instance by running yamllint directly (instead of make and pre-commit), and see if you find something.

[jenstroeger]

Unfortunately, no: we won't accept a PR that changes yamllint output for a specific debugging use-case.

Hmm bummer; I’d argue it improved the error message in general, but ok.

Any idea about what might be going on here? I’m unable to reproduce this locally on Windows… so… 🤔