Mass testing fixes

[isidentical]

CC: @pablogsal @lysnikolaou

Kept each commit as a single fix, there are a couple more incoming.

[isidentical]

This is the list of files (you can find the exact file on GitHub/PyPI) that I've found, will report back with a better list once I have a few common fixes ready: https://gist.github.com/isidentical/1762ae58c6ff41b29fc2ae970df63108

[isidentical]

Seems like we are close to full roundtripping. Two cases that hit my eye (I also need to test my latest changes to ensure they don't break anything extra :D) that I don't have time today to fix (will take a look at them tomorrow if it is not fixed by then)

# This seems like a bug with escaping

--- /tmp/ast-of/pypi/yt-dlp-2022.11.11/yt_dlp/extractor/common.py
+++ /tmp/ast-of/pypi/yt-dlp-2022.11.11/yt_dlp/extractor/common.py
@@ -2576,7 +2576,7 @@
             Name(id='filename', ctx=Store())],
            value=JoinedStr(
             values=[
-             Constant(value='\\\\?\\'),
+             Constant(value='\\?\\'),
              FormattedValue(
               value=Name(id='absfilepath', ctx=Load()),
               conversion=-1)]))],

# A lot of occurences, we should be able to eliminate these quite eazily since there is no real bug

--- /tmp/ast-of/pypi/docformatter-1.5.0/docformatter.py
+++ /tmp/ast-of/pypi/docformatter-1.5.0/docformatter.py
@@ -2622,6 +2622,7 @@
       Return(
        value=JoinedStr(
         values=[
+         Constant(value=''),
          FormattedValue(
           value=Name(id='open_quote', ctx=Load()),
           conversion=-1),

Trees differ: pypi/docformatter-1.5.0/docformatter.py
Trees differ: pypi/poetry-1.2.2/tests/console/commands/cache/test_list.py
Trees differ: pypi/statsmodels-0.13.5/statsmodels/tsa/ardl/model.py
Trees differ: pypi/apache-beam-2.43.0/apache_beam/dataframe/io.py
Trees differ: pypi/polars-0.14.29/polars/internals/lazyframe/frame.py
Trees differ: pypi/pyflakes-2.5.0/pyflakes/test/test_api.py
Trees differ: pypi/great_expectations-0.15.34/scripts/build_api_docs.py
Trees differ: pypi/great_expectations-0.15.34/tests/data_context/store/test_validations_store.py
Trees differ: pypi/great_expectations-0.15.34/tests/data_context/store/test_html_site_store.py
Trees differ: pypi/great_expectations-0.15.34/tests/data_context/store/test_expectations_store.py
Trees differ: pypi/great_expectations-0.15.34/tests/data_context/store/test_store_backends.py
Trees differ: pypi/yt-dlp-2022.11.11/yt_dlp/extractor/common.py
Trees differ: pypi/docutils-0.19/docutils/core.py
Trees differ: pypi/hatch-1.6.3/tests/cli/project/test_metadata.py
Trees differ: pypi/west-0.14.0/src/west/app/project.py

[pablogsal]

These changes LGTM. Is there any reason why there is still a WIP in the title?

[pablogsal]

We could also add a new tokenizer mode kind for this. I don't know what would simplify this more

[isidentical]

Was thinking about a mode, but since each mode currently maps to a single get function I thought it wasn't that necessary and this flag can kept in the mode level (and even removed in the future if we can leverage the other meta information here)

[pablogsal]

Thanks a lot for doing this check and fixing the errors @isidentical! We are so close! 🚀

[isidentical]

These changes LGTM. Is there any reason why there is still a WIP in the title?

I need to fix a final bug (the first example in here: #199 (comment)), but then we should be able to land this (which would mean we can roundtrip pretty much everything)

[isidentical]

Landing this now, will take care of the weird backslashes in the subsequent one.