Support for 3.6 f-string syntax #342
Comments
|
This might be because of the |
|
The (terrible) patch below solves this issue. It monkey-patches internal objects in Sample transformationprint( F"omg"
Rf'''tripple strings
are cool
'''
f"such yapf"
)print(F"omg" Rf'''tripple strings
are cool
''' f"such yapf")The patch (brace yourself)diff --git a/yapf/yapflib/pytree_utils.py b/yapf/yapflib/pytree_utils.py
index 1bebcb8..2384a04 100644
--- a/yapf/yapflib/pytree_utils.py
+++ b/yapf/yapflib/pytree_utils.py
@@ -41,6 +41,58 @@ OPENING_BRACKETS = frozenset({'(', '[', '{'})
CLOSING_BRACKETS = frozenset({')', ']', '}'})
+from lib2to3.pgen2 import tokenize
+import re
+tokenize.Triple = tokenize.group("[ubUB]?[rR]?'''",
+ '[ubUB]?[rR]?"""',
+ "[rR]?[fF]?'''",
+ '[fF]?[rR]?"""')
+tokenize.String = tokenize.group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+ r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"',
+ r"[fF]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+ r"[rR]?[fF]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+ r'[fF]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"',
+ r'[rR]?[fF]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"')
+tokenize.ContStr = tokenize.group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+ tokenize.group("'", r'\\\r?\n'),
+ r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+ tokenize.group('"', r'\\\r?\n'),
+ r"[fF]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+ tokenize.group("'", r'\\\r?\n'),
+ r"[rR]?[fF]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+ tokenize.group("'", r'\\\r?\n'),
+ r'[fF]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+ tokenize.group('"', r'\\\r?\n'),
+ r'[rR]?[fF]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+ tokenize.group('"', r'\\\r?\n'))
+tokenize.PlainToken = tokenize.group(tokenize.Number, tokenize.Funny, tokenize.String, tokenize.Name)
+tokenize.Token = tokenize.Ignore + tokenize.PlainToken
+tokenize.PseudoExtras = tokenize.group(r'\\\r?\n', tokenize.Comment, tokenize.Triple)
+tokenize.PseudoToken = tokenize.Whitespace + tokenize.group(
+ tokenize.PseudoExtras, tokenize.Number, tokenize.Funny, tokenize.ContStr,
+ tokenize.Name)
+
+(tokenize.tokenprog, tokenize.pseudoprog, tokenize.single3prog,
+ tokenize.double3prog) = list(
+ map(re.compile, (tokenize.Token, tokenize.PseudoToken, tokenize.Single3,
+ tokenize.Double3)))
+
+tokenize.endprogs.update({'f': None, 'F': None})
+
+for t in ('f', 'F', 'fr', 'rf', 'Fr', 'rF', 'fR', 'Rf', 'FR', 'RF'):
+ tt = f'{t}"""'
+ tokenize.endprogs[tt] = tokenize.double3prog
+ tokenize.triple_quoted[tt] = tt
+ tt = f"{t}'''"
+ tokenize.endprogs[tt] = tokenize.single3prog
+ tokenize.triple_quoted[tt] = tt
+ tt = f'{t}"'
+ tokenize.single_quoted[tt] = tt
+ tt = f"{t}'"
+ tokenize.single_quoted[tt] = tt
+
+
+
class Annotation(object):
"""Annotation names associated with pytrees."""
CHILD_INDENT = 'child_indent' |
|
Whoa! That's certainly an interesting patch. :-) I know nothing about |
|
It's a terrible solution, nothing sustainable long-term. I couldn't possibly submit a PR for that and be able to look in the mirror. The actual issue is that yapf uses the lib2to3 parser for both Python 2 and 3, even though it was only made for Python 2. The syntaxes are quite different and the gap will continue to widen on each Python 3 release. yapf needs a proper Python 3 parser! |
|
I totally agree (with all of the sentiments :-) ). I was mostly hoping to get past this hurdle, which seems to be affecting a lot of people. Anyway, you're right that I'd love a proper Python 3 parser, but there doesn't appear to be one around that gives the same AST as |
|
So, what is the status of this? As 3.6 gets more adoption, it seems this is going to become a bigger issue. Is there a way to have YAPF just skip fstrings when doing formating as a workaround for this? |
|
Let me stress the fact yapf uses a Python 2 only lexer/parser. As keywords and syntactic sugars get added to Python 3, yapf will become more and more unable to format anything without layers afters layers of terrible monkey patches such as the one I wrote. |
|
Is there a plan to add a Python 3 lexer/parser? Or is that just way too large of a project and there isn't enough need by the maintainers? |
|
Can we accept the really hacky fix then? Once you start using f-strings, there's no going back to |
|
I'm also in favor of the hacky fix... I want to use yapf on our team but we're also using f-strings now and there's no going back. :) If the hacky fix isn't merged to master could we get it in a python3 branch? That way we'd still have a central place to install from via pip... |
|
@zopieux It looks like earlier on this thread, @gwelymernans asked if you could turn your patch into a proper pull request. Would you be willing to do that? I'd love to see this move forward! |
|
@bolinfest I thought I made my point clear enough by now: this is not something that can be turned into a "proper" PR. This is a hack. There are potentially many other subtle syntax changes introduced by Python 3.x releases waiting to explode in our faces. In my opinion, the current yapf for Python 3 works by coincidence and I shall not contribute to this monstrosity. I would gladly accept contributing to some linter (yapf or other) that would use a proper Python 3 lexer/parser. But at the moment, yapf maintainers – as most Google open source projects – do not seem to be willing to progress on Python 3 stuff, let alone cutting-edge versions. |
|
@zopieux By "proper" PR, I don't mean "proper" in terms of code quality, but "proper" in the sense that it is a pull request that the maintainers could potentially formally accept and ingest rather than just some code pasted in a bug report. I get that the underpinnings of yapf make all Python 3 work a hack, but it seems like myself and others would rather have a giant hack rather than nothing, which seems to be our alternative. |
|
I'm not unwilling to progress on Python 3 stuff. It's quite simply that I have no choice in the matter. |
|
Notice that this is something they plan on fixing in |
|
Looks like the python issue has now been marked as 'resolved'. 🎉 |
|
Link to actual commit that is more or less equivalent to my patch: python/cpython@e8412e6 |
|
You should have had more faith in your solution @zopieux ! Nice work coming up with that |
|
Now that this has been fixed upstream, what is the procedure to get this into yapf so that I can start using f-strings? |
|
@RenatoUtsch waiting for 3.6.2? |
|
Makes sense. Thanks. |
|
HUZZAH!!! Our long national nightmare will soon be over. :-D |
|
I already using 3.6.2rc1, it works fine with yapf. |
|
FYI: Python 3.6.2 was just released |
|
I had the same issue and I can confirm that upgrading Python from 3.6.1 to 3.6.2 resolved the issue for me. |
|
I think this issue should be closed. |
|
f-string still crashes yapf on my end, with yapf 0.22.0, Python 3.7.0
returns the same traceback message as the original post: |
|
@ligyxy I can't reproduce on 3.7.0: $ docker run -it python:3.7.0 /bin/bash
root@7bcee8f355e1:/# pip install yapf==0.22.0
Collecting yapf==0.22.0
Downloading https://files.pythonhosted.org/packages/d5/ae/9d2e8f43f2ce467991c8310e361bbf4f1e1bf32afc6441b4e3416685b7ef/yapf-0.22.0-py2.py3-none-any.whl (166kB)
100% |████████████████████████████████| 174kB 1.3MB/s
Installing collected packages: yapf
Successfully installed yapf-0.22.0
root@7bcee8f355e1:/# python
Python 3.7.0 (default, Aug 4 2018, 02:33:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yapf.yapflib.yapf_api import FormatCode
>>> FormatCode('f"test"')
('f"test"\n', False) |
|
Ah, it's my bad. I was using Python 3.6.0 wrongly. |
|
fstrings are not working in python 3.74 yapf 0.28.0 from commandline: Test file: Running FormatCode directly works: |
|
@tvykruta this issue has been closed a while ago. By the way, when calling |
Apologies for the brief report, but here's a simple bug reproduction session in the shell:
The text was updated successfully, but these errors were encountered: