make format YAPF Exceptions

[JakeMercer]

The default python3 package available via apt in Ubuntu 16.04 is version 3.5.2.

YAPF, the tool used by make format to restyle Python scripts, is designed for Python 2 and has had several "bolt-ons" to support Python 3. The standard version supplied by the apt repositories is not supported by YAPF as indicated by the following line in their README:

YAPF supports Python 2.7 and 3.6.4+. (Note that some Python 3 features may fail to parse with Python versions before 3.6.4.)

Issues such as the following may occur as a result of calling make format with Python 3.5.2:

(env) user@SymbiFlow:~/Development/prjxray$ make format
find . -name \*.cc -and -not -path './third_party/*' -and -not -path './.git/*' -and -not -path './env/*' -and -not -path './build/*' -print0 | xargs -0 -P $(nproc) clang-format-5.0 -style=file -i
find . -name \*.h -and -not -path './third_party/*' -and -not -path './.git/*' -and -not -path './env/*' -and -not -path './build/*' -print0 | xargs -0 -P $(nproc) clang-format-5.0 -style=file -i
./.github/update-contributing.py
if [ -e env/bin/activate ]; then . env/bin/activate; fi; find . -name \*.py -and -not -path './third_party/*' -and -not -path './.git/*' -and -not -path './env/*' -and -not -path './build/*' -print0 | xargs -0 yapf -p -i
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/concurrent/futures/process.py", line 272, in _queue_management_worker
    result_item = reader.recv()
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 251, in recv
    return ForkingPickler.loads(buf.getbuffer())
TypeError: __init__() missing 3 required positional arguments: 'type', 'value', and 'context'

This error is obfuscated by parallel processing, the actual issue is as a result of lib2to3 not understanding particular Python 3 syntax; the real issue's stack trace is as follows (without -p argument):

(env) user@SymbiFlow:~/Development/prjxray$ find . -name \*.py -and -not -path './third_party/*' -and -not -path './.git/*' -and -not -path './env/*' -and -not -path './build/*' -print0 | xargs -0 yapf -i
Traceback (most recent call last):
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.5/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=12, value=',', context=('', (240, 49))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Development/prjxray/env/bin/yapf", line 8, in <module>
    sys.exit(run_main())
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/__init__.py", line 317, in run_main
    sys.exit(main(sys.argv))
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/__init__.py", line 205, in main
    verbose=args.verbose)
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/__init__.py", line 255, in FormatFiles
    in_place, print_diff, verify, verbose)
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/__init__.py", line 280, in _FormatFile
    logger=logging.warning)
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/yapflib/yapf_api.py", line 91, in FormatFile
    verify=verify)
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/home/user/Development/prjxray/env/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.5/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=22, value='=', context=('', (177, 12))

Here is a discussion on the subject on the YAPF GitHub.

Suggestions/Possible Solutions:

• Extend the README to indicate that Python 3.6.4 or higher is required and add a check to the top-level Makefile to ensure a compatible version is being used.
• Test a newer version of Ubuntu which uses a compatible Python version and update the README accordingly.

[JakeMercer]

On another note, make format seems to work on Travis; this would suggest that the CI is not using the environment described by the README.