Compiler subsystem #26

Deric-W · 2021-01-27T18:27:04Z

Implement Issue #25

This commit adds the compiler subpackage to implement issue #25. The compiler subpackage currently contains abstract base classes and will receive implementations later.

The RegexParser is a Parser implementation identifying the start and end of a code section with regular expressions. This is the standart implementation modified to implement issue #25.

- add CompileError for exceptions during compiling - replace Mapping in Code.execute with Dict because exec needs a dict - move line offset handling from CodeBuilder to Parser - replace name and loader with ModuleSpec

This commit changes RegexParser to comply to the new abc. Furthermore, CompileError is changed to show the section which failed.

This commit adds the current implementation of code objects to the compiler subpackage. This requires to raise the minimal python version to 3.8 as stated in issue #25.

This is necessary because the generic code implementation requires the CodeType.replace method, which is available in python 3.8+.

This commit adds a facade to the compiler subsystem as mentioned in issue #25.

This commits adds the dedenting feature as a decorator as mentioned in issue #25.

To allow for better testing, the code for detecting code lines is moved into a seperate method.

Before this commit, every builder kept track the section number themself, resulting in redundancy and susceptibility to errors. This commit modifies compiler.CodeBuilder.add_* to accept the current section number.

This commit adds a implementation of compiler.Code in python bytecode as mentioned in issue #25. It is implemented using a generator and currently slower than the generic implementation to allow for variables to be passed to .execute. If MAKE_FUNCTION gets faster or the compiler subsystem gets changed it may be faster.

This commit replaces a few type hints in the compiler subpackage with generics. This allows detaching a CodeBuilderDecorator which was detached from another CodeBuilderDecorator and fixes the result of CodeBuilder.copy() not having the same type as his 'parent'.

To allow for better error discovery, this commit replaces CompileError with a normal SyntaxError with the correct line number set.

To allow for Python version 3.7 to be used, GenericCodeBuilder now uses the ast module instead of CodeType.replace.

This commit updates the workflow to not run MyPy on pypy-3.7 because it is not supported.

pep8speaks · 2021-01-27T18:27:12Z

Hello @Deric-W! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file cache_handlers/files_mtime.py:

Line 14:80: E501 line too long (209 > 79 characters)
Line 25:80: E501 line too long (106 > 79 characters)
Line 27:80: E501 line too long (90 > 79 characters)
Line 28:80: E501 line too long (91 > 79 characters)
Line 35:80: E501 line too long (122 > 79 characters)
Line 36:80: E501 line too long (111 > 79 characters)
Line 38:80: E501 line too long (101 > 79 characters)
Line 39:80: E501 line too long (92 > 79 characters)
Line 43:80: E501 line too long (115 > 79 characters)
Line 52:80: E501 line too long (101 > 79 characters)
Line 53:80: E501 line too long (108 > 79 characters)

In the file pyhp/__init__.py:

Line 23:1: F822 undefined name 'compiler' in all
Line 23:1: F822 undefined name 'libpyhp' in all
Line 23:1: F822 undefined name 'main' in all

In the file pyhp/compiler/__init__.py:

Line 21:1: F822 undefined name 'parsers' in all
Line 21:1: F822 undefined name 'generic' in all
Line 21:1: F822 undefined name 'bytecode' in all
Line 21:1: F822 undefined name 'util' in all
Line 45:80: E501 line too long (80 > 79 characters)
Line 94:80: E501 line too long (84 > 79 characters)
Line 103:80: E501 line too long (90 > 79 characters)
Line 104:80: E501 line too long (105 > 79 characters)
Line 107:80: E501 line too long (85 > 79 characters)

In the file pyhp/compiler/bytecode.py:

Line 4:80: E501 line too long (80 > 79 characters)
Line 26:80: E501 line too long (92 > 79 characters)
Line 61:80: E501 line too long (94 > 79 characters)
Line 76:80: E501 line too long (80 > 79 characters)
Line 79:80: E501 line too long (91 > 79 characters)
Line 99:80: E501 line too long (87 > 79 characters)
Line 108:80: E501 line too long (96 > 79 characters)
Line 116:80: E501 line too long (81 > 79 characters)

In the file pyhp/compiler/generic.py:

Line 35:80: E501 line too long (91 > 79 characters)
Line 40:80: E501 line too long (94 > 79 characters)
Line 55:80: E501 line too long (80 > 79 characters)
Line 58:80: E501 line too long (91 > 79 characters)
Line 79:80: E501 line too long (87 > 79 characters)
Line 94:80: E501 line too long (81 > 79 characters)

In the file pyhp/compiler/parsers.py:

Line 23:80: E501 line too long (99 > 79 characters)
Line 31:80: E501 line too long (91 > 79 characters)
Line 35:80: E501 line too long (90 > 79 characters)
Line 36:80: E501 line too long (105 > 79 characters)
Line 39:80: E501 line too long (110 > 79 characters)
Line 40:80: E501 line too long (92 > 79 characters)
Line 41:80: E501 line too long (88 > 79 characters)
Line 45:80: E501 line too long (95 > 79 characters)
Line 47:80: E501 line too long (81 > 79 characters)
Line 50:80: E501 line too long (84 > 79 characters)
Line 52:80: E501 line too long (87 > 79 characters)

In the file pyhp/compiler/util.py:

Line 25:80: E501 line too long (93 > 79 characters)
Line 48:80: E501 line too long (106 > 79 characters)
Line 52:80: E501 line too long (92 > 79 characters)
Line 54:80: E501 line too long (82 > 79 characters)
Line 59:80: E501 line too long (108 > 79 characters)
Line 62:80: E501 line too long (81 > 79 characters)
Line 68:80: E501 line too long (83 > 79 characters)
Line 88:80: E501 line too long (80 > 79 characters)
Line 96:80: E501 line too long (98 > 79 characters)
Line 98:80: E501 line too long (94 > 79 characters)
Line 99:80: E501 line too long (92 > 79 characters)
Line 101:80: E501 line too long (99 > 79 characters)
Line 102:80: E501 line too long (89 > 79 characters)
Line 110:80: E501 line too long (90 > 79 characters)

In the file pyhp/main.py:

Line 30:80: E501 line too long (81 > 79 characters)
Line 57:80: E501 line too long (90 > 79 characters)
Line 59:80: E501 line too long (95 > 79 characters)
Line 66:80: E501 line too long (97 > 79 characters)
Line 67:80: E501 line too long (91 > 79 characters)
Line 69:80: E501 line too long (106 > 79 characters)
Line 70:80: E501 line too long (88 > 79 characters)
Line 73:80: E501 line too long (82 > 79 characters)
Line 82:80: E501 line too long (90 > 79 characters)
Line 89:80: E501 line too long (90 > 79 characters)
Line 97:80: E501 line too long (99 > 79 characters)
Line 103:80: E501 line too long (87 > 79 characters)
Line 110:80: E501 line too long (90 > 79 characters)
Line 112:80: E501 line too long (99 > 79 characters)
Line 130:80: E501 line too long (102 > 79 characters)
Line 143:80: E501 line too long (82 > 79 characters)
Line 149:80: E501 line too long (85 > 79 characters)
Line 152:80: E501 line too long (81 > 79 characters)

In the file tests/compiler/__init__.py:

Line 28:80: E501 line too long (90 > 79 characters)

In the file tests/compiler/test_bytecode.py:

Line 25:80: E501 line too long (98 > 79 characters)
Line 87:80: E501 line too long (89 > 79 characters)
Line 127:80: E501 line too long (91 > 79 characters)
Line 140:80: E501 line too long (88 > 79 characters)

In the file tests/compiler/test_generic.py:

Line 156:80: E501 line too long (91 > 79 characters)
Line 169:80: E501 line too long (96 > 79 characters)

In the file tests/compiler/test_parsers.py:

Line 16:80: E501 line too long (85 > 79 characters)

In the file tests/compiler/test_util.py:

Line 30:80: E501 line too long (91 > 79 characters)

In the file tests/test_main.py:

Line 24:80: E501 line too long (92 > 79 characters)
Line 36:80: E501 line too long (92 > 79 characters)
Line 43:80: E501 line too long (104 > 79 characters)
Line 57:80: E501 line too long (111 > 79 characters)
Line 69:80: E501 line too long (112 > 79 characters)
Line 81:80: E501 line too long (116 > 79 characters)
Line 93:80: E501 line too long (108 > 79 characters)
Line 105:80: E501 line too long (114 > 79 characters)
Line 117:80: E501 line too long (115 > 79 characters)
Line 129:80: E501 line too long (114 > 79 characters)
Line 138:80: E501 line too long (80 > 79 characters)
Line 141:80: E501 line too long (126 > 79 characters)
Line 153:80: E501 line too long (111 > 79 characters)
Line 165:80: E501 line too long (114 > 79 characters)
Line 174:80: E501 line too long (92 > 79 characters)
Line 175:80: E501 line too long (109 > 79 characters)
Line 180:80: E501 line too long (114 > 79 characters)
Line 192:80: E501 line too long (92 > 79 characters)
Line 193:80: E501 line too long (109 > 79 characters)
Line 198:80: E501 line too long (143 > 79 characters)
Line 210:80: E501 line too long (94 > 79 characters)
Line 213:80: E501 line too long (140 > 79 characters)
Line 230:80: E501 line too long (80 > 79 characters)
Line 231:80: E501 line too long (81 > 79 characters)
Line 232:80: E501 line too long (81 > 79 characters)
Line 233:80: E501 line too long (81 > 79 characters)

codecov · 2021-01-27T18:27:43Z

Codecov Report

Merging #26 (efdee0b) into master (4abb7ac) will increase coverage by 36.85%.
The diff coverage is 90.03%.

@@             Coverage Diff             @@
##           master      #26       +/-   ##
===========================================
+ Coverage   30.59%   67.45%   +36.85%     
===========================================
  Files           5        9        +4     
  Lines         317      507      +190     
===========================================
+ Hits           97      342      +245     
+ Misses        220      165       -55

Flag	Coverage Δ
Linux	`64.36% <89.92%> (+36.02%)`	⬆️
Windows	`67.45% <90.03%> (+36.85%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pyhp/__main__.py	`0.00% <0.00%> (ø)`
pyhp/main.py	`48.27% <46.15%> (+5.81%)`	⬆️
pyhp/compiler/__init__.py	`90.00% <90.00%> (ø)`
pyhp/compiler/generic.py	`97.18% <97.18%> (ø)`
pyhp/compiler/bytecode.py	`98.55% <98.55%> (ø)`
pyhp/__init__.py	`100.00% <100.00%> (ø)`
pyhp/compiler/parsers.py	`100.00% <100.00%> (ø)`
pyhp/compiler/util.py	`100.00% <100.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4abb7ac...efdee0b. Read the comment docs.

Deric-W added 29 commits January 21, 2021 15:02

add compiler subpackage

a697e8c

This commit adds the compiler subpackage to implement issue #25. The compiler subpackage currently contains abstract base classes and will receive implementations later.

add RegexParser

d280444

The RegexParser is a Parser implementation identifying the start and end of a code section with regular expressions. This is the standart implementation modified to implement issue #25.

multiple improvements to compiler abc

a0e98e3

- add CompileError for exceptions during compiling - replace Mapping in Code.execute with Dict because exec needs a dict - move line offset handling from CodeBuilder to Parser - replace name and loader with ModuleSpec

fix RegexParser, improve CompileError

d740e95

This commit changes RegexParser to comply to the new abc. Furthermore, CompileError is changed to show the section which failed.

fix line_offset of RegexParser always being 0

dc926c7

add generic code implementation to compiler

b55eee9

This commit adds the current implementation of code objects to the compiler subpackage. This requires to raise the minimal python version to 3.8 as stated in issue #25.

remove pypy3 from tested versions

5d7d217

This is necessary because the generic code implementation requires the CodeType.replace method, which is available in python 3.8+.

add __eq__ implementation to Code objects

e98570a

add Compiler facade

3874e2b

This commit adds a facade to the compiler subsystem as mentioned in issue #25.

add __all__ to compiler submodules, optimize GenericCodeBuilder.copy

685b5ed

add Dedenter

04d0366

This commits adds the dedenting feature as a decorator as mentioned in issue #25.

add tests for compiler abc

c88e286

add deepcopy support to CodeBuilder

b83d3ac

update setup.cfg classifiers

600f927

add Dedenter.is_code

80daf79

To allow for better testing, the code for detecting code lines is moved into a seperate method.

move the tracking of the section number into the parser

e37d469

Before this commit, every builder kept track the section number themself, resulting in redundancy and susceptibility to errors. This commit modifies compiler.CodeBuilder.add_* to accept the current section number.

update docstrings

e74531b

add tests for correct line numbers and file names in tracebacks

047505c

fix Code.__eq__, make section passed to CompileError optional

8aab53f

replace pyhp.embed with pyhp.compiler (Issue #25)

19440c7

replace CompileError with SyntaxError and correct lineno

16e5402

To allow for better error discovery, this commit replaces CompileError with a normal SyntaxError with the correct line number set.

improve message of indentation error raised by Dedenter

e52b8eb

add type checking to Github workflow

ac669b7

refactor GenericCodeBuilder to allow for Python 3.7+

5db5224

To allow for Python version 3.7 to be used, GenericCodeBuilder now uses the ast module instead of CodeType.replace.

update setup.cfg Python version to 3.7+

9319966

dont run MyPy on pypy-3.7

ae70c67

This commit updates the workflow to not run MyPy on pypy-3.7 because it is not supported.

small cleanups

efdee0b

Deric-W linked an issue Jan 27, 2021 that may be closed by this pull request

enhanced compiler system #25

Closed

Deric-W merged commit fa5275f into master Jan 27, 2021

Deric-W deleted the compiler-subsystem branch January 27, 2021 18:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compiler subsystem #26

Compiler subsystem #26

Deric-W commented Jan 27, 2021

pep8speaks commented Jan 27, 2021

codecov bot commented Jan 27, 2021 •

edited

Loading

Compiler subsystem #26

Compiler subsystem #26

Conversation

Deric-W commented Jan 27, 2021

pep8speaks commented Jan 27, 2021

codecov bot commented Jan 27, 2021 • edited Loading

Codecov Report

codecov bot commented Jan 27, 2021 •

edited

Loading