Add a coder to reread syntaxtrees #3371

sebasguts · 2019-03-21T17:05:13Z

See here for the initial discussion and history

fingolfin · 2019-03-26T22:45:40Z

The first commit "WIP: add SyntaxTreeDefaultCoder" should be removed.

fingolfin

Overall this is looking close to be ready for merging, but I still left some comments.

src/code.c

src/code.h

src/syntaxtree.c

codecov · 2019-03-27T14:35:15Z

Codecov Report

Merging #3371 into master will increase coverage by <.01%.
The diff coverage is 96.44%.

@@            Coverage Diff             @@
##           master    #3371      +/-   ##
==========================================
+ Coverage   85.33%   85.34%   +<.01%     
==========================================
  Files         699      699              
  Lines      345897   346082     +185     
==========================================
+ Hits       295181   295365     +184     
- Misses      50716    50717       +1

Impacted Files	Coverage Δ
src/error.h	`100% <ø> (ø)`	⬆️
src/code.h	`97.29% <ø> (ø)`	⬆️
src/code.c	`99.78% <100%> (ø)`	⬆️
src/intrprtr.c	`99.84% <100%> (ø)`	⬆️
src/syntaxtree.c	`96.12% <95.81%> (-0.75%)`	⬇️
src/scanner.c	`92.34% <0%> (ø)`	⬆️
src/scanner.h	`66.66% <0%> (ø)`	⬆️
lib/maxsub.gi	`70.43% <0%> (+0.13%)`	⬆️
src/read.c	`96.85% <0%> (+0.15%)`	⬆️
src/stringobj.c	`94.38% <0%> (+0.29%)`	⬆️
... and 1 more

sebasguts · 2019-04-04T12:47:02Z

@fingolfin I think I adressed all your comments. I also added sensible error messages for checking if a needed entry in a syntax tree node record is bound.

coveralls · 2019-04-04T14:04:18Z

Coverage increased (+0.006%) to 85.152% when pulling 3c4614a on sebasguts:sg/syntaxtree into 1afd3f7 on gap-system:master.

sebasguts · 2019-04-11T11:47:40Z

This PR is now based on #3370

src/syntaxtree.c

src/code.h

src/scanner.c

src/scanner.h

src/syntaxtree.c

sebasguts · 2019-04-11T15:17:24Z

I think I addressed all your comments, and also added the checks we discussed today.

Should we try to add the context object, too, or would be a different PR a better place.

src/syntaxtree.c

sebasguts · 2019-04-26T19:02:06Z

@fingolfin I think I adressed all of your comments

sebasguts · 2019-05-03T09:35:05Z

@ChrisJefferson @fingolfin ping?

ChrisJefferson · 2019-05-04T11:06:55Z

So, possibly stupid question, is this supposed to be complete, or progress towards a rereader?

I tried extending the test which tests SYNTAX_TREE on every function, by the following horrible patch:

@@ -25,11 +25,17 @@ gap> for n in NamesGVars() do
 >            v := ValueGlobal(n);
 >            if IsFunction(v) and not IsKernelFunction(v) then
 >                SYNTAX_TREE(v);
+>                        if not test_tree(v) then
+>                           Print("failed round trip: ",v,"\n");
+>                        fi;
 >            elif IsOperation(v) then
 >                for i in [1..6] do
 >                    for x in METHODS_OPERATION(v, i) do
 >                        if IsFunction(x) and not IsKernelFunction(v) then
 >                        SYNTAX_TREE(x);
+>                        if not test_tree(x) then
+>                           Print("failed round trip: ",v,"\n");
+>                        fi;
 >                        fi;
 >                    od; 
 >                od;

And GAP segfaults.

sebasguts · 2019-05-04T11:09:37Z

@ChrisJefferson It does not work on operations, no. But I should really check that when calling SyntaxTree The segfault is really bad . . .

sebasguts · 2019-05-04T11:10:40Z

Wait, this is a different issue. Sorry.

ChrisJefferson · 2019-05-04T11:11:36Z

This is enough to upset it:

gap> f := function() return [,[]]; end;
function(  ) ... end
gap> SYNTAX_TREE_CODE(SYNTAX_TREE(f));

sebasguts · 2019-05-04T11:15:26Z

I will fix that. I am not sure how that slipped. Thank you.

Anyway, there is a thing that I am aware of that does not work yet, namely HVAR references that do not point to the "read" function context anymore, but outside.

I plan to overcome this by adding a T_REF_VALUE (and T_ASS_VALUE) to GAP, so we can store those references not as HVARs anymore, but as values directly. However, this of course may change the overall semantics of a function . . .

ChrisJefferson · 2019-05-04T11:16:45Z

(I don't know if this is reasonable), if in cases where SYNTAX_TREE_CODE failed it just returned 'fail', then we could obviously test for when it works, or returns fail, and then still loop over all the functions? (I don't know if that's reasonable given the way it works internally)

sebasguts · 2019-05-04T11:31:18Z

@ChrisJefferson Currently, they throw an error, which I would guess is more reasonable. But I could do a no_error version for testing, if you would prefer?

The problem rereading those HVAR functions is that rereading them works, but their result is arbitrary, see here (which is worse than a segfault)):

gap> xx := (i -> ( j-> i + j ) )(3);
function( j ) ... end
gap> SYNTAX_TREE(SYNTAX_TREE_CODE(SYNTAX_TREE(xx)))=SYNTAX_TREE(xx);
true
gap> SYNTAX_TREE_CODE(SYNTAX_TREE(xx))(3);
Z(3)

ChrisJefferson · 2019-05-04T11:36:24Z

I was just thinking about if it would be reasonable to merge something like the diff I made, to test a large set of functions. If we are still too far away from that, we can also merge the current partial code, and work on improving it later.

sebasguts · 2019-05-04T11:38:53Z

No, absolutely reasonable. I will include it.

sebasguts · 2019-05-04T12:12:20Z

@ChrisJefferson I added the full cycle test, and they pass
@fingolfin I made additional changes to CodeFuncExprEnd, please have a new look at the changes there.

fingolfin · 2019-05-06T12:12:06Z

tst/testinstall/syntaxtree.tst

 >            elif IsOperation(v) then
 >                for i in [1..6] do
 >                    for x in METHODS_OPERATION(v, i) do
 >                        if IsFunction(x) and not IsKernelFunction(v) then
->                        SYNTAX_TREE(x);
+>                            if not test_tree(v) then
+>                                Print("failed round trip: ",n,"\n");


Would be more useful like this:

Suggested change

> Print("failed round trip: ",n,"\n");

> Print("METHODS_OPERATION(", n, ",", i, ") failed round trip\n");

fingolfin · 2019-05-06T12:13:31Z

src/code.c

-        {
-            CodeReturnVoidWhichIsNotProfiled();
-            nr++;
+        while( 1 ){


Code formatting!

Suggested change

while( 1 ){

while (1) {

Also, this code really needs to be commented. Both what it is doing (see my suggestion below), but also more importantly, why it is doing it. Which I don't know right now...

Why doesn't it suffice to insert the following into the existing code, to achieve the same effect?

// find the last statement while (T_SEQ_STAT <= TNUM_STAT(stat1) && TNUM_STAT(stat1) <= T_SEQ_STAT7) { TNUM_STAT(stat1) <= T_SEQ_STAT7) { UInt size = SIZE_STAT(stat1) / sizeof(Stat); stat1 = READ_STAT(stat1, size - 1); }

Well, logically, this does exactly the same, except it is not intertwined with the addition of the return void to the statement.

About the why: The thing is that if we reread a syntax tree, the statements are already nested, i.e., if there are more than 7 statements in the body, the last one is already a new SEQ_STAT*. This ensures that the return; is only added if there is really no return in the last statement.

When a function is coded, the statements are packed into nested statements later.

I discovered this problem when I realized that "long" functions get an additional return;

to indicate that the statement stack is complete and does not need any more additions.

sebasguts requested review from fingolfin and markuspf March 21, 2019 17:05

wucas added the gapdays2019-spring Issues and PRs that arose at https://www.gapdays.de/gapdays2019-spring label Mar 22, 2019

fingolfin reviewed Mar 26, 2019

View reviewed changes

sebasguts force-pushed the sg/syntaxtree branch from 8e327f0 to bd5364f Compare March 27, 2019 14:33

sebasguts force-pushed the sg/syntaxtree branch from 1de6c33 to 2906998 Compare April 4, 2019 12:45

sebasguts force-pushed the sg/syntaxtree branch 2 times, most recently from 5438351 to 581bb00 Compare April 4, 2019 13:06

sebasguts force-pushed the sg/syntaxtree branch from 581bb00 to 4a17596 Compare April 11, 2019 11:47

fingolfin reviewed Apr 11, 2019

View reviewed changes

src/syntaxtree.c Outdated Show resolved Hide resolved

src/syntaxtree.c Outdated Show resolved Hide resolved

src/syntaxtree.c Outdated Show resolved Hide resolved

src/code.h Outdated Show resolved Hide resolved

fingolfin reviewed Apr 11, 2019

View reviewed changes

src/syntaxtree.c Outdated Show resolved Hide resolved

fingolfin reviewed Apr 11, 2019

View reviewed changes

src/syntaxtree.c Show resolved Hide resolved

sebasguts force-pushed the sg/syntaxtree branch 3 times, most recently from e0351b8 to 76c7e11 Compare April 11, 2019 15:15

fingolfin reviewed Apr 12, 2019

View reviewed changes

src/syntaxtree.c Outdated Show resolved Hide resolved

fingolfin reviewed Apr 12, 2019

View reviewed changes

src/syntaxtree.c Show resolved Hide resolved

fingolfin reviewed Apr 12, 2019

View reviewed changes

src/syntaxtree.c Outdated Show resolved Hide resolved

src/syntaxtree.c Show resolved Hide resolved

src/syntaxtree.c Show resolved Hide resolved

src/syntaxtree.c Outdated Show resolved Hide resolved

fingolfin mentioned this pull request Apr 24, 2019

kernel: rename expression and statement "TNUMs" to avoid confusion with actual TNUMs #3422

Merged

8 tasks

sebasguts force-pushed the sg/syntaxtree branch from 76c7e11 to ce6af4f Compare April 26, 2019 18:59

sebasguts force-pushed the sg/syntaxtree branch from ce6af4f to c050276 Compare April 26, 2019 19:07

sebasguts force-pushed the sg/syntaxtree branch from 9d0f69e to 3ade001 Compare May 4, 2019 11:25

sebasguts force-pushed the sg/syntaxtree branch from 3ade001 to 812dc61 Compare May 4, 2019 12:11

fingolfin requested changes May 6, 2019

View reviewed changes

sebasguts added 7 commits May 7, 2019 15:08

Added additional parameter to CodeFuncExprEnd

b9888a6

to indicate that the statement stack is complete and does not need any more additions.

Add additional parameter to CodeLazyFloatExpr

7b96a68

Changed name of PushValue to AddValueToBody

7f15263

Renamed NewStatWithProf into NewStatOrExpr

8f8b912

Added RequirePlainRec to error.h

8aa13e1

Finished first version of Syntax tree coder

4057770

Added test function to compare syntax trees

3c4614a

sebasguts force-pushed the sg/syntaxtree branch from 812dc61 to 3c4614a Compare May 7, 2019 13:10

fingolfin approved these changes May 7, 2019

View reviewed changes

fingolfin added kind: enhancement Label for issues suggesting enhancements; and for pull requests implementing enhancements kind: new feature release notes: to be added PRs introducing changes that should be (but have not yet been) mentioned in the release notes topic: kernel labels May 7, 2019

fingolfin merged commit 5ecdc93 into gap-system:master May 7, 2019

fingolfin added release notes: added PRs introducing changes that have since been mentioned in the release notes and removed release notes: to be added PRs introducing changes that should be (but have not yet been) mentioned in the release notes labels Aug 22, 2019

olexandr-konovalov added this to the GAP 4.11.0 milestone Aug 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a coder to reread syntaxtrees #3371

Add a coder to reread syntaxtrees #3371

sebasguts commented Mar 21, 2019

fingolfin commented Mar 26, 2019

fingolfin left a comment

codecov bot commented Mar 27, 2019 •

edited

Loading

sebasguts commented Apr 4, 2019

coveralls commented Apr 4, 2019 •

edited

Loading

sebasguts commented Apr 11, 2019

sebasguts commented Apr 11, 2019

sebasguts commented Apr 26, 2019

sebasguts commented May 3, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

sebasguts commented May 4, 2019

fingolfin May 6, 2019

fingolfin May 6, 2019

fingolfin May 6, 2019

fingolfin May 6, 2019

sebasguts May 6, 2019

sebasguts May 6, 2019

	> Print("failed round trip: ",n,"\n");
	> Print("METHODS_OPERATION(", n, ",", i, ") failed round trip\n");

Add a coder to reread syntaxtrees #3371

Add a coder to reread syntaxtrees #3371

Conversation

sebasguts commented Mar 21, 2019

fingolfin commented Mar 26, 2019

fingolfin left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 27, 2019 • edited Loading

Codecov Report

sebasguts commented Apr 4, 2019

coveralls commented Apr 4, 2019 • edited Loading

sebasguts commented Apr 11, 2019

sebasguts commented Apr 11, 2019

sebasguts commented Apr 26, 2019

sebasguts commented May 3, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

ChrisJefferson commented May 4, 2019

sebasguts commented May 4, 2019

sebasguts commented May 4, 2019

fingolfin May 6, 2019

Choose a reason for hiding this comment

fingolfin May 6, 2019

Choose a reason for hiding this comment

fingolfin May 6, 2019

Choose a reason for hiding this comment

fingolfin May 6, 2019

Choose a reason for hiding this comment

sebasguts May 6, 2019

Choose a reason for hiding this comment

sebasguts May 6, 2019

Choose a reason for hiding this comment

codecov bot commented Mar 27, 2019 •

edited

Loading

coveralls commented Apr 4, 2019 •

edited

Loading