Handle lists being modified by use of ~ #2477

ChrisJefferson · 2018-05-17T17:25:46Z

Adjust how lists are built when ~ is used, so we do not crash
even if the list is modified as it is being constructed.

This also makes the behaviour of GAP consistent between bytecode
and immediate code.

codecov · 2018-05-17T19:09:52Z

Codecov Report

Merging #2477 into master will decrease coverage by <.01%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master    #2477      +/-   ##
==========================================
- Coverage   74.27%   74.27%   -0.01%     
==========================================
  Files         484      484              
  Lines      245311   245311              
==========================================
- Hits       182205   182200       -5     
- Misses      63106    63111       +5

Impacted Files	Coverage Δ
src/exprs.c	`93.96% <91.66%> (-0.18%)`	⬇️
hpcgap/lib/hpc/stdtasks.g	`63.68% <0%> (-1.03%)`	⬇️

fingolfin

Please set labels, and format the code

Otherwise looks good to me!

fingolfin · 2018-05-17T19:40:36Z

src/exprs.c

@@ -919,9 +919,18 @@ Obj             EvalPermExpr (
 **
 **  'EvalListExpr'  just  calls 'ListExpr1'  and  'ListExpr2' to evaluate the
 **  list expression.
+**
+**  The 'safe' argument to ListExpr1 and ListExpr2 handle code which abuses ~
+**  access and change the partially built list.


handle -> handles ? though I don't see how an argument can "handle" anything by itself?

fingolfin · 2018-05-17T19:41:12Z

src/exprs.c

+**  access and change the partially built list.
+**
+**  When 'safe' is 1, we use a slower code path which ensures the list is
+**  in a valid state after each element is added, and not crash if the


missing words... perhaps "... is added, and that we don't crash ..."

fingolfin · 2018-05-17T19:42:00Z

src/exprs.c

@@ -999,8 +1008,9 @@ Obj             EvalListTildeExpr (
 **  '[ [1], ~[1] ]'  requires that the value of  one subexpression is entered
 **  into the list value before the next subexpression is evaluated.
 */
-Obj ListExpr1 (
-    Expr                expr )
+ALWAYS_INLINE Obj ListExpr1 (


Why ALWAYS_INLINE?

Will fix all other comments.

With this one, ALWAYS_INLINE because I want the 'safe' compiled out when safe=0, so the fast route (without tilde) is as fast as it used to be.

fingolfin · 2018-05-17T19:42:10Z

src/exprs.c

@@ -1015,15 +1025,21 @@ Obj ListExpr1 (
    else {
        list = NEW_PLIST( T_PLIST, len );
    }
-    SET_LEN_PLIST( list, len );
+    if(safe) {


add space after if

fingolfin · 2018-05-17T19:42:36Z

src/exprs.c

@@ -1053,17 +1069,25 @@ void ListExpr2 (
          {
            if (posshole == 1)
              {
-                SET_FILT_LIST(list, FN_IS_NDENSE);
+                if(!safe || TNUM_OBJ(list) == T_PLIST) {


add space after if (or just git clang-format)

fingolfin · 2018-05-17T19:44:14Z

tst/testinstall/tilde.tst

+gap> (function() return [Length(~),Length(~),Length(~)]; end)();
+[ 0, 1, 2 ]
+gap> rem := function(l) Remove(l); return 1; end;;
+gap> [2,rem(~),rem(~),rem(~)];


Wow, that's pretty evil :-)

I agree. ^^

I don't expect anyone to ever write this, but I feel there shouldn't be ways of making the interpreter crash (at least, without calling ALL_CAPS functions).

Yeah, sure, it's good to add that test. People do evil things, we shouldn't reward them for it with a crash ;-)

ChrisJefferson · 2018-05-18T15:57:02Z

Hopefully changes made, added to release notes.

fingolfin · 2018-05-22T07:16:26Z

src/exprs.c

@@ -919,9 +919,19 @@ Obj             EvalPermExpr (
 **
 **  'EvalListExpr'  just  calls 'ListExpr1'  and  'ListExpr2' to evaluate the
 **  list expression.
+**
+**  The 'safe' argument to ListExpr1 and ListExpr2 should be set to 1 when
+**  the list expression may be modified when its elements are being


"when its elements are being constructed" -> "while its ..." ? (It took me a moment to parse the current phrasing, but then I did understand it, so it's probably fine as it is -- I just try to be super sensitive and try to simulate how I might read a paragraph if I knew less about the kernel; but of course that's imperfect. Perhaps e.g. @ssiccha would be willing to chime and say whether the above text seems clear to him -- hopefull it is, but if not, it'd be useful to know which bits are not.

@fingolfin @ChrisJefferson Apart from knowing what exactly an expression is (system.h says its a UInt, but that's all I found out quickly), the formulation is clear to me.

"when its elements are being constructed" -> "while its ..."

"while its" also sounds better to me

fingolfin · 2018-05-22T07:18:50Z

src/exprs.c

@@ -919,9 +919,19 @@ Obj             EvalPermExpr (
 **
 **  'EvalListExpr'  just  calls 'ListExpr1'  and  'ListExpr2' to evaluate the
 **  list expression.
+**
+**  The 'safe' argument to ListExpr1 and ListExpr2 should be set to 1 when


Shouldn't this text be part of the documentation comment of ListExpr1 and ListExpr2 instead?

I understand that there are prototypes for those two functions below, but considering the rest of their documentation is elsewhere... Perhaps we could move them up, before this function, to avoid using prototypes? Then they could also be made static.

fingolfin · 2018-05-22T07:25:25Z

src/exprs.c

@@ -1053,18 +1065,25 @@ void ListExpr2 (
          {
            if (posshole == 1)
              {
-                SET_FILT_LIST(list, FN_IS_NDENSE);
+                if (!safe || TNUM_OBJ(list) == T_PLIST) {


Hmm, is that safe? Couldn't the tnum be, say T_PLIST_DENSE at this point? That would suggest using if (!safe || IS_PLIST(list)) - or perhaps even IS_LIST?

But why is this necessary at all? SET_FILT_LIST uses a tnum dispatch table -- if the user did something really bad here, wouldn't that just trigger an error via that dispatch table?

You are right, let me rethink this.

fingolfin · 2018-05-22T07:27:31Z

src/exprs.c

+**  When 'safe' is 1, we use a slower code path which ensures the list is
+**  in a valid state after each element is added, and we don't crash if the
+**  size or TNUM of the list is changed during the construction of elements
+**  in the list.


Perhaps that should say "we don't crash if the size or type of the list changed, or even if gets swapped for a non-list object" ? After all, the user could use SWAP_OBJ to replace it by, say, a bigint.

OTOH, perhaps I am just overthinking that; and people using SWAP_OBJ deserve what they get ;-).

fingolfin · 2018-05-22T07:32:03Z

src/exprs.c

+**  the list expression may be modified when its elements are being
+**  constructed. This can occur when an element of the list refers to ~.
+**
+**  When 'safe' is 1, we use a slower code path which ensures the list is


Hmm, when I was reading the code below, I constantly had to refer back to this comment, because I could not remember whether safe meant: either "we are safe; it is safe to assume that the list won't be messed with", or perhaps "play it safe; use safe code which works even if the list gets messed with".

Since these are opposite, that's a bit of a problem, IMHO. Is it just me? Perhaps we can find a better name. Say protectAgainstListModification or safelyGrowList, safelyExtendList; or perhaps reverse the value, and say assumeListIsPlist or ...? Sorry, no great idea.

I could just call it tildeInUse, seeing as that's all we use it for :)

Yes, that actually sounds very good to me!

ChrisJefferson · 2018-05-24T21:44:44Z

Now changed/improved

fingolfin · 2018-05-25T11:48:42Z

src/exprs.c

 */
-Obj ListExpr1 (
-    Expr                expr )
+ALWAYS_INLINE Obj ListExpr1(Expr expr, Int safe)


The documentation now says tildeInUse, great -- but the code still uses safe :-)

fingolfin · 2018-05-25T11:52:01Z

src/exprs.c

+    }
+    else {
+        SET_LEN_PLIST(list, len);
+    }


Could also use SET_LEN_PLIST(list, tildeInUse ? 0 : len); for compactness -- but of course it's also perfectly fine as it is :-).

fingolfin · 2018-05-25T11:55:52Z

src/exprs.c

-      SET_FILT_LIST(list, FN_IS_DENSE);
-
+    if (!posshole && !safe) {
+        SET_FILT_LIST(list, FN_IS_DENSE);


Why do we need the && !safe (resp. && !tildeInUse) here? Isn't the list fully constructed at this point?

Ahh, because we might have used unbind while constructing it, through the use of ~? Is that the reason?
Perhaps a small comment could be added to point that out (or, if I am wrong, then point out the actual reason)?

I'm just assuming that SET_FILT_LIST is always an optimisation, and as you say, who knows what state the list is in, or if it's even a list.

Adjust how lists are built when ~ is used, so we do not crash even if the list is modified as it is being constructed. This also makes the behaviour of GAP consistent between bytecode and immediate code.

fingolfin requested changes May 17, 2018

View reviewed changes

ChrisJefferson force-pushed the tilde-crash branch from d0ba07a to 3c45d0f Compare May 18, 2018 15:54

ChrisJefferson added topic: kernel release notes: added PRs introducing changes that have since been mentioned in the release notes labels May 18, 2018

fingolfin reviewed May 22, 2018

View reviewed changes

ChrisJefferson force-pushed the tilde-crash branch 2 times, most recently from e4c2711 to 773e1f5 Compare May 24, 2018 21:44

fingolfin requested changes May 25, 2018

View reviewed changes

Handle lists being modified by use of ~

0a6ee3c

Adjust how lists are built when ~ is used, so we do not crash even if the list is modified as it is being constructed. This also makes the behaviour of GAP consistent between bytecode and immediate code.

ChrisJefferson force-pushed the tilde-crash branch from 773e1f5 to 0a6ee3c Compare May 25, 2018 12:03

fingolfin approved these changes May 25, 2018

View reviewed changes

fingolfin merged commit 38aacc5 into gap-system:master May 26, 2018

fingolfin mentioned this pull request Jun 5, 2018

Different results in direct and in functions when using ~ #2469

Closed

ChrisJefferson deleted the tilde-crash branch June 11, 2018 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle lists being modified by use of ~ #2477

Handle lists being modified by use of ~ #2477

ChrisJefferson commented May 17, 2018

codecov bot commented May 17, 2018 •

edited

Loading

fingolfin left a comment

fingolfin May 17, 2018

fingolfin May 17, 2018

fingolfin May 17, 2018

ChrisJefferson May 18, 2018

fingolfin May 18, 2018

fingolfin May 17, 2018

fingolfin May 17, 2018

fingolfin May 17, 2018

ssiccha May 17, 2018 •

edited

Loading

ChrisJefferson May 18, 2018

fingolfin May 18, 2018

ChrisJefferson commented May 18, 2018

fingolfin May 22, 2018

ssiccha May 22, 2018

ssiccha May 22, 2018 •

edited

Loading

fingolfin May 22, 2018

fingolfin May 22, 2018

ChrisJefferson May 22, 2018

fingolfin May 22, 2018

fingolfin May 22, 2018

ChrisJefferson May 22, 2018

fingolfin May 22, 2018

ChrisJefferson commented May 24, 2018

fingolfin May 25, 2018

ChrisJefferson May 25, 2018

fingolfin May 25, 2018

fingolfin May 25, 2018

ChrisJefferson May 25, 2018

Handle lists being modified by use of ~ #2477

Handle lists being modified by use of ~ #2477

Conversation

ChrisJefferson commented May 17, 2018

codecov bot commented May 17, 2018 • edited Loading

Codecov Report

fingolfin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssiccha May 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisJefferson commented May 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssiccha May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisJefferson commented May 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 17, 2018 •

edited

Loading

ssiccha May 17, 2018 •

edited

Loading

ssiccha May 22, 2018 •

edited

Loading