gap: support gap 4.13.1 in sagelib (does NOT touch sage-the-distro) #37884

tornaria · 2024-04-27T18:56:51Z

In gap 4.13 there are some improvements, e.g. converting fp groups to permutation groups, computing abelianization of fp groups, which lead to different generators.

This PR fixes doctests so they pass using gap 4.13.

I don't see a way to have this work with gap 4.12.2 at the same time, so this is not in principle ready to merge.

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.

⌛ Dependencies

#37883: don't use deprecated LaTeX() gap function.

github-actions · 2024-04-28T04:05:52Z

Documentation preview for this PR (built with commit 3faaa24; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

tornaria · 2024-04-28T15:53:03Z

The failures here are real:

----------------------------------------------------------------------
sage -t --random-seed=130925441006892428060280565126983766193 src/sage/categories/simplicial_sets.py  # 7 doctests failed
sage -t --random-seed=130925441006892428060280565126983766193 src/sage/groups/finitely_presented.py  # 5 doctests failed
sage -t --random-seed=130925441006892428060280565126983766193 src/sage/groups/perm_gps/permgroup_named.py  # 3 doctests failed
----------------------------------------------------------------------

and the reason why this PR can't be merged if gap is not updated to 4.13.

But really, either output (the one with gap 4.12 or the one with 4.13) is just fine. Hence, updating gap to 4.13 in sage-the-distro and apply this PR at the same time, is a pain for using system gap (either we restrict the version of gap that is allowed from system, or we end up with failing doctests on some systems).

We really need to come up with a system to be able to support different versions of dependencies in doctests... at this time we have a similar problem with the difference between singular 4.3 and 4.4 (which we worked around with an ugly hack of patching the output of singular when running via doctests!).

tscrim · 2024-04-28T22:50:26Z

We had specific tags for Python 2 and 3 differences. Since there are only a few differences (some of which are really not easy to rectify by changing the doctests), one option might be to introduce such tags for GAP versions. Or we just tell people that some tests might not pass for trivial reasons if you do not have a sufficiently updated version of GAP...

tornaria · 2024-04-29T02:58:30Z

We had specific tags for Python 2 and 3 differences. Since there are only a few differences (some of which are really not easy to rectify by changing the doctests), one option might be to introduce such tags for GAP versions. Or we just tell people that some tests might not pass for trivial reasons if you do not have a sufficiently updated version of GAP...

Here's one idea to think about:

tag the test with, say, "gap-4.12.2" to indicate that the output corresponds to gap version 4.12.2 (this is the current version in sage-the-distro).
somewhere we have some kind of "translation" file (akin to "po" files in gettext) in which have alternate versions of doctest outputs for different versions.

A quick example with one test (very crude first try):

In src/sage/algebras/fusion_rings/fusion_double.py we have

        sage: b13^2 # long time (4s), gap-4.12.2
        b0 + b2 + b4 + b15 + b16 + b17 + b18 + b24 + b26 + b27

The traslation file for gap has something like:

#: src/sage/algebras/fusion_rings/fusion_double.py:134
msgid b0 + b2 + b4 + b15 + b16 + b17 + b18 + b24 + b26 + b27
msgstr[4.13.0] b0 + b3 + b4

I'm not sure if the doctest label has to include the gap version or even the program name (maybe just have a common tag to mark the test to be applied this mechanism).

tornaria · 2024-04-29T03:04:59Z

Here's another (maybe simpler?) idea. See src/sage/arith/long.pxd where we have this doctest:

        sage: a^(2**258)
        Traceback (most recent call last):
        ...
        OverflowError: exponent must be at most 2147483647           # 32-bit
        OverflowError: exponent must be at most 9223372036854775807  # 64-bit

so the output is labeled here... If we could have labels e.g "gap < 4.13" and "gap >= 4.13" or something like this... Would that we workable?

tscrim · 2024-05-10T04:47:38Z

Sorry for loosing track of this. Yes, I think having something as a tag on the output with, e.g., # gap < 4.13 is a good solution. It is clear to the reader and to implement in the tester. This is what I had in mind.

mkoeppe · 2024-06-07T17:27:10Z

Or, for simplicity, how about we just update gap to 4.13 and require the same version from the system.

RIght now, only >= 4.12.2 is accepted, but there's not really anything relevant that actually has 4.12.2 but not 4.13 - https://repology.org/project/gap/versions

mkoeppe · 2024-06-07T18:44:55Z

I've opened #38169 for the update, based on this PR.

mkoeppe · 2024-06-09T19:00:43Z

Let's close this in favor of #38169

@tornaria

- Based on sagemath#37884 by @tornaria - Fixes sagemath#37616 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    - Depends on sagemath#38144 (merged here for testing) URL: sagemath#38169 Reported by: Matthias Köppe Reviewer(s):

dimpase · 2024-06-14T20:07:44Z

@mkoeppe, I am reopening this, as I cannot leave comments on #38169

I see a serious slowdown at startup by libgap 4.13.0. And lots of timeouts while testing.e.g. src/sage/features/gap.py
(it hangs while testing for presence of a GAP package)

mkoeppe · 2024-06-14T20:38:06Z

Then perhaps for the upcoming release we should reject system gap 4.13

enriqueartal · 2024-06-14T21:44:37Z

@mkoeppe, I am reopening this, as I cannot leave comments on #38169

I see a serious slowdown at startup by libgap 4.13.0. And lots of timeouts while testing.e.g. src/sage/features/gap.py (it hangs while testing for presence of a GAP package)

I had also these issues with system gap 4.13.0

mkoeppe · 2024-06-15T03:12:14Z

perhaps for the upcoming release we should reject system gap 4.13

Happening in #38222

tornaria · 2024-09-22T19:32:40Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine. The doctests in sage math fail only because they test something that is not part of the interface (generators, order, etc).

As an example, void-linux/void-packages#51902 is using this PR. We are also shipping sagemath 10.4 with this.

enriqueartal · 2024-09-22T19:48:47Z

Which other tests fail? In a fresh installation of 10.5beta5 with gap_packages installed, the only gap-related failed tests can be apparently solved with an upstream approved patch. I do not know if there are some failures related to other optional packages. I checked 4.13 with #37128 and the only failures were related with reorderings of generators, math results are correct.

dimpase · 2024-09-22T19:55:44Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine.

Do you mean to say that on void you somehow could not reproduce
gap-system/gap#5786 ?

This would be rather strange.

The doctests in sage math fail only because they test something that is not part of the interface (generators, order, etc).

As an example, void-linux/void-packages#51902 is using this PR. We are also shipping sagemath 10.4 with this.

tornaria · 2024-09-23T01:28:28Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine.

Do you mean to say that on void you somehow could not reproduce gap-system/gap#5786 ?

This would be rather strange.

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

dimpase · 2024-09-23T08:39:45Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine.

Do you mean to say that on void you somehow could not reproduce gap-system/gap#5786 ?

This would be rather strange.

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

by "this PR", do you mean the GAP upstream PR ?

tornaria · 2024-09-23T21:41:14Z

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

by "this PR", do you mean the GAP upstream PR ?

"This PR" means #37884 (i.e. the one you are currently looking at).

FWIW, I'm shipping gap 4.13.1 unpatched.

dimpase · 2024-09-23T22:07:42Z

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

by "this PR", do you mean the GAP upstream PR ?

"This PR" means #37884 (i.e. the one you are currently looking at).

FWIW, I'm shipping gap 4.13.1 unpatched.

Could you please double-check that you are not linking the sagelib with libgap.so from GAP 4.12.2 ?

antonio-rojas · 2024-09-24T06:19:20Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine.

Do you mean to say that on void you somehow could not reproduce gap-system/gap#5786 ?
This would be rather strange.

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

The failing tests are optional on gap_package_grape. You won't reproduce it if you don't have gap-packages installed.

dimpase · 2024-09-24T11:27:02Z

I'm lost here. Both gap 4.13 and 4.13.1 work just fine.

Do you mean to say that on void you somehow could not reproduce gap-system/gap#5786 ?
This would be rather strange.

I mean sagemath seem to be working fine, and all doctests pass, as long as the changes in this PR are applied.

The failing tests are optional on gap_package_grape. You won't reproduce it if you don't have gap-packages installed.

that's not quite true. All you need is a GAP package GRAPE installed.
E.g. on Gentoo it can be installed separately, and currently I'm getting these errors with the unpatched system-wide GAP 4.13.1.
(on other systems, e.g. with GAP as supplied by Sage, it can be installed with GAP's PackageManager)

@orlitzky - is there a Gentoo bug filed for this?

It's also not clear to me why Gentoo does not package GAP's PackageManager.

orlitzky · 2024-09-24T14:11:20Z

that's not quite true. All you need is a GAP package GRAPE installed. E.g. on Gentoo it can be installed separately, and currently I'm getting these errors with the unpatched system-wide GAP 4.13.1. (on other systems, e.g. with GAP as supplied by Sage, it can be installed with GAP's PackageManager)

@orlitzky - is there a Gentoo bug filed for this?

No, I hadn't realized it was an issue with GRAPE, although in hindsight I had probably installed dev-gap/grape in order to keyword it ~riscv.

It's also not clear to me why Gentoo does not package GAP's PackageManager.

Because you should use the Gentoo packages instead :)

My usual tirade about toy package managers being insufficient and insecure, but also they tend to break system packages in unexpected ways. We have two examples of GAP packages (Browse and now GRAPE) that can mess things up by being installed. I'm sure there would be more if users could install any untested package locally.

dimpase · 2024-09-24T14:36:00Z

There is no issue with GRAPE per se. The issue is with core GAP 4.13.[0-1], which, with GRAPE installed, causes test failures.
The linked core GAP PR fixes the GAP issue, so you might want to add the corresponding patch.

orlitzky · 2024-09-24T14:43:08Z

Ok, thanks for the heads up. I also had the failure mentioned in #38169 (comment) the last time I tried, but now that you have made me think about it, that too is probably caused by some other package.

I'm leaving for a conference in a minute but I'll try to sort it out next week and will probably add that patch in Gentoo.

enriqueartal · 2024-09-24T15:35:11Z

I used gap upstream patch and now I do not have gap-related failure tests as predicted. I would be for a positive review (the patch must be added) but maybe I am skipping something.

dimpase · 2024-09-24T18:07:25Z

that's not quite true. All you need is a GAP package GRAPE installed. E.g. on Gentoo it can be installed separately, and currently I'm getting these errors with the unpatched system-wide GAP 4.13.1. (on other systems, e.g. with GAP as supplied by Sage, it can be installed with GAP's PackageManager)
@orlitzky - is there a Gentoo bug filed for this?

No, I hadn't realized it was an issue with GRAPE, although in hindsight I had probably installed dev-gap/grape in order to keyword it ~riscv.

It's also not clear to me why Gentoo does not package GAP's PackageManager.

Because you should use the Gentoo packages instead :)

My usual tirade about toy package managers being insufficient and insecure, but also they tend to break system packages in unexpected ways. We have two examples of GAP packages (Browse and now GRAPE) that can mess things up by being installed. I'm sure there would be more if users could install any untested package locally.

GRAPE has not messed up anything. It is used in several places in Sage, and these places are not tested if GRAPE is not installed. This is not what "breaking" means, at all.

Sorry, but GAP is a living, extendable, system with people developing packages (I wrote or co-wrote a few)! And PackageManager is the easiest way to try them out. It installs things into ~/.gap/. I suppose you don't mean that I must start by making a Gentoo package out of any GAP package I'd like to try on a Gentoo box?!
There certainly should be GAP PackageManager as a Gentoo package.
Just like there is pip for python.

orlitzky · 2024-10-09T12:07:48Z

I've applied gap-system/gap#5796, but there are still failures with GRAPE, e.g.

File "src/sage/graphs/generators/smallgraphs.py", line 5666, in sage.graphs.generators.smallgraphs.U42Graph216
Failed example:
    G=graphs.U42Graph216()                    # optional - gap_package_grape
Exception raised:
    Traceback (most recent call last):
      File "/home/mjo/src/sage.git/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/mjo/src/sage.git/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.graphs.generators.smallgraphs.U42Graph216[0]>", line 1, in <module>
        G=graphs.U42Graph216()                    # optional - gap_package_grape
          ^^^^^^^^^^^^^^^^^^^^
      File "/home/mjo/src/sage.git/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/sage/graphs/generators/smallgraphs.py", line 5693, in U42Graph216
        adj = adj_list()  # for each vertex, we get the list of vertices it is adjacent to
              ^^^^^^^^^^
      File "sage/libs/gap/element.pyx", line 2520, in sage.libs.gap.element.GapElement_Function.__call__ (build/cythonized/sage/libs/gap/element.c:27814)
        sig_on()
    sage.libs.gap.util.GAPError: Error, no method found! Error, no 1st choice method found for `+' on 2 arguments
    The 2nd argument is 'fail' which might point to an earlier problem

The GAP example provided by @dimpase does work though, so something was fixed:

$ sage -gap
 ┌───────┐   GAP 4.13.1 of 2024-06-11
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: amd64
 Configuration:  gmp 6.3.0, GASMAN, readline
 Loading the library and packages ...
 Packages:   GAPDoc 1.6.7, PrimGrp 3.4.4, SmallGrp 1.5.3, TransGrp 3.6.5
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> gg:=SpecialUnitaryGroup(4,2);
SU(4,2)
gap> hl:=Z(2)*[
>                 [0,0,1,0],
>                 [1,1,0,0],
>                 [0,1,0,1],
>                 [0,1,1,0],
>                 [1,1,0,1]];
[ [ 0*Z(2), 0*Z(2), Z(2)^0, 0*Z(2) ], [ Z(2)^0, Z(2)^0, 0*Z(2), 0*Z(2) ], 
  [ 0*Z(2), Z(2)^0, 0*Z(2), Z(2)^0 ], [ 0*Z(2), Z(2)^0, Z(2)^0, 0*Z(2) ], 
  [ Z(2)^0, Z(2)^0, 0*Z(2), Z(2)^0 ] ]
gap> o216:=Orbit(gg,Set(hl),OnSets);
[ [ [ 0*Z(2), 0*Z(2), Z(2)^0, 0*Z(2) ], [ 0*Z(2), Z(2)^0, 0*Z(2), Z(2)^0 ], 
      [ 0*Z(2), Z(2)^0, Z(2)^0, 0*Z(2) ], [ Z(2)^0, Z(2)^0, 0*Z(2), 0*Z(2) ], 
      [ Z(2)^0, Z(2)^0, 0*Z(2), Z(2)^0 ] ], 
...

enriqueartal · 2024-10-09T12:53:59Z

In Fedora 40 with gap and gap_packages 4.13.1, this order works. Actually I needed to apply a couple of patches since gap-system/gap#5796 depended on another one.

orlitzky · 2024-10-09T13:32:28Z

In Fedora 40 with gap and gap_packages 4.13.1, this order works. Actually I needed to apply a couple of patches since gap-system/gap#5796 depended on another one.

I rebuilt sage from scratch, and now it works? So the GRAPE problem is gone, but now I have to wonder why updates to the system libgap are not noticed by sage...

enriqueartal · 2024-10-09T13:34:10Z

Actually I am not using system packages because I got plenty of errors, which disappiear with the spkg.

orlitzky · 2024-10-09T13:42:30Z

Well I committed the patch for GAP in Gentoo, now it's a Sage problem :P

I'm going to focus on the remaining test failures in #38169 today. It would be criminal if we released another version of Sage with GAP 4.12.2.

dimpase · 2024-10-09T13:59:42Z

I'm under impression that Sage libgap interface is more flaky with GAP 4.13

dimpase · 2024-10-09T15:09:29Z

In Fedora 40 with gap and gap_packages 4.13.1, this order works. Actually I needed to apply a couple of patches since gap-system/gap#5796 depended on another one.

I rebuilt sage from scratch, and now it works? So the GRAPE problem is gone, but now I have to wonder why updates to the system libgap are not noticed by sage...

must be another copy of GAP somewhere.

orlitzky · 2024-10-09T15:17:07Z

must be another copy of GAP somewhere.

No such luck. I only ever have two copies of GAP, the Gentoo one, and the SPKG. This was after a git clean -x -f -d to eliminate the SPKG, so only the Gentoo version (that I updated) was left.

I'm telling myself it has something to do with the saved workspace so that I can move on.

dimpase · 2024-10-09T16:17:52Z

yeah, saved workspace. They usually live in ~/.sage/

The latter keeps creating unpleasant surprises for me.

Follow-up to: * sagemath#37884 * sagemath#38169 With the four additional work items I mentioned in a comment on the latter: 1. Everything has been rebased 2. There's a new feature to detect the polenta GAP package 3. The three failing simplicial sets tests have been marked `# needs gap_package_polenta` 4. I backported gap-system/gap#5796 to the GAP spkg so that the optional GRAPE tests will pass (untested). Let's see what the CI has to say... URL: sagemath#38804 Reported by: Michael Orlitzky Reviewer(s): Dima Pasechnik, Enrique Manuel Artal Bartolo

orlitzky · 2024-10-28T15:35:30Z

Done as part of #38804

tornaria requested review from enriqueartal, miguelmarco, tscrim and antonio-rojas April 27, 2024 18:57

tornaria added the s: needs review label Apr 27, 2024

tornaria force-pushed the gap-4.13 branch from 7a30869 to c49562e Compare May 3, 2024 00:12

mkoeppe added this to the sage-10.4 milestone May 14, 2024

enriqueartal mentioned this pull request May 24, 2024

Save and load finitely presented groups coming from libgap groups #37128

Merged

5 tasks

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

mkoeppe added the c: group theory label Jun 7, 2024

mkoeppe closed this Jun 9, 2024

github-actions bot removed the s: needs review label Jun 9, 2024

dimpase reopened this Jun 13, 2024

github-actions bot added the v: minimal label Jun 13, 2024

mkoeppe modified the milestones: sage-10.4, sage-10.5 Jun 15, 2024

orlitzky mentioned this pull request Oct 12, 2024

GAP 4.13.1 #38804

Merged

orlitzky closed this Oct 28, 2024

github-actions bot removed the s: needs review label Oct 28, 2024

gap: support gap 4.13.1 in sagelib (does NOT touch sage-the-distro) #37884

gap: support gap 4.13.1 in sagelib (does NOT touch sage-the-distro) #37884

Conversation

tornaria commented Apr 27, 2024

📝 Checklist

⌛ Dependencies

github-actions bot commented Apr 28, 2024 • edited Loading

tornaria commented Apr 28, 2024

tscrim commented Apr 28, 2024

tornaria commented Apr 29, 2024

tornaria commented Apr 29, 2024

tscrim commented May 10, 2024

mkoeppe commented Jun 7, 2024

mkoeppe commented Jun 7, 2024

mkoeppe commented Jun 9, 2024

dimpase commented Jun 14, 2024

mkoeppe commented Jun 14, 2024

enriqueartal commented Jun 14, 2024

mkoeppe commented Jun 15, 2024

tornaria commented Sep 22, 2024

enriqueartal commented Sep 22, 2024

dimpase commented Sep 22, 2024

tornaria commented Sep 23, 2024

dimpase commented Sep 23, 2024

tornaria commented Sep 23, 2024

dimpase commented Sep 23, 2024

antonio-rojas commented Sep 24, 2024

dimpase commented Sep 24, 2024 • edited Loading

orlitzky commented Sep 24, 2024

dimpase commented Sep 24, 2024

orlitzky commented Sep 24, 2024

enriqueartal commented Sep 24, 2024 • edited Loading

dimpase commented Sep 24, 2024

orlitzky commented Oct 9, 2024

enriqueartal commented Oct 9, 2024

orlitzky commented Oct 9, 2024

enriqueartal commented Oct 9, 2024

orlitzky commented Oct 9, 2024

dimpase commented Oct 9, 2024

dimpase commented Oct 9, 2024

orlitzky commented Oct 9, 2024

dimpase commented Oct 9, 2024 • edited Loading

orlitzky commented Oct 28, 2024

github-actions bot commented Apr 28, 2024 •

edited

Loading

dimpase commented Sep 24, 2024 •

edited

Loading

enriqueartal commented Sep 24, 2024 •

edited

Loading

dimpase commented Oct 9, 2024 •

edited

Loading