update Cremona's tables for Sage #11587

I have a few questions. Also since I do not know SQLITE and do not have the time to learn it just for this, the code will need to be reviewed by someone else.

There are two spkgs at the link, and one patch on this ticket. Please can you explain how they are related? Is one of the spkgs intended to be part of the standard Sage distribution so that every copy of Sage has the mini-database, as is the case now? And is the other one a replacement for the optional large database?

Secondly, you warn that the database takes some time to create. Is this once-only time which you had to spend in creating the database which is part of the spkg, or is it time which must be spent by anyone installing the large optional spkg?

To test this, I assume that the following steps are necessary; please correct me if I am wrong.

apply patches at Rewrite/improve/fix SQLDatabase and SQLQuery objects #11642
apply the patch here (trac_11587.patch)
sage -b
sage -i http://wstein.org/home/ohanar/cremona-database/elliptic_curves-0.2.spkg
sage -b, test everything works
sage -i http://wstein.org/home/ohanar/cremona-database/database_cremona_ellcurve-20110801.spkg
sage -b, test everything still works

Lastly: I expect that at regular intervals over the next few months I will be releasing updates to the database (for example, the range 170k=180k is about 70% done already). Please leave rather clear instructions for what to do for future updates; in particular, will it be neceaary to rebuild everything, or just to read in the new files (e.g. *.170000-179999 next time)?

ohanar · 2011-08-02T18:17:38Z

comment:6

Replying to @JohnCremona:

There are two spkgs at the link, and one patch on this ticket. Please can you explain how they are related?

The patch updates sage.databases.cremona to wrap the new SQLite databases. It provides about the same functionality as the old ZODB (there are some slight modifications to actually reflect the documentation, which wasn't respected before -- specifically the mini-database only has data from allcurves).

Is one of the spkgs intended to be part of the standard Sage distribution so that every copy of Sage has the mini-database, as is the case now? And is the other one a replacement for the optional large database?

Correct. The elliptic_curve spkg is included every copy of sage, and includes both the mini-database, as well as a very small database of curves William finds interesting. The database_cremona_ellcurve spkg includes (most of) the data from allcurves, allgens, allbsd, and degphi for every curve that you have listed.

Secondly, you warn that the database takes some time to create. Is this once-only time which you had to spend in creating the database which is part of the spkg, or is it time which must be spent by anyone installing the large optional spkg?

Currently the spkgs simply install pre-built SQL databases, so currently this is a once-only thing. That said, on my home system it only takes ~4mins to create the database from scratch.

To test this, I assume that the following steps are necessary; please correct me if I am wrong.

apply patches at Rewrite/improve/fix SQLDatabase and SQLQuery objects #11642
apply the patch here (trac_11587.patch)
sage -b
sage -i http://wstein.org/home/ohanar/cremona-database/elliptic_curves-0.2.spkg
sage -b, test everything works
sage -i http://wstein.org/home/ohanar/cremona-database/database_cremona_ellcurve-20110801.spkg
sage -b, test everything still works

Yes, that looks correct. The only thing to be aware of is that #11642 depends on #11640 (which is just a small patch), and that there are optional doctests you should run in step 7.

Lastly: I expect that at regular intervals over the next few months I will be releasing updates to the database (for example, the range 170k=180k is about 70% done already). Please leave rather clear instructions for what to do for future updates; in particular, will it be neceaary to rebuild everything, or just to read in the new files (e.g. *.170000-179999 next time)?

Currently there is simple script to rebuild everything, but (in theory) it is also possible to simply add the new files. The former:

rm SAGE_ROOT/data/cremona/cremona.db
SAGE_ROOT/sage
sage: sage.databases.cremona.build('cremona','/path/to/ecdata.tgz')

the later should be (although I haven't tested it):

make a directory that only contains the new files
SAGE_ROOT/sage
sage: cData = sage.databases.cremona.LargeCremonaDatabase('cremona',False)
sage: cData._init_from_ftpdata('/path/to/said/directory')

That said, I've marked myself as the maintainer of the database_cremona_ellcurve spkg, so I'll try to keep it up to date (of course it would help if you let me know when you update your tables of course (: ).

ohanar · 2011-08-10T03:02:16Z

comment:7

I've updated the database_cremona_ellcurve spkg to include the latest tables, which can still be found at http://wstein.org/home/ohanar/cremona-database (the old one is still there there as well).

I'm also attaching an updated patch that mainly updates the documentation to reflect the current state of the tables (upper bound of 180,000 vs 130,000), as well as the spkg version. It also limits the outputs of a few queries where only the first entry is needed.

ohanar · 2011-08-24T23:52:34Z

fixes the broken doctest for cremona_mini

boothby · 2011-08-25T08:51:12Z

comment:8

Attachment: trac_11587.patch.gz

looks good, works good

boothby · 2011-08-25T08:51:12Z

Author: Andrew Ohana

boothby · 2011-08-25T08:51:12Z

Changed reviewer from John Cremona to John Cremona, Tom Boothby

nexttime · 2011-09-08T14:31:07Z

Dependencies: #11642

nexttime · 2011-09-08T14:31:07Z

comment:9

That's +1.4 MB, or +78% for the standard spkg.

nexttime · 2011-09-08T14:31:07Z

Changed keywords from none to elliptic curves ellcurve database

JohnCremona · 2011-10-09T16:25:04Z

comment:51

Replying to @nexttime:

Replying to @JohnCremona:

Any effort in removing this upgrade which takes more than the few minutes it would take to fix the very minor problem in it would be a complete waste of time.

I fully agree, but unfortunately it's relatively easy to "unmerge" this ticket, and it's in general not up to the release manager to fix such things (that go beyond simple rebasing, fixing noise, permissions or commit messages etc., which IMHO at least partially already is a voluntary service).

Of course -- I just meant that you should do the absolute minimum and not spend effort making the old system more clever. And I just have not had time to fix the patch.

loefflerd · 2011-10-16T18:41:09Z

apply over previous patches

loefflerd · 2011-10-16T18:41:50Z

comment:52

Attachment: trac_11587-fix_speed_regression.patch.gz

JohnCremona · 2011-10-16T19:23:52Z

comment:53

Replying to @loefflerd:

Not tested but David's patch looks perfect and I believe his comment. Better if someone else test this as David and I use the same machine.

If this is not merged soon it will become out of date since I have currently reached 198800 and expect to release an extension up to 200k within a week!

ohanar · 2011-10-30T00:51:30Z

comment:54

Everything works as expected.

@loefflerd
I don't think that those lines of code were actually ever used in the old version, since the small database actually included the generators despite the documentation saying otherwise (the new version follows what the documentation specified).

@JohnCremona
I've posted a new spkg with your expanded tables, same location as before.

JohnCremona · 2011-10-30T11:21:25Z

comment:55

Replying to @ohanar:

Everything works as expected.

@loefflerd
I don't think that those lines of code were actually ever used in the old version, since the small database actually included the generators despite the documentation saying otherwise (the new version follows what the documentation specified).

@JohnCremona
I've posted a new spkg with your expanded tables, same location as before.

Many thanks to all. As predicted, the database was updated to go up to 200k a few days ago.

jdemeyer · 2011-10-31T11:27:38Z

Merged: sage-4.7.3.alpha0

JohnCremona · 2011-11-03T13:55:13Z

comment:58

I just updated the description to reflect the fact that the optional db now goes up to 200k.

jdemeyer · 2011-11-03T14:10:28Z

comment:59

Replying to @ohanar:

I've posted a new spkg with your expanded tables, same location as before.

Not really same location as before, I updated the ticket description.

jdemeyer · 2011-11-03T16:14:43Z

Milestone sage-4.7.3 deleted

jdemeyer · 2011-11-03T16:17:24Z

Changed merged from sage-4.7.3.alpha0 to sage-4.8.alpha0

ohanar · 2011-11-27T11:36:29Z

comment:62

I've posted an updated spkg based on Cremona's update to 210000 (http://wstein.org/home/ohanar/cremona-database/database_cremona_ellcurve-20111121.spkg).

I noticed that I never put in a way to install the old spkg for old copies of sage, so I added a recursive sage -i for now, until we have a standard that frees the user from downloading two different versions of an spkg. It should work for any version of sage released within the last year, although I haven't tested it super thoroughly (it works with stable 4.7.x, 4.8.alpha3, and 4.3.3.alpha0).

jdemeyer · 2011-11-27T20:44:04Z

comment:63

Replying to @ohanar:

I've posted an updated spkg based on Cremona's update to 210000 (http://wstein.org/home/ohanar/cremona-database/database_cremona_ellcurve-20111121.spkg).

Please make a new ticket for making this an optional spkg (this ticket is closed).

ohanar · 2011-11-28T02:10:32Z

comment:64

Replying to @jdemeyer:

Please make a new ticket for making this an optional spkg (this ticket is closed).

Sure, my only reason for posting it in this ticket was due to the unaddressed issue of dealing with old copies of Sage.

JohnCremona · 2011-12-14T14:54:22Z

comment:65

Doing a full test after installing the new optional database (onto 4.8.alpha4) I find one doctest failure in devel/sage/doc/en/bordeaux_2008/elliptic_curves.rst:

    sage: E = EllipticCurve([1,2,5,7,17])
    sage: E.conductor()       # not in the Tables
    154907
    sage: E.gens()            # a few seconds
    [(1 : 3 : 1), (67/4 : 507/8 : 1)]

since that curve is now in the database and the listed generators are different. I suggest instead

sage: E = EllipticCurve([0,0,1,-19,60])
sage: E.conductor()        # not in the Tables
1129211
sage: E.gens()             # a few seconds
[(-5 : 5 : 1), (-4 : 8 : 1), (-15/4 : 67/8 : 1), (-3 : 9 : 1)]

Should the ticket be reopened -- this is not an issue in testing without the optional db installed?

JohnCremona · 2011-12-18T17:28:59Z

comment:66

Replying to @JohnCremona:

Should the ticket be reopened -- this is not an issue in testing without the optional db installed?

See #12184

williamstein added this to the sage-4.7.2 milestone Jul 11, 2011

williamstein added c: elliptic curves labels Jul 11, 2011

williamstein assigned JohnCremona Jul 11, 2011