Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: experimental support for diskcache-style (sqlite backed) sconsign #4619

Closed
wants to merge 1 commit into from

Conversation

mwichmann
Copy link
Collaborator

@mwichmann mwichmann commented Oct 18, 2024

This is a low-touch conversion to use an sqlite backend instead of a flat "dblite" file for the sconsign database, without really changing SCons logic - but includes some hacks where rather than full detection, things are just commented out.

New SCons/sdiskcache.py module supports diskcache usage.

Definitely not ready to merge.

diskcache: https://pypi.org/project/diskcache/ and https://grantjenks.com/docs/diskcache/

Experimental: with a --convert flag, sconsign utility will read an existing sconsign file and set up a sqlite-backed Diskcache in .sconsign.sqlite.

=========================
TODO on the testing front:

  • test/Repository: fiddling the "writable" attribute in the framework doesn't help, sqlite db opened unwritable is always unwritable, Get: OperationalError : attempt to write a readonly database. Affects:
    test/Repository/Install-Local.py
    test/Repository/Local.py
    test/Repository/SharedLibrary.py
    test/Repository/StaticLibrary.py
    test/Repository/VariantDir.py
    test/Repository/link-object.py
    test/Repository/option-n.py
    test/Repository/variants.py
  • test/Configure/VariantDir-SConscript.py: hardcodes .dblite as the DB suffix.
  • test/Configure/implicit-cache.py: hardcodes .dblite as the DB suffix.
  • test/SConsignFile/default.py: hardcodes .dblite as the DB suffix
  • test/SConsignFile/explicit-file.py: hardcodes .dblite as the DB suffix
  • test/option/hash-format.py: why does this do an assert, anyway? e.g. Missing files: /tmp/scons/testcmd.305909.0g2s8ny7/.sconsign.dblite
  • test/sconsign/corrupt.py: hardcodes .dblite. Since it's not a flat file, corrupt is different. Expose Cache.check()?
  • test/sconsign/script/Configure.py: currently hitting unpickled data
  • test/sconsign/script/dblite.py: possibly should just not run, but maybe add parallel one for sdiskcache
  • test/option/option-n.py: see FW
  • testing/framework/TestSCons.py: get_sconsignname, unlink_sconsignfile don't work for sqlite db
  • Some tests should use unlink_sconsignfile(possibly renamed) that don't. From a grep:
  test/Configure/VariantDir-SConscript.py
  test/MSVS/vs-10.0-exec.py
  test/MSVS/vs-10.0Exp-exec.py
  test/MSVS/vs-11.0-exec.py
  test/MSVS/vs-11.0Exp-exec.py
  test/MSVS/vs-14.0-exec.py
  test/MSVS/vs-14.0Exp-exec.py
  test/MSVS/vs-14.1-exec.py
  test/MSVS/vs-14.2-exec.py
  test/MSVS/vs-14.3-exec.py
  test/MSVS/vs-6.0-exec.py
  test/MSVS/vs-7.0-exec.py
  test/MSVS/vs-7.1-exec.py
  test/MSVS/vs-8.0-exec.py
  test/MSVS/vs-8.0Exp-exec.py
  test/MSVS/vs-9.0-exec.py
  test/MSVS/vs-9.0Exp-exec.py
  test/option/option-n.py
  test/option/hash-format.py
  • test/SConsignFile` - Check - may need work, or may be unique to file-based. may need equivalent one for DB versions?

Contributor Checklist:

  • I have created a new test or updated the unit tests to cover the new/changed functionality.
  • I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
  • I have updated the appropriate documentation

@mwichmann mwichmann changed the title WIP: expeimental support for diskcache-style (sqlite backed) sconsign WIP: experimental support for diskcache-style (sqlite backed) sconsign Oct 25, 2024
This is a low-touch conversion to use an sqlite backend instead
of a flat "dblite" file for the sconsign database, without really
changing SCons logic - but includes some hacks where rather than full
detection, things are just commented out.

New SCons/sdiskcache.py module supports diskcache usage.

Definitely not ready to merge.

diskcache: https://pypi.org/project/diskcache/ and
https://grantjenks.com/docs/diskcache/

Experimental: with a --convert flag, sconsign utility will read existing
sconsign file and set up a sqlite-backed Diskcache.

=========================
TODO on the testing front:

test/Repository: fiddling the "writable" attribute in the framework
  doesn't help, sqlite db opened that way always unwritable,
  Get: OperationalError : attempt to write a readonly database
  Affects:
    test/Repository/Install-Local.py
    test/Repository/Local.py
    test/Repository/SharedLibrary.py
    test/Repository/StaticLibrary.py
    test/Repository/VariantDir.py
    test/Repository/link-object.py
    test/Repository/option-n.py
    test/Repository/variants.py

test/Configure/VariantDir-SConscript.py:
test/Configure/implicit-cache.py: hardcodes .dblite as the DB suffix.
test/SConsignFile/default.py: hardcodes .dblite as the DB suffix
test/SConsignFile/explicit-file.py: hardcodes .dblite as the DB suffix
test/option/hash-format.py: why does this do an assert, anyway?
  e.g. Missing files: `/tmp/scons/testcmd.305909.0g2s8ny7/.sconsign.dblite'

test/sconsign/corrupt.py: hardcodes .dblite
  Since it's not a flat file, corrupt is different. Expose Cache.check()?

test/sconsign/script/Configure.py: currently hitting unpickled data
test/sconsign/script/SConsignFile.py: currently hitting unpicked data
  Note some of the "hardcoded" tests would

test/sconsign/script/dblite.py: possibly should just not run,
  but maybe add parallel one for sdiskcache

test/option/option-n.py: see FW
testing/framework/TestSCons.py: get_sconsignname, unlink_sconsignfile
  don't work for sqlite db

Some tests should use unlink_sconsignfile (possibly renamed) that don't.
From a grep:
  test/Configure/VariantDir-SConscript.py
  test/MSVS/vs-10.0-exec.py
  test/MSVS/vs-10.0Exp-exec.py
  test/MSVS/vs-11.0-exec.py
  test/MSVS/vs-11.0Exp-exec.py
  test/MSVS/vs-14.0-exec.py
  test/MSVS/vs-14.0Exp-exec.py
  test/MSVS/vs-14.1-exec.py
  test/MSVS/vs-14.2-exec.py
  test/MSVS/vs-14.3-exec.py
  test/MSVS/vs-6.0-exec.py
  test/MSVS/vs-7.0-exec.py
  test/MSVS/vs-7.1-exec.py
  test/MSVS/vs-8.0-exec.py
  test/MSVS/vs-8.0Exp-exec.py
  test/MSVS/vs-9.0-exec.py
  test/MSVS/vs-9.0Exp-exec.py
  test/option/option-n.py
  test/option/hash-format.py

test/SConsignFile - Check - may need work, or may be unique to file-based.
   may need equivalent one for DB versions?

Signed-off-by: Mats Wichmann <[email protected]>
@mwichmann mwichmann force-pushed the wip-sconsign-diskcache branch from e89f5d0 to 8cfbc2e Compare October 26, 2024 14:48
@bdbaddog
Copy link
Contributor

Closing as it's still far from a solution. per discord discussion.

@bdbaddog bdbaddog closed this Nov 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants