Experimental implementation of V002 proposal

[clach04]

Experimental implementation of V002 proposal. Available in https://github.com/clach04/jenc-py/tree/issue_7_V002_experiment

The same as V001 but adopt iteration count recommendation from OWASP. I.e. 210K taken 2024-11-12 from https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2

As per comment gsantner/markor#1378 (comment)

Discussion on Markor gsantner/markor#2471

[clach04]

From initial implementation.

Before V002 test suite run time under 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] was sub 0.2 seconds:

Ran 11 tests in 0.193s

With V002 and 210K iterations instead of 10K, about 1.6 secs:

Ran 11 tests in 1.613s

[clach04]

Proposal created/opened gsantner/markor#2471

[clach04]

similar comment posted to Markor discussion:

I still do not have an android build (even though, I've written the code 😆).

What I have done is implement some testing with a desktop/command-line, python implementation. This implementation is single-threaded, it does NOT attempt to do anything in parallel.

I ran the tests at least 3 times ('cos this is supposed to be a fun project). I have an SSD and I will ignore the first run so as to not test disk-IO timing, and instead focus on the CPU processing time.

I have essentially done 2 benchmarks with 4 different encryption implementations (2 jenc).

1. is 10K files, just under 50Mb taken from https://github.com/Zettelkasten-Method/10000-markdown-files - this test encrypts EACH of the 10,000 files - I consider this a stress-test. This does NOT reflect my personal use case. It is significantly more notes/files that I have, also I do NOT encrypt all my notes only a subset
2. My real "work" notes, I do have a larger set of notes for personal use (I've just not had chance to benchmark this yet). This is 1587 *.txt files and 42 encrypted files.

The benchmark searches both the plain text and the encrypted files so one can argue that case 2 above is not solely testing crypto impact, but it matches my real use cases :-)

Stats for 2:

txt
1587 File(s)      4,710,616 bytes

tombo - chi / Tombo / Blowfish derived
42 File(s)         82,856 bytes

jenc v001
41 File(s)         85,456 bytes

jenc v002
42 File(s)         86,228 bytes

tombo_aes256_zip - aes zip
49 File(s)        336,841 bytes
NOTE there are 42 encrypted notes in .aes.zip, and 7 regular zip files without encrypted notes.
ptgrep will start to search ALL of them

Results - 1

Results 1 - Raw

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 94.72 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 49.17 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v001_jenc

Query time: 47.97 seconds


(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1055.48 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1000.81 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 987.80 seconds

(py3.12.5venv) C:\code\puren_tonbo>ptgrep -p password --time --search_encrypted does_not_exist  C:\code\notes\10000-markdown-files\10k_v002wip_jenc

Query time: 1005.16 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.42 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.44 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_aes256_zip

Query time: 10.35 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.60 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.60 seconds

(py3.12.5venv) C:\code\notes\10000-markdown-files>ptgrep -p password --time --search_encrypted does_not_exist 10k_tombo_chi

Query time: 10.41 seconds

Results 1 - Summary

1. 10.42 seconds - 10k_aes256_zip
2. 10.60 seconds - Tombo
3. 49.17 seconds - under a minute - jenc v001, as used by Markor
4. 1055.48 seconds - ~17 minutes - jenc v002 proposal/experiment

Results - 2

Results 2 - Raw

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.14 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.13 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo

Query time: 0.13 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 5.54 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 0.16 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_aes256_zip

Query time: 0.16 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.32 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.31 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v001_jenc

Query time: 0.31 seconds

(py3.12.5venv) C:\tmp\work_notes>echo.


(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.20 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.16 seconds

(py3.12.5venv) C:\tmp\work_notes>ptgrep --time --search_encrypted really_does_not_exist tombo_v002_jenc

Query time: 4.18 seconds

Results 1 - Summary

1. 0.13 seconds - Sub-second - My current, very old Blowfish based from the Tombo project
2. 0.16 seconds - Sub-second - aes256_zip, AE-1 only Zip file with AES-256 - Standard WinZip/7z (not the old ZipCrypto!)
3. 0.32 seconds - Sub-second - jenc v001, as used by Markor
4. 4.20 seconds - over 4 seconds - jenc v002 proposal/experiment

Background Notes

I used a new feature in puren_tonbo to generate the test data, there is a new tool for bulk converting files:

For 10K notes:

git clone https://github.com/Zettelkasten-Method/10000-markdown-files
cd 10000-markdown-files
python -m puren_tonbo.tools.ptrecrypt --cipher .v001_jenc  --new_extension .jenc -p password  --destination_directory 10k_v001_jenc "10000 markdown files"
python -m puren_tonbo.tools.ptrecrypt --cipher .v002_jenc  --new_extension .jenc -p password  --destination_directory 10k_v002wip_jenc "10000 markdown files"
python -m puren_tonbo.tools.ptrecrypt --cipher .aes256.zip --new_extension .aes256.zip  -p password  --destination_directory 10k_aes256_zip "10000 markdown files"

For my notes:

Copy tombo folder a few times and name using dirs below, re-encrypt in-place:

REM prompt for password, these are my real work notes. Do not leak secret ;-)
ptrecrypt --cipher .v001_jenc  --new_extension .jenc --skip_unencrypted --existing-files=delete    tombo_v001_jenc
ptrecrypt --cipher .v002_jenc  --new_extension .jenc --skip_unencrypted --existing-files=delete    tombo_v002_jenc
ptrecrypt --cipher .aes256.zip                       --skip_unencrypted --existing-files=delete    tombo_aes256_zip
ptrecrypt --cipher .zip                              --skip_unencrypted --existing-files=delete    tombo_zip