Optimize HashTask using parallelism

[aberenguel]

I was processing a case (using 4.1.4 version) with a big file inside the case (~68G) and I noticed that HashTask was taking a lot of time.

Analyzing the code, I noticed that hashes could be parallelized.

I implemented it and did some experiments.

File Size	Time (s) - current code	Time (s) - parallelized ¹	Time 2 (s) - parallelized ²
10M	0.154 / 0.174	0.146 / 0.15	0.143 / 0.158
500M	6.66 / 6.31	4.08 / 3.67	2.66 / 2.85 / 2.84
10G	116.54 / 117.69 / 1117.70	68.63 / 73.09 / 74.67	55.33 / 52.97 / 54.83

¹ Implemented reading buffer and hashing in sequence.
² Reads buffer while another threads are hashing.

One counterpoint is that with this optimization the HashTask is using more thread the it is supposed to use (just one).

I'll submit the code and reference in this issue.

[lfcnassif]

Very nice, thank you!!!

But, have you tested it with real cases with files with different sizes to see if times of other tasks remain equal (or increase) or if overall time decreases? Maybe more threads could compete for CPU with other tasks...

PS1: On #1834 I tested an approach that I think is similar but more general, and unfortunately overall results weren't good...

PS2: Maybe a similar approach could be done to EntropyTask, I have seen it spending a lot of time with big files, but that is another improvement...

[aberenguel]

But, have you tested it with real cases with files with different sizes to see if times of other tasks remain equal (or increase) or if overall time decreases? Maybe more threads could compete for CPU with other tasks...

I've just tested using a folder with a single file as evidence. I'll test in some real images I have here.

[aberenguel]

I noticed that my personal computer CPU (AMD Ryzen 7 5700G) has instructions for SHA algorithms (https://en.wikipedia.org/wiki/Intel_SHA_extensions).

I know that openssl uses such instructions if supported. So I hashed a file with 10G. The performance is huge:

$ pv file-10G.bin | openssl dgst -sha256
9,31GiB 0:00:06 [1,38GiB/s] [=====================================================>] 100%            
SHA2-256(stdin)= 792cd8bfd20e81a5c01648d1e45fcc301b255ff94a4865d9183440944fde0364

Compared with classical sha256sum command:

pv file-10G.bin | sha256sum 
9,31GiB 0:00:27 [ 342MiB/s] [=====================================================>] 100%            
792cd8bfd20e81a5c01648d1e45fcc301b255ff94a4865d9183440944fde0364  -

[aberenguel]

I looked for some JCE Provider that supported such instructions and found that one:
https://github.com/corretto/amazon-corretto-crypto-provider

So I measured the HashTask processing same file.

Using default Provider:

[engine.task.HashTask]                  Hash: md5 => 25258ED0BD157CC9D06D9675C9B4B3C3
[engine.task.HashTask]                  Hash: sha-1 => 0229C4D391927DC3B27DA0809F79FFE0EB6DAA2F
[engine.task.HashTask]                  Hash: sha-256 => 792CD8BFD20E81A5C01648D1E45FCC301B255FF94A4865D9183440944FDE0364
[engine.task.HashTask]                  Hash time: 20.032

Using AmazonCorrettoCryptoProvider:

[engine.task.HashTask]                  Hash: md5 => 25258ED0BD157CC9D06D9675C9B4B3C3
[engine.task.HashTask]                  Hash: sha-1 => 0229C4D391927DC3B27DA0809F79FFE0EB6DAA2F
[engine.task.HashTask]                  Hash: sha-256 => 792CD8BFD20E81A5C01648D1E45FCC301B255FF94A4865D9183440944FDE0364
[engine.task.HashTask]                  Hash time: 13.739

[aberenguel]

I think the use AmazonCorrettoCryptoProvider can be extremely worth and deserves another issue to not influence in performance metrics of this issue.

[lfcnassif]

I looked for some JCE Provider that supported such instructions and found that one: https://github.com/corretto/amazon-corretto-crypto-provider

So I measured the HashTask processing same file.

Using default Provider:

[engine.task.HashTask]                  Hash: md5 => 25258ED0BD157CC9D06D9675C9B4B3C3
[engine.task.HashTask]                  Hash: sha-1 => 0229C4D391927DC3B27DA0809F79FFE0EB6DAA2F
[engine.task.HashTask]                  Hash: sha-256 => 792CD8BFD20E81A5C01648D1E45FCC301B255FF94A4865D9183440944FDE0364
[engine.task.HashTask]                  Hash time: 20.032

Using AmazonCorrettoCryptoProvider:

[engine.task.HashTask]                  Hash: md5 => 25258ED0BD157CC9D06D9675C9B4B3C3
[engine.task.HashTask]                  Hash: sha-1 => 0229C4D391927DC3B27DA0809F79FFE0EB6DAA2F
[engine.task.HashTask]                  Hash: sha-256 => 792CD8BFD20E81A5C01648D1E45FCC301B255FF94A4865D9183440944FDE0364
[engine.task.HashTask]

Excellent!

But seems AmazonCorretoCryptoProvider unfortunately supports just Linux and MacOS... We try to use just native dependencies that support Windows and Linux at least. Maybe Linux users could add it into plugins folder if they want?

[aberenguel]

Excellent!

But seems AmazonCorretoCryptoProvider unfortunately supports just Linux and MacOS... We try to use just native dependencies that support Windows and Linux at least. Maybe Linux users could add it into plugins folder if they want?

There is an issue for that: corretto/amazon-corretto-crypto-provider#48