ZFP compression of distance matrix

[johnlees]

https://github.com/LLNL/zfp

Docs:
https://zfp.readthedocs.io/en/release0.5.5/tutorial.html#compressed-c-arrays
https://zfp.readthedocs.io/en/release0.5.5/python.html

The distance matrix is a 2xNxN tensor this might be effective for

[johnlees]

Looks like there's a nice solution to memory mapping in eigen here: https://stackoverflow.com/a/51256597

[johnlees]

Note that numpy.savez_compressed takes a 4.9Gb dist matrix to 4.2Gb (and takes a couple of minutes)

Using zfpy (note that max dimension is 2^32, so long form doesn't work):

>>> core_sq = pp_sketchlib.longToSquare(X[:, [0]], 10)
>>> acc_sq = pp_sketchlib.longToSquare(X[:, [1]], 10)
>>> stacked = np.dstack((core_sq, acc_sq))
>>> stacked.shape
(35905, 35905, 2)
>>> stack_compressed = zfpy.compress_numpy(stacked)
>>> sys.getsizeof(stack_compressed)
18638423905
>>> sys.getsizeof(stacked)
10313352328
>>> stack_compressed = zfpy.compress_numpy(stacked, tolerance=1e-4)
>>> sys.getsizeof(stack_compressed)
9282196129
>>> stack_compressed = zfpy.compress_numpy(stacked, tolerance=1e-3)
>>> sys.getsizeof(stack_compressed)
6784347833
>>> stack_compressed = zfpy.compress_numpy(stacked, precision=1e-3)
>>> sys.getsizeof(stack_compressed)
16507950401

Zeroing the lower triangle:

>>> lower_tri = np.tril_indices(stacked.shape[0])
>>> stacked[lower_tri, 0] = 0
>>> stacked[lower_tri, 1] = 0
>>> stack_compressed = zfpy.compress_numpy(stacked, tolerance=1e-3)
>>> sys.getsizeof(stack_compressed)
6784395913

Taking a slice which will fit from the long form:

>>> X_compressed = zfpy.compress_numpy(X[0:(2**24),:])
>>> sys.getsizeof(X_compressed)
248149257
>>> X_compressed = zfpy.compress_numpy(X[0:(2**24),:], tolerance=1e-3)
>>> sys.getsizeof(X_compressed)
88800441
>>> X_compressed = zfpy.compress_numpy(X[0:(2**24),:], tolerance=1e-4)
122346137

(very similar results to the above)

Not immediately obvious that this is easily useful. tol = 1e-3 gives ~1.5x compression, but when tol = 1e-4 (=~1/sketch size) compression is only about 10%. Storing the square form doubles the space required, and compressing that is more difficult.
It's possible that reordering the rows so that strains are together might help, but the C/Fortran ordering would make this more difficult in python.