feat(importer): add rabin fingerprinting chunk algorithm #223
Conversation
This is required to have feature parity with go-ipfs which supports rabin chunking algorithm.
Rabin chunker supports the following `chunkerOptions`
minChunkSize: {integer}
avgChunkSize: {integer}
maxChunkSize: {integer}
polynomial: {string}
window: {integer}
Rabin chunker uses the same defaults defined by go-ipfs-chunker.
Supports ipfs/js-ipfs#1283
License: MIT
Signed-off-by: Dan Ordille <[email protected]>
|
@dordille I finally had a chance to review the code. It looks good so far. Though the code coverage bot reports a decrease. It took me a while to figure out why, because I saw that you added tests. The problem is that your tests are not actually run, so please make them run and have another look at the coverage. For further testing, the Go implementation could be an inspiration. |
License: MIT Signed-off-by: Dan Ordille <[email protected]>
|
This looks good. I think you need to add one more test to cover the situation when you don't specify a |
License: MIT Signed-off-by: Dan Ordille <[email protected]>
|
Is there a way to force a jenkins build without pushing a commit, I don't think the current failure is a result of my changes. |
|
@dordille I think only people with merge permissions are able to retrigger a build. Sadly there's currently env related issues (workers running out of disk space). I'll make sure to have a clean run. If it should turn out that the failure is due to your code, I'll let you know. Sorry for the inconvenience. |
|
Jenkins, ugh. See ipfs-inactive/dev-team-enablement#70 for the reasons behind the disk space thing. Anyway this has been merged & published as |
This is required to have feature parity with go-ipfs which supports rabin chunking algorithm.
Rabin chunker supports the following
chunkerOptions, defaults mirror those in go-ipfs-chunker.minChunkSizeavgChunkSize/ 3avgChunkSizemaxChunkSizeavgChunkSize+ (avgChunkSize/ 2)windowpolynomialThis currently works as is, however the parameters polynomial and windowsize are not configurable by the rabin library used. This makes it difficult to compare the output the go-ipfs implementation, since both these parameters are set to different values there.
Could use some advice on testing, the nature of the rabin chunking makes it difficult to test things like expected block sizes, and number of blocks.
These changes are required for ipfs/js-ipfs/issues/1283