feat(porep): skip existing layers if they already exist

[dignifiedquire]

This allows resuming sealing operations, as only layers are generated that don't yet exist on disk.

Closes #884

[porcuquine]

'succesful'

[dignifiedquire]

fixed

[porcuquine]

Can you come up with a more descriptive name and/or add a comment to explain exactly what this does? We're getting into a soup of self-referential names which is hard to get a handle on by just looking at one part. It would be good to reverse that trend where we can.

[dignifiedquire]

added some docs and improved the name a little, honestly can't come up with something better

[porcuquine]

@dignifiedquire Can we also get a test — maybe an addition to the seal_lifecycle tests, which exercises this? That will help us release with confidence.

[dignifiedquire]

@dignifiedquire Can we also get a test — maybe an addition to the seal_lifecycle tests, which exercises this? That will help us release with confidence.

Added a check in replicate + extract that checks that it detects the already existing layers

[porcuquine]

This is looking good. There is one detail, I'm not sure is accounted for. I am not 100% sure of the DiskStore's behavior, though, so I will describe both options I see. In both cases, the issue is that a layer may have been only incompletely written to disk.

1. DiskStore writes its contents incrementally, and the file grows until the whole layer has been written. In that case, we know that a file of the correct size represents a completed layer. We should check for the size.

2. DiskStore first 'truncates' the file to the full size in order to ensure there is enough room on disk. (This is probably what it should do. If it doesn't, we should consider changing.) In this case, we need some other way of explicitly recording that the on disk representation of the completed layer is also complete.

Either way, it seems possible the file has been partially written then interrupted. In that case, even though the layer will have been completely created, we still need to discard the layer and begin from the last layer completely stored on disk.

Although this may be heavy, one thorough solution would be to build up an incremental digest of the layer throughout the sealing process. This is not wasteful, since replication will leave most cores idle. Then the digest can be written to a status file and checked when restarting. However, this may be overkill, since the primary issue we are trying to address is not disk corruption. In that case, the mere presence of the 'completion status' file would suffice to signal completion.

As above, it might also be that we can simply use file length as a check, but also as above, we should probably aim to fully allocate space for the layer files before writing them. This will allow for better performance (less fragmentation) and also better error handling when disk is full. Incidentally, if we don't do this, then a full disk will definitely create the situation of an only-partially-written DiskStore — a situation from which we can never recover.

Regarding recovery, since we cannot prevent the possibility of a partially-written file from existing, resumption should also ensure cleanup of the partial file. I think that will happen automatically, in the current design, though.

[porcuquine]

Added a check in replicate + extract that checks that it detects the already existing layers

@dignifiedquire Did you mean to push something that didn't make it? I don't see this.

[dignifiedquire]

@porcuquine this is the place where I check that things are not generated twice

[dignifiedquire]

@porcuquine Instead of using the DiskStore, which was a hack anyway, it is now using regular file writes, and doing them atomically, by writing to a tmp file, and renaming afterwards. This should eleviate the issues you described

[porcuquine]

Looks good. Would be great if someone can manually verify this too, but tests look reasonable.

[dignifiedquire]

closing in favor of #1292