Whiteouts generated when building a layer on top of an existing one

[ariel-miculas]

https://github.com/anuvu/puzzlefs/blob/master/builder/src/lib.rs#L220

If we're building the new layer as a self-contained filesystem, then whiteouts shouldn't be generated.

[hallyn]

This sounds correct to me. It seems like we should simply let the file be removed from (, not exist in) the generated filesystem.

@tych0 if you have time to comment on this, that would be great. (If you don't, I'll understand). I think this was OCIv1 thinking sneaking into the puzzlefs implementation.

I'm going to open another issue to discuss a closely related question.

[ariel-miculas]

So we're not planning to use overlays with multiple puzzlefs layers?

[hallyn]

I don't believe so.

[tych0]

I think there are two issues here, and it depends on what you care about:

1. if you care about build time, you don't want to re-compute the whole underlying filesystem, so it'll be faster to just add a whiteout
2. if you care about "canonicalization" (maybe data sharing), you want to recompute the underlying filesystem and shouldn't add whiteouts

This kind of thing is what the "do computation when we want to" under https://github.com/anuvu/puzzlefs#design-goals is talking about; maybe it's too complicated to have both ways, though.

[hallyn]

So the whiteout is there just to keep from having to rewrite the whole metadata stream?

If the preexisting image has a 100G file /root/data.gz, and the delta removes that, then the 100G of data chunks will not be a part of the new image, is that right? But the inode entry will still be there, followed by a whiteout entry for the same path?

Sorry, I really need to draw some diagrams.

[tych0]

Yep, exactly. If you care about buildtime, you don't want to recompute everything and a whiteout will be fast. But perhaps the canonicalization step will be useful in a jenkins style build where developers are not iterating. It wasn't really clear which would be more useful, or if we'd need both configurations.