Implicit vs explicit groups

[jbms]

Currently the zarr v3 spec does not require that every group have a metadata document, and instead allows a group to be defined implicitly by the presence of a descendant.

Pros:

• No need to write an empty metadata document just to have a group.
• Multiple machines can create distinct arrays in parallel within the same, possibly not-yet-existing group without any coordination or store-level locking/atomic operations to create the group metadata document.

Cons:

• Possibility of simultaneously creating a group and array with the same name. This could be fixed by allowing arrays to also be groups.
• If group-level storage transformers are allowed, there is a possible race condition: one machine may attempt to create "a/b" as an array, and another machine may simultaneously create "a" as a group with a storage transformer. We may end up with a corrupted result, where "a/b" does not correctly take into account the storage transformer.

An alternative to implicit groups is to use a separator other than "/", such as ".", if the equivalent of implicit groups are desired.

[jbms]

Also, even if racy due to a lack of support for atomic read/modify/write, writing the empty group files from multiple machines concurrently could still be fine; since the contents would be the same from all writers, it doesn't matter which writer wins.

[jstriebel]

• Possibility of simultaneously creating a group and array with the same name. This could be fixed by allowing arrays to also be groups.

Since the names of the metadata documents will be both zarr.json for arrays and groups as of #200 this case seems to be eliminated.

• If group-level storage transformers are allowed, there is a possible race condition: one machine may attempt to create "a/b" as an array, and another machine may simultaneously create "a" as a group with a storage transformer. We may end up with a corrupted result, where "a/b" does not correctly take into account the storage transformer.

True, but I don't see how this can be circumvented. Even if implicit groups use a different delimiter than /, both could be created simultaneously, and one of both would win, the other would probably not exist. I don't see a way around this race condition.

@jbms Do you think there is something to do as part of the v3 core spec? Can we close this?

[jbms]

Yes we can close this.