unixfs v1.5 metadata support #223

Stebalien · 2019-10-23T04:44:29Z

This deviates slightly from what was discussed but should be enough to get us started. We can tweak as necessary as we go along.

Stebalien · 2019-10-23T04:46:40Z

UNIXFS.md

+
+1. The repeated `metadata` field in a directory applies metadata to each file in the directory.
+2. An intermediary node with a `Type` of `Metadata` applies metadata to an individual file. While this feature should be supported, it has been deprecated.
+3. An optional `defaultMetadata` field to specify the _default_ metadata for files in the directory. If unspecified, the default mode is 0755 and the default modification time is the epoch.


This can save us a ton of space as most files in a directory have the same mode. However, it is slightly complicated.

@mib-kd743naq suggested keeping a reverse mapping of, e.g., mode -> [file offsets] but, while very compact, that felt too complicated.

On the other hand, we could also just rely on future compression. Repeated identical metadata fields will compress really well.

but, while very compact, that felt too complicated.

This is mostly because of huge/sharded directories right? Otherwise, seems reasonably straightforward.

we could also just rely on future compression

Seems like a bad idea because future should hopefully come after unixfsv2. Also, sometimes compression + encryption don't play nicely which might cause problems if we rely too heavily on it.

I'll channel @mib-kd743naq's alter-ego here ;P The idea of defaultMetadata looks deceptively simple, but would be hard to make "convergent". The typical mode of producing UFS1.5 datasets would be tar-file ( or equivalent ) importers. A tar importer works best ( and most faithfully ) in linear streaming fashion, which in turn means that the "best" defaultMetadata is difficult to know ahead of time. This is especially important for mega-directories like https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts and is even harder to determine in cases of sharded directories.

The flip side of doing "deduplication of metadata" as we go is to simply keep a map during encoding of "what metadata have we seen so far" keyed off the encoded protobuf content itself, together with a plain integer increment counter.

With all that said - it appears to NOT matter. Using the dataset of the current kernel, with each individual entry decorated with the proper metadata based on a spec similar[*] to the proposed the numbers are way closer than I anticipated. Also, yes, transparent block compression will help a lot here, but there still will be a lot of indirection for a compressor to deal with, and there is nothing simpler than simply having nothing extra included ;) Additionally transparent compression is a separate :canofworms: that may take longer than UFS2.0 to shake out :(

So the data based on https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.3.7.tar.sign

To see individual directory makeups simply run:

for h in \ bafybeif7ceybwlonoct3bnjtnnfookm2iezslo2ypwrsae6ilkc2ucvsfm \ bafybeigoe7cnxqy6hjlcci3ae62l5fmws5pxeq62d77lhbvgphhm5ugtnq \ bafybeico6ybaiaqohfsydgqxexwcjb4vnlhzojyxhjcw6s3xhm7weaikb4 \ bafybeibj2hpul5vgomel56rl6xf63rviyym4go5pg2fmwwqioaltshh6fa do tgt=$h/linux-5.3.7/Documentation/PCI/endpoint echo -e "\n\nhttps://ipfs.io/ipfs/$tgt\n==============\n" ipfs block get $( ipfs resolve $tgt | cut -b 7- ) | protoc --decode_raw done | less

And the stats:

dumb ( each directory entry gets an exactly one "struct 7" ): ipid : bafybeif7ceybwlonoct3bnjtnnfookm2iezslo2ypwrsae6ilkc2ucvsfm count_data_blocks : 230,613 count_link_blocks : 42,341 count_all_blocks : 272,954 size_data_blocks : 843,884,873 size_link_blocks : 13,694,638 size_all_blocks : 857,579,511 size_payload : ^ dedup ^ stebalien-first ( Stebalien's proposal, if more than 2 entries are present in a directory, the first one becomes the "default struct 8" ): ipid : bafybeigoe7cnxqy6hjlcci3ae62l5fmws5pxeq62d77lhbvgphhm5ugtnq count_data_blocks : 230,613 count_link_blocks : 42,341 count_all_blocks : 272,954 size_data_blocks : 843,884,873 size_link_blocks : 13,036,969 size_all_blocks : 856,921,842 size_payload : ^ dedup ^ stebalien-most ( Steven's proposal, if more than 2 entries are present in a directory, an analysis takes place of which is the most-encountered, and that one becomes default ): ipid : bafybeico6ybaiaqohfsydgqxexwcjb4vnlhzojyxhjcw6s3xhm7weaikb4 count_data_blocks : 230,613 count_link_blocks : 42,341 count_all_blocks : 272,954 size_data_blocks : 843,884,873 size_link_blocks : 13,013,814 size_all_blocks : 856,898,687 size_payload : ^ dedup ^ olcbean ( the proposal to just have a running reverse index per metadata blob ): ipid : bafybeibj2hpul5vgomel56rl6xf63rviyym4go5pg2fmwwqioaltshh6fa count_data_blocks : 230,613 count_link_blocks : 42,341 count_all_blocks : 272,954 size_data_blocks : 843,884,873 size_link_blocks : 12,941,959 size_all_blocks : 856,826,832 size_payload : ^ dedup ^

Once I am back from a trip we really need to chat some time next week why I keep pushing for a slightly different organization of the metadata.

Note: this is unixfsv1.5. We don't have to be perfect, just good enough.

Given this data, I'm starting to think that we might just want to drop the default metadata field and default to 0644 for files and 0755 for directories. Those numbers show a 0.09% overhead which I can easily live with.

in linear streaming fashion

Note: the reverse index is just as tricky to compute.

This is mostly because of huge/sharded directories right? Otherwise, seems reasonably straightforward.

I'm not concerned about sharded directories. We'd just store the reverse map in every terminal shard.

My main concern is that I'd like to just stick this metadata inside the directory entries in unixfsv2, right next to the links to the files.

This is "cleaner" (subjectively).

It makes it easy to move a directory entry from one directory to another (just copy the node).

When decoding the directory, we'd need to reverse the reverse index (turn it back into something where we can easily lookup metadata by file name and/or offset). At the moment, in all of the proposals for UnixFSv2 that I've seen, we'll be able to define the mapping between go structs and encoded IPLD objects using a nice IPLD schema. If we use this reverse map, we're going to either:

Need to introduce a second step and a second intermediate object.

Turn a "reverse index" into an advanced layout.

One data point from Peergos, though I'm not sure how relevant it is to unixfsv2 given the lack of encryption: we decided to put file metadata (name, modification time, mime type etc.) on the file itself, rather than the directory. This is because of using capability based access control - we want to be able to easily grant someone access to modify a file without them being able to read or modify or even identify sibling files in the same directory. So our directories are just encrypted links to files (and the encrypted name of the directory) which might be under a different IPNS signing key and an encrypted link to any subsequent block in the same directory (large directories).

That's good to know.

In that case, we'd probably have a special "encrypted" file type. Normal files will have a "data" field referring to the actual file data but this file type would have a "file" field pointing to the actual encrypted file.

However, we'd still likely want some unencrypted metadata ("file type", capabilities?, etc.).

We take it even further. Unless you have a read cap to a directory we don't want you to even be able to know how many files are in the directory (we pad and bulk encrypt all the child links together), and we even go so far as to make a directory indistinguishable from a small file at the ipld level (without a read cap). That way you can grant read access to a file, but even with that capability the only thing you can figure out about the parent directory is the name (to get a well defined path).

Ah, interesting. so you encrypt the directory entries themselves. At that point, I think we'll need to change the structures entirely.

unixfsv3 then :-)

Stebalien · 2019-10-23T04:47:38Z

UNIXFS.md

+There are three ways to specify file metadata:
+
+1. The repeated `metadata` field in a directory applies metadata to each file in the directory.
+2. An intermediary node with a `Type` of `Metadata` applies metadata to an individual file. While this feature should be supported, it has been deprecated.


I left this in here because we technically support it. However, I'd be happy to remove it if it's not in-use. go-ipfs has some internal support but it's hard to actually use this feature so I doubt any one is doing so (and it doesn't work properly in many cases).

...and completely unspecified, does Data contain the metadata or would it now be the metadata field?

(Not expecting that question to be answered - just stating the state of affairs 😉)

Data contains the metadata object. Note: this entire thing was completely unspecified until this spec was written...

Stebalien · 2019-10-23T04:52:19Z

UNIXFS.md

-	optional string MimeType = 1;
+	optional string mimeType = 1;
+	optional uint32 mode = 2;
+	optional int64 mtime = 3;


64 bits because:

We're using varints anyways.

I'd rather not deal with this again in 2038 even if we've moved on to a new format.

(also, it's signed because the world did not start in 1970)

I'd rather not deal with this again in 2038 even if we've moved on to a new format.

❤️ this is some dedication to the project 😂

Agree that the world didn't start in 1970 :) In that case should the default value be something other than 01/01/1970?

Stebalien · 2019-10-23T04:56:52Z

UNIXFS.md

@@ -46,10 +46,15 @@ message Data {

 	optional uint64 hashType = 5;
 	optional uint64 fanout = 6;
+
+	repeated Metadata metadata = 7;


I chose repeated metadata structs like this as it allows us to use one byte to represent every missing piece of metadata. We could alternatively use:

A map of integers to metadata.

A bitmap to indicate which files have metadata.

However, both feel like overkill. I'm guessing directories will either have metadata or won't.

Ideally, we'd be able to just embed this metadata in the links section of the parent object but that is not to be...

Just to be clarify, does the order the metadata appears here maps directly to the ordering of the Links in the dag-pb node? i.e. metadata[3] applies to Links[3].

...and this only applies when Type is Directory right? Can this be documented please?

Also, if we have one or more files in a directory with metadata then every file has to have metadata right so the order doesn't get mixed up?

Separately, did we consider allowing a single metadata property on the root node for each file so that we can still know metadata when the file is accessed directly? If the reason is in order to not change a file's hash because of meta then maybe mention the reason behind this design decision (perhaps this is not something you typically put in a spec, but, where else is appropriate)?

How does this work for HAMT shards? Links in a sharded dag-pb node don't map directly to files and secondly listing perms for every file in a sharded directory would make a directory node unfeasibily large.

It's very likely metadata will be used for HAMT shards so any solution we come up with here should work with them. Can we please document the solution?

Also, if we have one or more files in a directory with metadata then every file has to have metadata right so the order doesn't get mixed up?

Yes. I've expanded the spec a bit.

Separately, did we consider allowing a single metadata property on the root node for each file so that we can still know metadata when the file is accessed directly? If the reason is in order to not change a file's hash because of meta then maybe mention the reason behind this design decision (perhaps this is not something you typically put in a spec, but, where else is appropriate)?

It was considered and dropped for the reason you gave. I'll document it.

How does this work for HAMT shards? Links in a sharded dag-pb node don't map directly to files and secondly listing perms for every file in a sharded directory would make a directory node unfeasibily large.
It's very likely metadata will be used for HAMT shards so any solution we come up with here should work with them. Can we please document the solution?

Should be fixed.

please no int maps

@rvagg could you expand on that? ( I am clearly missing prior discussion of a downside... )

This probably isn't the place to litigate this, it always ends in too much text being spilled, but int maps are something we're trying to avoid for IPLD. They're just not straightforward across codecs and programming languages. https://github.com/ipld/specs/blob/master/data-model-layer/data-model.md#motivation

alanshaw

In general this sounds like it's going in the right direction. My only real concern is HAMT shards - see comment in review for more info.

alanshaw · 2019-10-23T11:00:26Z

UNIXFS.md

@@ -46,10 +46,15 @@ message Data {

 	optional uint64 hashType = 5;
 	optional uint64 fanout = 6;
+
+	repeated Metadata metadata = 7;


Just to be clarify, does the order the metadata appears here maps directly to the ordering of the Links in the dag-pb node? i.e. metadata[3] applies to Links[3].

...and this only applies when Type is Directory right? Can this be documented please?

Also, if we have one or more files in a directory with metadata then every file has to have metadata right so the order doesn't get mixed up?

Separately, did we consider allowing a single metadata property on the root node for each file so that we can still know metadata when the file is accessed directly? If the reason is in order to not change a file's hash because of meta then maybe mention the reason behind this design decision (perhaps this is not something you typically put in a spec, but, where else is appropriate)?

How does this work for HAMT shards? Links in a sharded dag-pb node don't map directly to files and secondly listing perms for every file in a sharded directory would make a directory node unfeasibily large.

It's very likely metadata will be used for HAMT shards so any solution we come up with here should work with them. Can we please document the solution?

alanshaw · 2019-10-23T11:03:17Z

UNIXFS.md

+1. The repeated `metadata` field in a directory applies metadata to each file in the directory.
+2. An intermediary node with a `Type` of `Metadata` applies metadata to an individual file. While this feature should be supported, it has been deprecated.
+3. An optional `defaultMetadata` field to specify the _default_ metadata for files in the directory. If unspecified, the default mode is 0755 and the default modification time is the epoch.
+  * Files: The default metadata is applied as-is.


What is the default mimeType string? ""? No default (field is optional anyway)?

"unspecified"/null? Good question.

Note: We've also discussed just dropping support for mime types this way. The original idea was to do this so that the gateway could use the metadata but it's quite a bit simpler to just use a .ipfsattr file.

I think it'll useful to know in the directory listing which links are directories, which are symlinks and which are regular files. For example WebUI currently has to stat every file in a directory listing to determine this and display the correct icon.

Yeah, please consider storing the entire UNIX 32bit st_mode structure. Or alternatively just go with https://golang.org/pkg/os/#FileMode

I agree. Including the actual file type is definitely a good idea.

For those following along - this is what including the type ( i.e. full st_mode/mode_t ) allows. This is the same CID as the one used earlier in #223 (comment). Observe how a user now has clear visual cues what is a directory and thus explorable. All can be displayed, backwards-compatibly, without downloading any extra blocks.

Note: temporary hacked together gateway, will not persist: http://5.9.79.235:2020/ipfs/bafybeif7ceybwlonoct3bnjtnnfookm2iezslo2ypwrsae6ilkc2ucvsfm/linux-5.3.7/

alanshaw · 2019-10-23T11:07:42Z

UNIXFS.md

+There are three ways to specify file metadata:
+
+1. The repeated `metadata` field in a directory applies metadata to each file in the directory.
+2. An intermediary node with a `Type` of `Metadata` applies metadata to an individual file. While this feature should be supported, it has been deprecated.


...and completely unspecified, does Data contain the metadata or would it now be the metadata field?

(Not expecting that question to be answered - just stating the state of affairs 😉)

UNIXFS.md

alanshaw · 2019-10-23T11:28:14Z

UNIXFS.md

+
+1. The repeated `metadata` field in a directory applies metadata to each file in the directory.
+2. An intermediary node with a `Type` of `Metadata` applies metadata to an individual file. While this feature should be supported, it has been deprecated.
+3. An optional `defaultMetadata` field to specify the _default_ metadata for files in the directory. If unspecified, the default mode is 0755 and the default modification time is the epoch.


Suggestion: Instead of defaultMetadata, what about by convention if you specify only 1 metadata property it applies to all directory contents?

alanshaw · 2019-10-23T11:32:56Z

UNIXFS.md

+
+Fields:
+
+* `mimeType` -- The mime-type of the file. This generally shouldn't be used.


This generally shouldn't be used.

Please can we document why? We wouldn't have to do sniffing in many cases if it's explicit.

alanshaw · 2019-10-23T11:35:25Z

UNIXFS.md

-	optional string MimeType = 1;
+	optional string mimeType = 1;
+	optional uint32 mode = 2;
+	optional int64 mtime = 3;


I'd rather not deal with this again in 2038 even if we've moved on to a new format.

❤️ this is some dedication to the project 😂

alanshaw

2 minor clarifications you might want to add but assuming @achingbrain is ok with the implementation for HAMT this SGTM 🚢🚢🚢

UNIXFS.md

alanshaw · 2019-10-24T08:44:55Z

UNIXFS.md

+For some context, this solution was chosen over embedding the metadata in the file itself because:
+
+1. It allows accessing the metadata without downloading the target files.
+2. More importantly, it avoids changing the root hash of the target files to allow for better deduplication.


Should we mention the tradeoff? If I have QmDir and file.txt (QmFileTxt) then I'll only be able to see metadata for the file if it's accessed via /ipfs/QmDir/file.txt. The file will have no metadata when accessed like /ipfs/QmFileTxt.

@alanshaw from #220 (comment) :

As far as I can tell, for the original set of users waiting on this, there is no usecase where a piece of data needs POSIX attributes without a filename. The filename is already only available as a directory wrapper node, there is no other way to attach name to an "anonymous stream of data". What makes other attributes different enough to warrant a non-directory anonymous wrapper-node?

I.e. what is the use-case of needing to know mtime/nodetype/permissions of an object without knowing its name?

I don't think metadata is tied to file names - you could be copying /ipfs/QmHashOfSomeBinaryFile from ipfs-at-large to a location in your mfs, at which point it would receive a filename, but if it was executable, you'd want to retain that information.

In this design, the metadata is stored in the parent directory. That doesn't mean that it's tied to the file name.

It does mean that simplifying a path from /ipfs/QmDir/file.txt to /ipfs/QmFile will drop the metadata (along with the filename).

This doesn't seem like the right way to do it to me. If I copy a file to another location on my filesystem, it doesn't lose its permissions, nor will it lose its permissions if I remove it from its containing directory (to put it in a tarball, for example).

It will involve changing the CID of a file if it's metadata changes but that seems like a reasonable tradeoff to me, since the CIDs of the underlying file chunks will not change (assuming the file size is > 256KB). We'll probably see the most CID churn in a users' mfs, but we generally don't care about CIDs in mfs as much as paths.

A counter thought: why should metadata that changes independently from the data (e.g. permissions) be anywhere in the directory tree, couldn't it live outside the tree?

If I copy a file to another location on my filesystem, it doesn't lose its permissions

True, but when I email you a file or upload it to Dropbox it does lose its permissions. Similarly, using unix time as a metadata attribute makes very little sense in a distributed context. For example, Alice and Bob both download an Ubuntu ISO and add it to IPFS the timestamps will be different.

It will involve changing the CID of a file if it's metadata changes but that seems like a reasonable tradeoff to me, since the CIDs of the underlying file chunks will not change

If I recall correctly, the way IPFS currently works makes this a pretty big performance hit. For example, say Alice and Bob both have the same files, but with different metadata. Then if Charlie tries to download Alice's CID while Alice is offline then he gets no data. Similarly, even if Alice is online Charlie will end up downloading the data from Alice but not Bob since his node probably won't make another DHT request for the file CID (which would be expensive anyway).

There are alternative solutions for people that want to share metadata that do not involve putting OS metadata near files for everyone else, which I think are worth discussing. However, if unixfsv1.5 is just a temporary measure to tide us over until unixfsv2 lands then as long as we only use the metadata we actually need (i.e. mtime + execute bit) then this is fine by me.

when I email you a file or upload it to Dropbox it does lose its permissions.

Those aren't filesystem operations so I would not expect them to retain permissions. Ok, tarballing something up isn't specifically a filesystem operation either, but it is sensitive to it's environment so does preserve perms, I think we should endeavour to do the same thing.

Similarly, using unix time as a metadata attribute makes very little sense in a distributed context.

I agree, distributed time is a bit weird. That said, the file chunks of the ISO will be the same, just the root node containing the metadata will be different. That said Alice and Bob could use different chunking strategies, or whatever so it's already easy to have two CIDs that point to the same file.

the way IPFS currently works

This doesn't seem like a good thing to base spec decisions on. Changing how IPFS works is just another spec decision away, after all..

Those aren't filesystem operations so I would not expect them to retain permissions.

I think that people having mounted Dropbox folders on their machine almost exactly mimics the behavior that users would expect from IPFS.

could use different chunking strategies, or whatever so it's already easy to have two CIDs that point to the same file.

That's true, but A) if they the chunking defaults there's no problem B) there are possible solutions for dealing with multiple chunkers since fundamentally the two files can have a canonical CID (e.g. sha256 of the entire file without any unixfs intermediate blocks), however these solutions cannot properly work if the actual content is different (which it would be since metadata contains information)

Changing how IPFS works is just another spec decision away, after all

Unless there's a proposal for how to fix the problem I'm wary of "we'll fix it later". Also, my understanding is that unixfsv1.5 is supposed to be a temporary fix and so the probability of "later" coming before unixfsv1.5 is deprecated is even lower.

This doesn't seem like the right way to do it to me. If I copy a file to another location on my filesystem, it doesn't lose its permissions, nor will it lose its permissions if I remove it from its containing directory (to put it in a tarball, for example).

@achingbrain re-reading the thread I realized this is also a flawed model. In general it is the tool that does the copying that preserves various bits of metadata, and moreover only does so if the context allows it to. Consider:

rabbit@Ahasver:/dev/shm/stuff$ ls -l total 0 rabbit@Ahasver:/dev/shm/stuff$ touch source rabbit@Ahasver:/dev/shm/stuff$ chmod 666 source rabbit@Ahasver:/dev/shm/stuff$ umask 0022 rabbit@Ahasver:/dev/shm/stuff$ cp source dest rabbit@Ahasver:/dev/shm/stuff$ ls -l total 0 -rw-r--r-- 1 rabbit rabbit 0 Nov 13 10:29 dest -rw-rw-rw- 1 rabbit rabbit 0 Nov 13 10:29 source

The default behaviour is to create the new file with the same permissions, umask changes the defaults.

If we access a file via it's CID we won't have the option to preserve permissions as they are simply not there.

alanshaw · 2019-10-24T08:49:16Z

UNIXFS.md

+
+The `defaultMetadata` field can be used in conjunction with the repeated `metadata` field to specify metadata "defaults".
+
+1. The first `defaultMetadata` field encountered when traversing a sharded directory takes precedence.


Somewhat related point, does defaultMetadata apply to just the directory it is defined in, or can I specify it in a parent directory and have a parent/sub or parent/sub/sub inherit it? I assume no, but maybe worth clarifying?

No. But I'd like to revisit #223 (comment) and consider dropping it before moving on.

achingbrain · 2019-10-29T09:31:34Z

UNIXFS.md

+For some context, this solution was chosen over embedding the metadata in the file itself because:
+
+1. It allows accessing the metadata without downloading the target files.
+2. More importantly, it avoids changing the root hash of the target files to allow for better deduplication.


I don't think metadata is tied to file names - you could be copying /ipfs/QmHashOfSomeBinaryFile from ipfs-at-large to a location in your mfs, at which point it would receive a filename, but if it was executable, you'd want to retain that information.

UNIXFS.md

achingbrain · 2019-10-29T09:54:45Z

UNIXFS.md

+
+For some context, this solution was chosen over embedding the metadata in the file itself because:
+
+1. It allows accessing the metadata without downloading the target files.


When we do a directory listing, we have to fetch at least the root node of every directory entry (or traverse a HAMT) in order to say whether it's a file entry or a sub directory entry so we already have a mechanism for accessing metadata without having to materialise the whole file, so I'm not sure what advantage is gained here.

You're right.

We could consider additionally storing the file type but I'm worried that will be even more complicated.

I'm not sure storing the file type in the directory is a good idea since we'd lose that information if we referenced the DAGNode directly (e.g. via /ipfs/Qmfoo) - I think it's fine as it is.

c/o @warpfork, quite a good discussion about metadata+files that's relevant here I think, feeding into unixfsv2 design decisions https://hackmd.io/4lqtycvdQN2WTspBLpy3qw

achingbrain · 2019-10-29T10:02:49Z

UNIXFS.md

+For some context, this solution was chosen over embedding the metadata in the file itself because:
+
+1. It allows accessing the metadata without downloading the target files.
+2. More importantly, it avoids changing the root hash of the target files to allow for better deduplication.


Since the file content is by far larger than the UnixFS data describing the file and is not affected by changing the metadata, is this a de-duplication win that's worth only having metadata available if a file/dir is accessed via a directory?

I guess if depends if the file fits into a single chunk or not?

Yes, but I don't know if that's a bad thing or not. My assumption is that most non-trivial uses of IPFS will see files that are more than one block in size.

Most files are actually pretty small and 256KiB (the default block size) is pretty large.

One extra byte per file in a directory isn't worth the complexity.

alanshaw

I just re-read this and it looks to me like a good enough incremental improvement until unixfs2 lands. Shall we ship it 🚢?

dirkmc · 2019-11-11T14:36:15Z

UNIXFS.md

-	optional string MimeType = 1;
+	optional string mimeType = 1;
+	optional uint32 mode = 2;
+	optional int64 mtime = 3;


Agree that the world didn't start in 1970 :) In that case should the default value be something other than 01/01/1970?

dirkmc · 2019-11-11T14:37:40Z

UNIXFS.md

+      {}
+  },
+  Links: [
+    {Name: "foo", ...}, // mode: 0644 (default)


Above we say that the mode defaults to 0755 if unspecified but here we say mode: 0644 (default)

ribasushi · 2019-11-11T16:32:39Z

... before unixfsv1.5 is deprecated

... good enough incremental improvement until unixfs2

The apparent lack of alignment on what the IPFS project is "locking" itself into by this spec makes me uneasy.

The bottom line is that anything that ships here is going to be supported forever ( n.b. an uncomfortably long time), just like v1 of unixfs. That is there is no future where a web gateway can elect not to render all the metadata as specified here, and still be considered compliant. Doing otherwise is a betrayal of one of the core principles of IPFS: an address is forever. What good is an immutable data structure if endpoints do not honor the contracts for displaying it?

Given the above we are not really discussing a "stop-gap" solution, we are discussing another fork of the ipfs web gateway and cli tools. Distinct from v1, and distinct from v2. In this context it seems to me that the path chosen is too "narrow".

I wonder if we can put together a 2d matrix of sorts ( in the wiki? ). On the x axis ( columns ) it would have various interested group ( packagers, mirror operators, archivers, etc ). An don the y axis ( rows ) it would have different pieces of metadata. The resulting "cluster of interest" is what should likely not be left out of v1.5, even if there's a lot of pressure from certain "columns" to get something out there.

P.S. I would have put this matrix together myself, if my free time wasn't vapourized by a set of unrelated events all happening at the same time :/

aschmahmann · 2019-11-11T17:13:33Z

That is there is no future where a web gateway can elect not to render all the metadata as specified here, and still be considered compliant.

I may be in the minority here, but why not? Gateways are just one way of viewing data and UnixFS is just one of the multitude of IPLD formats people can use to store their data. Would it be so bad if even immediately after UnixFS v1.5 the gateways still didn't show any ACL metadata?

@ribasushi Would having a document/proposal for how IPFS plans to deal with the transitions of updating UnixFS help with your concerns? I am certainly wary of giving this spec first party support indefinitely so hearing about how the transition is going to work would ease my concerns.

ribasushi · 2019-11-11T20:17:34Z

Would it be so bad if even immediately after UnixFS v1.5 the gateways still didn't show any ACL metadata?

The answer is naturally "no". What would be bad is if they start showing it, and then stop doing so at some point in the future. Same for various get-like CLI calls, MFS, and so on.

Would having a document/proposal for how IPFS plans to deal with the transitions of updating UnixFS help

Absolutely! Now that you verbalized this - it seems obvious that most ( all? ) current and future directly user-visible specs need to include an "Obsolescence statement" akin to that of the Raspberry Pi foundation. This would allow a project like e.g. Peergos to make informed decisions regarding which specs they can follow and which ones they would need to re-implement in-house to de-risk their product offerings ( which may have very different content half-lives compared to the typical user of a certain spec ).

rvagg · 2019-11-12T02:44:29Z

UNIXFS.md

+
+UnixFS currently supports three metadata fields:
+
+* `mimeType` -- The mime-type of the file. This generally shouldn't be used.


perhaps this "shouldn't" could be explained? is it deprecated? does it have a special case use?

I'm also very confused by this. I didn't know it exists in any existing data; it won't exist in Unixfsv2; and I thought we'd had multiple roundabouts in github issues in which multiple people highlighted the issue that mimetypes are not generally considered a property of a filesystem (rather, they tend to show up in applications, servers, and other things that are routing control flow in some way) and that they shouldn't thus be in unixfsv{n}.

This type was already reserved by the existing metadata feature.

Is there any reason not to leave it there, in the deprecated state that it becomes there?

If it's deprecated, it can still be used -- but "shouldn't" be. Which is what this text says anyway. If it's deprecated and we don't want it to be used, we should prefer to draw proportionally less attention to it, and leaving it in the depricated notes should be sufficient for that.

Stebalien · 2019-11-13T17:10:40Z

So, we just had yet another discussion and have decided to put the metadata in files.

TL;DR:

It's much easier to implement and maintain.
The metadata goes with the file.
We can still deduplicate by putting the actual data into a separate block instead of inlining it into the root block (when unusual metadata is present). That is, the root "file" block can just contain metadata and a link to the real data.
We might be able to embed this root file block into an inline CID (with some minimal metadata).

@achingbrain will write up a more thorough decision^† along with a new spec.

^†Decision for everyone at the meeting. Nothing is final till it's in a spec on GitHub, merged, and implemented. But we need to move forward on this ASAP.

I didn't include the pros and cons of the other options because I don't think they should be included in the spec. That is, if I'm implementing this, I read the spec to understand what I should implement and how it should behave, I don't want to read a bunch of exposition on how we got to where we are. The decision log is important though, should it go in `ipfs/notes`? Or maybe we could have accomanying `*-notes.md` files? Follows on from/supercedes #223 s

I didn't include the pros and cons of the other options because I don't think they should be included in the spec. That is, if I'm implementing this, I read the spec to understand what I should implement and how it should behave, I don't want to read a bunch of exposition on how we got to where we are. The decision log is important though, should it go in `ipfs/notes`? Or maybe we could have accomanying `*-notes.md` files? Follows on from/supercedes #223

I didn't include the pros and cons of the other options because I don't think they should be included in the spec. That is, if I'm implementing this, I read the spec to understand what I should implement and how it should behave, I don't want to read a bunch of exposition on how we got to where we are. The decision log is important though, should it go in `ipfs/notes`? Or maybe we could have accompanying `*-notes.md` files? Follows on from/supersedes #223

achingbrain · 2019-12-12T17:32:59Z

Superseded by #226

Stebalien force-pushed the feat/unixfs-v1.5 branch from 8a9fdd2 to 1335b49 Compare October 23, 2019 04:49

Stebalien added 2 commits October 22, 2019 21:51

unixfs: add metadata to spec

94be0a0

unixfs: add a default metadata field

583bf06

Stebalien force-pushed the feat/unixfs-v1.5 branch from 1335b49 to 583bf06 Compare October 23, 2019 04:52

Stebalien commented Oct 23, 2019

View reviewed changes

Stebalien requested review from alanshaw, mikeal, whyrusleeping, vmx, warpfork, achingbrain, aschmahmann, dirkmc and djdv October 23, 2019 04:53

Stebalien commented Oct 23, 2019

View reviewed changes

alanshaw reviewed Oct 23, 2019

View reviewed changes

Stebalien added 3 commits October 23, 2019 23:51

unixfs: be very explicit on how metadata works

429f6d6

unixfs: add some context explaining why we put metadata in directories

6ccb03b

unixfs: add context on why the metadata object has been deprecated

f3577ef

alanshaw approved these changes Oct 24, 2019

View reviewed changes

achingbrain reviewed Oct 29, 2019

View reviewed changes

Stebalien added 2 commits October 31, 2019 18:58

unixfs: expand on how metadata works with shareded directories

6cf354f

unixfs: remove default metadata

2d63707

One extra byte per file in a directory isn't worth the complexity.

achingbrain mentioned this pull request Nov 7, 2019

RFC: unite the files APIs ipfs-inactive/interface-js-ipfs-core#552

Draft

alanshaw approved these changes Nov 11, 2019

View reviewed changes

dirkmc reviewed Nov 11, 2019

View reviewed changes

rvagg reviewed Nov 12, 2019

View reviewed changes

achingbrain mentioned this pull request Nov 14, 2019

docs: adds spec sections for mode and mtime metadata (Unixfs v1.5) #226

Merged

achingbrain closed this Dec 12, 2019

achingbrain deleted the feat/unixfs-v1.5 branch December 12, 2019 17:33

ribasushi mentioned this pull request Feb 10, 2020

Research/quantify performance envelopes of multiple CDC algorighms #227

Open

24 tasks


		Fields:

		* `mimeType` -- The mime-type of the file. This generally shouldn't be used.


		The `defaultMetadata` field can be used in conjunction with the repeated `metadata` field to specify metadata "defaults".

		1. The first `defaultMetadata` field encountered when traversing a sharded directory takes precedence.


		For some context, this solution was chosen over embedding the metadata in the file itself because:

		1. It allows accessing the metadata without downloading the target files.


		UnixFS currently supports three metadata fields:

		* `mimeType` -- The mime-type of the file. This generally shouldn't be used.

unixfs v1.5 metadata support #223

unixfs v1.5 metadata support #223

Conversation

Stebalien commented Oct 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw Oct 24, 2019 • edited Loading

Choose a reason for hiding this comment

mib-kd743naq Oct 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien Nov 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

achingbrain Nov 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ribasushi commented Nov 11, 2019

aschmahmann commented Nov 11, 2019

ribasushi commented Nov 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Nov 13, 2019

achingbrain commented Dec 12, 2019

alanshaw Oct 24, 2019 •

edited

Loading

mib-kd743naq Oct 24, 2019 •

edited

Loading

Stebalien Nov 1, 2019 •

edited

Loading

achingbrain Nov 1, 2019 •

edited

Loading