Requirements

[kevina]

This issue is to gather the requirements and/or desired features for the next generation unixfs directory.

@whyrusleeping @Stebalien @magik6k

ref: ipfs/kubo#4229

[whyrusleeping]

also cc @diasdavid @dignifiedquire @Kubuxu

[daviddias]

also cc @pgte

[Stebalien]

Need:

• Executable bit (package manager)
• Extended attributes (package manager)
• Inline file type (performance)

Maybe:

• Inline small files (performance).
• Inline small directories (performance).

---

Question: Do we want to duplicate the file type? That is, store it in the file and in the directory or just in the directory.

[Kubuxu]

I would say it would be nice to preserve all information we can pull from filesystem. My guess is that basing off the tar format (as in what it stores) would be good start.

Storing extended attributes would be very nice too. It would allow for storage of for example ACLs. They could be a special case of generic metadata.

[whyrusleeping]

I think there should be a base mode of maximal deduplication, as well as a "preserve everything" mode.

…

On Mon, Sep 18, 2017, 10:35 AM Jakub Sztandera ***@***.***> wrote: I would say it would be nice to preserve all information we can pull from filesystem. My guess is that basing off the tar format (as in what it stores) would be good start. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABL4HKBnUqpR2YusFoqoZiJbMElsiCBoks5sjqmugaJpZM4PaPm1> .

[Stebalien]

@whyrusleeping actually, we can get the best of both worlds by storing this metadata in the directories but not the files. If we really need to attach the metadata to the files themselves, we could add a special metadata node that adds metadata to a file (although this would only be used when linking to files directly which probably won't be that common).

[Kubuxu]

I think it might be also useful to start producing directory wrapped files by default. It would be even more breaking change but would clear up more confusion and preserve more metadata.

[whyrusleeping]

Also needed is support for X-Attrs

[kevina]

@whyrusleeping and anyone, I can create a draft proposal to get things moving. However, I am a little confused on the format the spec should be in. Since there is ipld in the name I take it it should be in ipld, which is basically JSON, but with the use on integers. An extremely primitive spec might be:

[{"afile" {"/": "Qm...", "mtime": "...", "size" : 1234},
  "adir" {"/", "Qm...", "isdir": true},
  "aexec" {"/", "Qm...", "isexec": true, "mtime": "...", "size": 768783}}]

but the ipld spec does not spell out how this will be serialized in binary formats as far as I can tell. The translation to cbor is pretty straightforward, but I don't like the idea of including string hash keys in what should be a compact binary format, I would rather assign integer to properties such as "/", "mtime", "size", "isdir", "isexec". If we what to serialize to a protobuf we have to assign integer values to the properties anyway or have a very inefficient protobuf encoding.

Note on design choice of the spec above: (1) I believe very strongly that some sort of timestamp should be included by default. (2) There are multiple ways to define the "size" of the directory. The current unixfs defines this as the size of the directory and the contents, real unix filesystems define the size of a directory as the size of the directory entry, but not its contents, I think it is better just to leave it out.

[kevina]

As far what to include, the tar spec might be a good start: https://en.wikipedia.org/wiki/Tar_%28computing%29

Although there are some fields, such as the owner and user ID that are probably not generally useful and should not be included by default. Also there is the question if how unix like we want this, for example the "file mode" is very unix specific.

[Stebalien]

@kevina, a few notes:

• IPLD is a really just a meta format that supports many different serializations. It's a way of mapping unstructured binary to structured, linked data. Usually, we prefer using DagCBOR if at all possible.
• The {"/": "Qm..."} is just how we encode links in JSON (because, unlike e.g. CBOR, JSON doesn't support "tags" for custom types). You can't add other fields to links. If you find any documents saying otherwise, please tell me so I can fix them. Warning: the spec is a bit out of date (like all of our specs...).
• I doubt the overhead of string names will be all that bad, even for tiny files. First, we can omit key/value pairs with default values. Second, CIDs themselves are generally ~36 bytes (in CBOR, including tags etc., links cost 41 bytes IIRC) so links will likely dwarf the cost of string keys. There have been grumblings (i.e., I've been grumbling) about adding compression support but that's still in the "it would be nice" stage.

---

We may be able to use xattrs for this but it would be nice to support linking to "related" files in IPFS. This way, we can write importers that can, e.g., parse a merkdown/HTML file, find all linked IPFS content, and link to it from the metadata (so it gets pinned along with the file).

[Stebalien]

• Support for CARs.

CARs are (to be speced) content addressable archives. That is, archives of IPLD DAGs. The primary motivation is to be able to dump an IPLD DAG onto a hard drive, ship it somewhere, and them import it back into IPFS.

However, I believe we can use them to bride IPLD and IPFS. That is, when I call ipfs files cp /ipld/QmId... /my-car, IPFS would link /ipld/QmId into my mfs filesystem (with a filename my-car). If I were to cat this file, IPFS would generate a CAR of the IPLD DAG on the fly.

The power here is not the ability to generate CARs on the fly (although that's really convenient), it's the ability to map structured linked data into unixfs without losing its structure. For now, most tools will just see it as a byte stream (a CAR). However, we can give it some extra metadata marking it as an IPLD DAG so tools that understand IPLD can operate on it that way.

This may also be useful for access control (ACLs on files lead to ACLs on structured data) but I haven't put much thought into that yet.

[Kubuxu]

What we have to take into account is not only format for unixfs itself but we also have to create arbitrary sized, seekable bytearrays on IPLD.