progress-bar support

[JohnAllen]

I'd like to take a stab at implementing progress-bar support for js-ipfs. Any pointers and/or preferred implementation approaches? E.g., packages or homebrewed solutions?

[JohnAllen]

@diasdavid already referenced a related js-ipfs-unixfs-engine issue.

[daviddias]

Woot! Thank you @JohnAllen :D

With regards to implementation, as long as it provides the information necessary using the same pattern that go-ipfs uses, so that we can expose that through the http-api and use js-ipfs-api as a client, we are golden. Of course this will lead you to change a bit the ipfs-unixfs-engine, but that is expected.

With regards to UI, there is this list of awesome CLI utilities -- https://github.com/sindresorhus/awesome-nodejs#command-line-utilities -- which you can take inspiration from. One that seems to get this job right is: https://github.com/visionmedia/node-progress

Another note. As you implement this through the daemon, you will hit #823 as well. Consider implementing first all of what is required to have the progress bar working while offline and then fixing the buffering of files to be sent out.

[JohnAllen]

Cool appreciate the tips. And by "ipfs-unixfs-engine" you actually mean the "js-ipfs-unixfs-engine", correct? I ask because there isn't an "ipfs-unixfs-engine" repo.

[daviddias]

@JohnAllen that is correct :)

[pgte]

@JohnAllen js-ipfs-unixfs-engine implements a duplex stream, where the input has the individual files (represented by a path and the individual content as streams) and outputs the hash, path and node size for each file. So using the unixfs output stream you already have a notion of progress (either through the number of files or their relative size).

What you now need is the total size (either number of files and / or total size), which you can only compute by having direct access to the filesystem. This means that js-ipfs-unixfs-engine can't help you with that.

I think that the best place to figure get the total size before starting pumping the contents into the unixfs stream is in js-ipfs itself, namely on src/cli/commands/files/add.js.

Hope this helps. Ping me if you need help.

[pgte]

@JohnAllen Been reading through ipfs-inactive/js-ipfs-unixfs-engine#123, and there's a good point:

The emitted results contain the IPFS total object size and not contain the original file size. This needs to be solved first. Hit me if you need pointers in js-ipfs.unixfs-engine for that.

[JohnAllen]

@pgte Mind pointing me to where the object size is emitted in js-ipfs-unixfs-engine? Think I can do the rest just not sure where that is. Thanks!

[pgte]

TLDR: it looks like, if I'm reading the code right that, after all, you can use the file size output to calculate the total file output without changing unixfs.

Right now, the emitted file contains the file size:
https://github.com/ipfs/js-ipfs-unixfs-engine/blob/master/src/exporter/file.js#L43

The size is computed in the file.fileSize() method, which sums the total of the block sizes.

When the file is a single DAG node, the fileSize() contains the size of the data, which is the correct file size.

When the file is composed by more than one DAG node (for bigger files), the block sizes are computed from all the leave sizes (a file may be more than one node) here, which in turn are computed from the individual node file size.

Please tell me if you experience otherwise.

[JohnAllen]

FYI working on this. Hope to PR this week. Have simple numFiles based approach working but will work on basing off of fileSize soon.