Streaming versions of ipfs.files.add buffer entire data in memory before doing HTTP POST

[lidel]

ipfs.files.addPullStream([options])
ipfs.files.addReadableStream([options])

Problem Summary

As noted here streaming over HTTP does not work the way people expect: it ends before HTTP transport and entire thing is buffered in memory before HTTP POST is called with contents of the buffer.

This means when HTTP POST is sent to /api/v0/add browser has entire payload in memory.
Streaming of huge files is impossible over HTTP.

This is bad because:

• breaks progress reporting (https://github.com/ipfs/js-ipfs-api/issues/748)
  (0% until entire thing is received by remote server, then 100% instantly)
• web browsers crash (Uploading big files crush page in Chrome ipfs-inactive/js-ipfs-http-client#654)
  or return silent errors due to increased memory allocation
  (crash when dropping a big file ipfs-companion#464 (comment))

Potential Solutions

Option A: Streams API

Use a fetch call with Streams API.
Looks like something that will be possible in the future, step 10 from fetch.spec mentions ReadableStream

Blocked missing vendor support (https://caniuse.com/#feat=streams)

• Chrome: No support for BYOB ("bring your own buffer") stream readers
• Firefox: No support for WritableStream

Option B: Poor Man's Streaming

We could buffer up to a point and then send a chunk as a separate POST call to HTTP API.
Implementation may get ugly very fast, requires increasing complexity of go-ipfs.

Option C: (?)

(?)

Related Issues

cc ipfs-inactive/js-ipfs-http-client#654, ipfs-inactive/js-ipfs-http-client#788, https://github.com/ipfs/js-ipfs-api/issues/748, ipfs/ipfs-companion#464

[hacdias]

cc ipfs/ipfs-webui#762

[JiaJiaJiang]

I haven't read all code of the uploading logic because Iam not familiar with react, but if the data will not be processed before uploading ,here is an advice:
XHR can send a FormData object, and the formdata object can carry files that gave into the input element. This way will not cause the memory problem and all thing are handled by the browser. The uploading progress can be got from the xhr object too, nothing will be blocked.
Here is the example of FormData object with input files: https://developer.mozilla.org/en-US/docs/Web/API/FormData/append

[thienpow]

looks like a particular problem of Chromium based browser having this problem. Safari is ok to upload bigger than 150mb.

[lidel]

I believe this was fixed by @Gozala in #3184