I recently encountered a problem of dealing with arbitrary sized files in a browser based application.
Just to give a context this app also does e2e encryption on these files, the webcrypto api doesn’t have streaming interface either. So, the only way to deal with arbitrary sized files is to chunk them up and make sure that you use a correct form of authenticated encryption on the chunks themselves.
Second problem I encountered is how to save a big file from a browser chunk by chunk? Apparently, appending to a file is not possible. The only api you have is to save a Blob as a whole.
I looked at the Blob implementations in Chromium and Firefox. They do some kind of caching to disk for big Blobs. For example if the chunk is over 250MB, then Chrome directly saves it into disk instead of memory. Which is great, but those chunks are too big. What Chrome doesn’t do is to cache a concatenation of two Blobs that are originally in memory. (Actually, the full picture is more complicated)
Firefox caches Blobs to disk if there is a memory pressure. This worked, however stressing a system like this wasn’t satisfactory for me.
Luckily, there is another approach by using Service Workers. A service worker sits between your app and your backend. It can intercept calls and can respond to them like your server. We can achieve asynchronous file saving by the following steps.
We send a message to the service worker to prepare it. The message contains a file name and its size we want to save. It responds with a newly generated url.
The app then triggers a file download on that url.
The request will be catched by the service worker and it responds to it by putting a ReadableStream in its body.
Then, the app starts processing the chunks one after another. When a chunk is ready to be appended to the file, the app transfers it (no copying) to the service worker. The service worker then enqueues it in the ReadableStream.
Congrats, you just appended to a file asynchronously.
The original idea is not from me, but I simplified it a lot. Here is a working code : https://github.com/simonracz/async-save
Frontend development is weird.