Generating movie from images in memory

[danvoyce]

I'm trying to convert a Canvas animation into a video. As the animation may not run at 60fps I'm thinking I want to create a still image of each frame like so

thumbnailCanvas.toDataURL('image/jpeg', 1);

and then stitch them all back together to make the video.

As I want this to run completely client side, I don't have the option of creating a temp /images folder and iterating over that.

Is this possible with ffmpeg.js? I've been playing around with worker.run(... but can't seem to figure out the correct arguments and options.

Thanks for any help (and awesome lib btw!)

[jeromewu]

I would recommend you start from this example: https://github.com/ffmpegjs/ffmpeg.js/blob/master/examples/browser/image2video.html

I think it includes key steps you need to achieve your scenario, a trap (?) here is that when you write an image file with Worker.write(), this image actually saves under /data directory.

A quick snippet for your reference (without verification):

const cnt = 0;
const b64Image = thumbnailCanvas.toDataURL('image/jpeg', 1);
await worker.write(`thumbnail_${cnt++}.jpg`, b64Image);

await worker.run('-framerate 30 -pattern_type glob -i /data/thumbnail_*.jpg -c:a copy -shortest -c:v libx264 -pix_fmt yuv420p out.mp4', { outputPath: 'out.mp4' });
const { data } = await worker.read('out.mp4');
for (let i = 0; i < cnt; i++) {
  await worker.remove(`thumbnail_${i}.jpg`);
}

[danvoyce]

Amazing! Thanks so much. This works like a treat, almost...

The video doesn't play exactly how it should. The video below should be animating smoothly, but it seems as if some of the frames are in the wrong order.

Any thoughts?

[jeromewu]

I think the issue here is about the naming of the images, you need to add padding zeros in the sequence number. So instead of thumbnail_1.jpg, using thumbnail_01.jpg is a better option.

[danvoyce]

Yes you are right. After some digging I found you can use %d instead which solves my issue, so now I'm using

-framerate ${fps} -i /data/%d.jpg -s ${width}x${height} -c:v libx264 -pix_fmt yuv420p out.mp4

Again thanks so much, this is a game changer for me. I seriously owe you a beer!

[Infl1ght]

That method works, and this is awesome!
But obviously that trick has a serious problem with memory usage. Imagine if you need to store thousands of FullHD images before encoding, it would require a huge amount of RAM.
So it'd be better to use image2pipe ffmpeg option. Unfortunately, I didn't succeed it.
Have you any thoughts about using image2pipe with ffmpeg.js?

[jeromewu]

I don't think it is possible to use image2pipe at the moment as it requires Linux pipeline syntax which is not possible to achieve at the moment. I think the best bet so far is to produce multiple video files with certain amount of images and concat them together.

[boraturant]

Hi,

This method doesn't seem to be working now. Has there been any change where the files are stored? (/Data/) or any other mechanics?

Error:
could find no file with path '/data/img_%04d.jpg' and index in the range 0-4

I can read the file with the worker.read method.

//For loop

var dataURL = this.canvas.toDataURL('image/jpeg', 1);

await worker.write( `img_${String(frameNo).padStart(4, '0')}.jpg`,
				dataURL
			       );
frameNo++;
//End loop

await worker.run(
			'-framerate 30 -start_number 0  -i /data/img_%04d.jpg  -c:v libx264 out.mp4',
			{
				outputPath: 'out.mp4'
			}
);

[incube8r]

I followed this guide but ffmpeg.js does not seem to be able to read the files written:
https://stackoverflow.com/questions/61167521/encode-png-images-to-mp4

[jeromewu]

You can check this example: https://github.com/ffmpegjs/ffmpeg.js/blob/master/examples/browser/image2video.html

[freder]

await worker.write(`thumbnail_${cnt++}.jpg`, b64Image);

where does worker come from in this case?

[TomasGonzalez]

Did this broke?
I'm getting Uncaught (in promise) Error: ffmpeg.FS('readFile', 'out.mp4') error. Check if the path exists