Video Funhouse - Convert Videos In Your Web Browser

We (Nicole, Aaron, and I) have had a lot of fun doing Node Knockout the past few years. Last year (in 2012), our team got first place in the 'Completeness' category for our app asciigram, which converted a webcam stream into ASCII art.

This year we decided to up the game a little bit, and to try and convert any video file into another video format, while allowing filters to be applied to the video - all inside of the browser, without uploading anything. The result is Video Funhouse (here is our devcomo team page). I'm particularly proud of our project and team this year, since it was a very busy weekend all around.

Obviously, we knew we wouldn't be able to do this in a weekend on our own without relying on great libraries and tools. There is a project called Emscripten, which is an LLVM to JavaScript compiler, so we figured we could try building FFmpeg or avconv into JavaScript to do the heavy lifting for the project.

videoconverter.js

I've open sourced the build utilities and the ffmpeg.js file that we are using in the project as: videoconverter.js.

Building these projects with Emscripten was a little tricky, but actually quite simple when you consider what it was doing for us. What is also amazing about these tools is that I have little experience with build systems like this, and the tools generally just worked after a bunch of trial and error.

Why Would You Compile FFmpeg Into JavaScript?

Mostly just to see if it would work. Also, it seemed like it would be a fun project.

So, How Do You Compile FFmpeg Into JavaScript?

Here are some of notes I made while doing this, for future reference:

Compiling avconv seemed more promising at first, but when running it we bumped into some weird runtime errors with swscaler. Couldn't figure it out quickly, so focused on FFmpeg. We pulled down the latest Emscripten, and the latest FFmpeg:

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

git clone [email protected]:kripken/emscripten.git
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

You may need to also get the SDK to make sure Emscripten will work. The have documentation on their site about getting this to work. Here are the basic configuration options we ended up using. Most of the configuration options are actually disabling features that either don't make sense, or are there to save space. After a lot of trial and error, here is the minimal amount of flags to pass into emconfigure: --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic.

The first time we got a successful build it was a 100MB JS file. By removing a bunch of the other stuff, we get it down to 25MB. It could actually get smaller as well - we were unable to minify the script because the closure compiler process ran out of memory. The following commands are updated after we figured out some better options. The demo as it is right now doesn't run any optimizations on the JS file, and it is about double the size. We can't update the demo, as the weekend is over, but this commit is promising: ffmpeg.js: 190 additions, 1463201 deletions not shown.

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

emmake make;

emcc -O2 -s VERBOSE=1 -s ASM_JS= -s ALLOW_MEMORY_GROWTH=1 -v libavutil/*.o libavcodec/*.o libavformat/*.o libavdevice/*.o libswresample/*.o libavfilter/*.o libswscale/*.o *.o -o ../ffmpeg.js --pre-js ../ffmpeg_pre.js --post-js ../ffmpeg_post.js

Peformance

The FFmpeg process runs inside of a web worker to prevent locking up the UI. This was the first optimization, as it was necessary for doing any testing on the project at all. Data is piped back from stdout and stderr to the frontend of the demo app, which is listening for these and logging them to a pre tag. This logging could be limited or removed to limit the impact on the DOM, as a lot of data can be generated by the system.

We experimented with firing up 2 different workers, one for the metadata probe and actual processing, and one for taking screenshots of a video right after receiving it. But we noticed some general instability when both were running at once (many times a tab or browser would crash), so we removed the screenshotting functionality. I'm sure we could have traced it down and improved the performance if we had more than a weekend for the project.

The processing itself is a little slow, but I haven't done benchmarks to compare with native. I generally avoided processing videos of much size. Also, our demo app prints a lot of logs which may not be helping matters. We are looking into how to set up performance tests, but if anyone wants to help with this please let me know or submit a PR on the videoconverter.js project.

I believe this could be quite a bit faster - it couldn't actually use asm.js because it is explicitly disabled when ALLOW_MEMORY_GROWTH is defined. It would be worth experimenting to see if we could set a large max memory size and enable asm to see what kind of speedup we saw.

I'm interested to see if we can get some sort of performance benchmarking set up to compare how fast it runs in different JavaScript environments, and to see how it compares with native performance.

Potential Uses

This could be extended to bundle other encoding libraries. I wasn't able to figure out how to get the linking to work for libvpx, x264, libtheora, zlib, etc over the weekend, so certain formats cannot be encoded. It would be neat to have this functionality, as it would allow conversion into web friendly video, which could then allow further previewing and processing. As performance improves and FileSystem API support improve it may be possible to build a web-based video or audio editing tool using this.
Compile the other utilities - there are other programs that ship with FFmpeg that we are excluding right now. ffprobe in particular seems like it could be better for gathering the metadata after the initial load of the file.
As mentioned previously, this could be useful as a benchmarking tool.
Probably other things I haven't thought of as well.
Comments

sopel39 Says:

November 13th, 2013 at 2:53 am

Interesting stuff. I am trying to port ffmpeg myself, but rather for audio conversions. While I have managed to compile and run ffmpeg.js on audio files, the performance lacks quite badly (even though I do not have ALLOW_MEMORY_GROWTH enabled). Do you have some preliminary data on JS vs native performance? In my case I got really slow execution times (>10x slower vs native). Additionally I found Firefox slower than node.js, which is surprising given that Mozilla asm.js should be superior. Do you have some optimization ideas?

Brian Says:

November 13th, 2013 at 6:21 am

Have you enabled ASM and -O2 in the build script? In https://github.com/bgrins/videoconverter.js/blob/master/ffmpeg_build/build_lgpl.sh we are using -O2, but not ASM, since we need memory growth to be allowed. Also, how big is your generated ffmpeg.js file? You may consider disabling as much as possible in the configure call if you only need audio.

Regarding performance, Aaron is starting to set up some benchmarks to compare, but we don’t have enough data to say yet. You can check out https://github.com/bgrins/videoconverter.js/tree/master/perf-tests if you are interested. We are also setting up a jsPerf test – I will post later once that is ready.

sopel39 Says:

November 14th, 2013 at 5:35 am

I use -O3 for LLVM compilation and -O2 for bytecode to JS conversion. I compiled with minimal mp3 decoding and ogg encoding/decoding support. Output JS is rather small (I believe it never exceeds 10MB and I think it is optimized when compiled to HTML). Anyway, check Firefox console for asm logs, so you can be sure that asm actually kicks in. Why do you need growing memory?

Brian Says:

November 14th, 2013 at 6:17 am

Can you share your configure and emcc options to get these optimizations, especially for the LLVM compilation? I’d like to make sure I am optimizing and shrinking the file as much as possible. Here is what I am using: https://github.com/bgrins/videoconverter.js/blob/master/build/build_lgpl.sh.

Regarding memory growth, essentially any time I tried to convert a video I would get an error that I needed to enable that option. I’ve considered setting a very high max limit and compiling with asm.js just to compare the speedup.

sopel39 Says:

November 16th, 2013 at 6:55 am

My config script is:
RANLIB=emranlib ./configure –cross-prefix=em –cc=emcc –enable-cross-compile –target-os=none –arch=x86_32 –cpu=generic –disable-hwaccels –disable-stripping –disable-pthreads –disable-doc –disable-debug –disable-asm –disable-network –disable-everything –disable-ffplay –disable-ffprobe –disable-ffserver –disable-outdev=sdl –enable-optimizations –enable-protocol=file –enable-filter=aresample –enable-decoder=mp3 –enable-decoder=vorbis –enable-encoder=vorbis –enable-demuxer=mp3 –enable-muxer=mp3 –enable-demuxer=ogg –enable-muxer=ogg
for now, but I am planning to incorporate other libraries.

I think the biggest problem with practical use of ffmpeg.js for video is that without HTML5 filesystem API ffmpeg output has to saved to memory, which for bigger files cannot be done easily. Also I wonder what are the legal implications of having patented code distributed via compiled JS.

sopel39 Says:

November 16th, 2013 at 6:56 am

bugger … you do not need RANLIB env variable

Brian Says:

November 16th, 2013 at 2:48 pm

sopel39, Nice! I’m interested in how you have managed to incorporate mp3 and ogg. I’ve never quite been able to get those building alongside. Also, mind sharing your emcc call, and or wherever you optimize with -O3 and -O2?

Here are the two scripts I’m using for with/without asm:

Without ASM: https://github.com/bgrins/videoconverter.js/blob/master/build/build_lgpl.sh
With asm (needed to set TOTAL_MEMORY to 33554432 to get this work – maybe because there are more things enabled on my build?): https://github.com/bgrins/videoconverter.js/blob/master/build/build_lgpl_asm.sh

sopel39 Says:

November 17th, 2013 at 8:25 am

Hi Brian,

My emcc is: emcc -O2 -s OUTLINING_LIMIT=100000 -s TOTAL_MEMORY=67108864 dist/ffmpeg.bc -o dist/ffmpeg.js –pre-js ffmpeg_pre.js –post-js ffmpeg_post.js. Nothing really too fancy. I allocate around 64MB of memory, but I don’t think this is a big issue in modern browsers.

Anyway, I have finally put audio converter on the web: http://quick-apps.com/audio/ and the performance is really good using FF. It does not use external libs yet, but I will incorporate them later. The really interesting question is how to minimize the size of code that has to be loaded (different workers for encoding, decoding and cross worker binary communication?). Heck, you could probably port Java apps too with VMkit.

sopel39 Says:

November 21st, 2013 at 12:52 am

Hi,
I have compiled ffmpeg for audio transcoding with external libraries (lame, vorbis, fdk-aac) and pushed it to github: https://github.com/sopel39/audioconverter.js.

Brian Says:

November 21st, 2013 at 7:29 am

Wow, this is awesome! I just tried out the app, and it worked like a charm. There is so much potential for a site like this to simplify audio conversions for people. And thanks for sharing the build script – I’m going to have a look at if I can pull in some extra video codecs using the same method.

Pat Friedl Says:

October 25th, 2014 at 7:03 pm

Fantastic work on this. However, I’m running into an issue when trying to convert a set of PNGs into MP4 and it’s erroring out with “missing function madvise”. Do you know what configuration flags would be needed to enable the conversion of multiple images into an MP4? Thanks!

ffmpeg.js: Videos clientseitig konvertieren - entwickler.de Says:

May 8th, 2015 at 1:58 am

[…] ist auch der Weg, wie Grinstead zu der JavaScript-Variante gelangt ist. In seinem dazu entstandenen Blog Post schreibt er, dass er zunächst einige Laufzeitfehler beheben musste, bis eine halbwegs stabile […]

Brian Grinstead