Fun things to do with GStreamer command-lines

This is a bit of a long post, but bear with me ๐Ÿ™‚

Last week, I found out that OpenSolaris has recently added the Video4Linux2 APIs, and now provides a v4l2 driver for UVC compatible video cameras. That’s slightly funny, of course, because it has the word ‘Linux’ right there in the name. I think it’s really cool to see that the OpenSolaris guys aren’t suffering from NIH syndrome about that though.

To continue, I’ve had my eye on the Logitech Quickcam Pro 9000 for a little while, and it happens to be a UVC camera. This seemed like a good opportunity, so I ordered one. Nice webcam – I’m really pleased with it.

After that arrived, I was playing with V4L2 on one of the machines at work, and put a couple of patches from Brian Cameron into GStreamer‘s gst-plugins-good module to make our v4l2src plugin compile and work. The biggest difference from the Linux V4L2 implementation is that the header is found in /usr/include/sys/videodev2.h instead of /usr/include/linux/videodev2.h. That, and a small fix to gracefully handle ioctls that the OpenSolaris usbvc driver doesn’t implement, and I was up and away.

Coincidentally, Tim Foster was looking for talks for the Irish OpenSolaris User Group meeting last Thursday night. I thought people might enjoy a quick demo, so I volunteered.

I started off by showing that the camera shows up nicely in Ekiga, but I don’t have a SIP account so I wasn’t able to do much more there. Also, I really wanted to show the camera in GStreamer, since I’m a GStreamer guy. Note, for people who want to follow along at home I was using the version of GStreamer’s V4L2 plugin from CVS. It’s in the gst-plugins-good module, which is due for release on the 18th February.

I tried a simple pipeline to show the camera output in a window:

gst-launch v4l2src ! ffmpegcolorspace ! xvimagesink

This captures the video using v4l2 (v4l2src is the name of the element responsible), and feeds it via the ‘ffmpegcolorspace’ element to the ‘xvimagesink’ output. xvimagesink feeds video the XVideo extension, and ffmpegcolorspace provides for any format conversion between the frame buffer formats the camera supports, and what Xv can handle. Actually, this pipeline didn’t work by default on Tim’s laptop. For some reason, it tried to capture at 1600×1208 pixels, which the camera doesn’t support. It might work for you, not sure.

Anyway, the obvious fix was to explicitly choose a particular resolution to capture at:

gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! xvimagesink

This is the same as the first pipeline, with the addition of the ‘video/x-raw-yuv,width=320,height=240’ bit, which in GStreamer jargon is called a ‘caps filter’ – it filters down the set of formats that are allowed for data transfer between the 2 elements. By default, the pipeline will ask v4l2src and ffmpegcolorspace what formats they have in common, and pick one. By filtering it down, I’m forcing it to choose 320×240. Doing that made a little window pop up with the video in it. It looked a little like this, although this one was actually from later:

OSUG USB camera shot

Next, I thought I’d show how to save the incoming video to a file. In this case, as an AVI with MJPEG in it:

gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! jpegenc ! avimux ! filesink location=osug-1.avi

The difference here is, instead of feeding the video to an Xv window, it goes through a JPEG encoder (jpegenc), gets put into an AVI container (avimux) and then written to a file (filesink location=$blah). I let it run for 5 or 6 seconds, and then stopped it with ctrl-c. The result looked like this:


Apologies for the blurriness – I was waving the camera around and it didn’t really get a chance to focus.

Alternatively, I could have used something like this to record to Ogg/Theora:

gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! theoraenc ! oggmux ! filesink location=osug-1.ogg

I can play the recorded video back with a pipeline like:

gst-launch filesrc location=osug-1.avi ! decodebin ! ffmpegcolorspace ! xvimagesink

This uses filesink’s cousin ‘filesrc’ to read from an existing file, feeds it to the nice ‘decodebin’ element – which encapsulates the ability to decode any audio or video file GStreamer has installed plugins for, and then feeds the result (a sequence of raw YUV video buffers) to ‘ffmpegcolorspace ! xvimagesink’ for colorspace conversion and display in a window.

Anyone who watched the clip might be wondering why there is a guy in the front holding up a bright orange folder. For the next trick, I wanted to show the nice ‘alpha’ plugin. By default, alpha simply adds a alpha channel with a given opacity to the video as it passes through. However, it also has a green-screen mode. Or, in this case, orange-screening.

First, I played the video I captured in totem, and paused it at a suitable frame. Then I used the ‘Take Screenshot’ item in the Edit menu to save it out as a png – which actually became the first photo above. Next, I opened the png in the Gimp and used the eyedropper to isolate the RGB triple for the orange colour. Somewhere around Red=244, Green=161, Blue=11

At this point, I used live video for the rest of the demo, but I didn’t think to capture any of it. Instead, I’ll use the short clip I captured earlier as a canned re-enactment. So, I ran the video through a pipeline like this (using v4l2src etc instead of filesrc ! decodebin):

gst-launch filesrc location=osug-1.avi ! decodebin ! alpha method=custom target-r=245 target-g=161 target-b=11 angle=10 ! videomixer ! ffmpegcolorspace ! xvimagesink

This pipeline uses the alpha plugin in ‘custom’ (custom colour) mode, to add an alpha channel based on the bright orange colour-key, and then uses ‘videomixer’ to blend a default transparent looking background in behind it. Here:

OSUG demo 2 - colourkeying

The colour-key effect breaks up a little in places, because the skin tones and the wood of the desk get a little too close to the orange of the folder. A better choice of colour and filming conditions are really needed to do it well ๐Ÿ™‚

And now for the really tricky bit:

gst-launch filesrc location=osug-1.avi ! decodebin ! alpha method=custom target-r=245 target-g=161 target-b=11 angle=10 ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink \
filesrc location=bg.avi ! decodebin ! ffmpegcolorspace ! mix.

This pipeline adds two things to the previous one:
1) It adds a ‘name=mix’ property assignment to the instance of the videomixer element. This makes it easier to refer to the instance by name later in the pipeline.
2) It adds a second filesrc, reading in a file named bg.avi, decoding it and then feeding it into the element named ‘mix’ – which is the instance of the videomixer element. Adding a period to the name lets GStreamer know that the connection should be made to an existing instance of something with that name, rather than creating a new instance of some element as all the other links do.

To save the output, I replaced the ‘xvimagesink’ with ‘jpegenc ! avimux ! filesink location=osug-3.avi’ to produce this:

OSUG demo 3 - colourkeying and compositing

There you go – a quick demo of doing a couple of fun things with GStreamer that don’t require writing any code. Have fun!

It’s nice to be loved

I can’t remember if I’ve had to make a brown paper bag release, so we’ll call this the first one. On Monday, I released 0.10.16 of the GStreamer Core and Base modules, and earlier today I released 0.10.17 of the same after we started seeing some reports of crashes.

It turns out that 6 months ago (0.10.14) we broke our ABI by deprecating some parts of the GstMixer class structures that we shouldn’t have, because they were in public headers resulting in a couple of base classes (GstMixerTrack and GstMixerOptions) shrinking when compiling with GST_DISABLE_DEPRECATED. Noone noticed at the time, because those bits of the class structure weren’t being used by anyone, and noone noticed if sub-classes allocated more space for their class structures than they now technically needed to.

Since then though, all the modules that use that piece of ABI got re-compiled, and now rely on a smaller parent class size, so when we inadvertantly put the structure entries by disabling the GST_DISABLE_DEPRECATED define in release builds, the OSS, SunAudio and Pulseaudio mixer sub-classes all started crashing. Which we would have caught in the release candidate tarballs, except that none of us thought about the fact that our pre-release tarballs were still building with GST_DISABLE_DEPRECATED, where the final release was not. Our fix is to bless the ‘new smaller ABI’, since noone has cared in 6 months that it changed, and the newer Base release reflects that. Core got another release too with a small compilation fix for Cygwin, mostly to keep the version numbers in sync.

I say “it’s nice to be loved”, because the broken release was out less than 2 days, and we’re already nearing 60 dupes on the bug ๐Ÿ™‚

I know this release is karma punishing me for being so bold as to try and write down GStreamer’s first time-based release schedule. It’s too carefully calculated to show up only in the final release tarball to be anything else.

Limpley Stroked

We’ve arrived safely in Lower Limpley Stoke safely. We didn’t see any of the trouble on the M5 that we feared, and the Avon river is no more than picturesquely swollen 2 streets from here.

We’re staying in a lovely hotel built out of a giant old house.

I was amused to notice that the railway bridge is mislabelled as belonging to “Limpley Stroke”, hence the title.

At Guadec, I enjoyed the LugRadio beers night, especially the part where several people actively tried to create a new EBay account in the pub for the sole purpose of trying to put Jono’s bag on auction and raise some cash to buy more beer.

I didn’t enjoy the part where 25 euro mysteriously disappeared from within a pile of papers on the desk in our hotel room. Nothing else was taken, including the n800 that was also sitting in the room, thankfully.

Tomorrow we’re exploring Bath, going to see Glastonbury Tor and Abbey, and hopefully getting to glimpse Stonehenge.

I closed another bug today, this one with prejudice. For the record, GStreamer developers do not support running the gst-ffmpeg wrapper against the system installed FFmpeg – you can do it, but if it breaks because FFmpeg changed something (as they do), you get to keep both pieces.

Update: Fixed the bug number.

Killing some wakeups


I committed Marc-Andre‘s patch for bug #152864 today, and closed the bug, which has been open since September 2004.

It adds the required GStreamer parts to close Bug 370937 รขโ‚ฌโ€œ Exessive CPU Utilisation and fix 10 wakeups per second in the volume control applet – at least when an ALSA mixer device is chosen. Other mixers still require polling, so they’ll wake up. Hopefully that will stop too as we implement poll-based notifications in the other mixer elements in GStreamer, where we can.

Does anyone know if OSS provides a select/poll based way to know when someone changes the mixer settings? Google searches haven’t been helpful. aumix at least seems to be using SIGALRM, which isn’t promising.

The Sun Audio mixerctl mentions being able to get a SIGPOLL signal sent when someone changes the mixer settings. That’s not a good interface for GStreamer (or anyone, really) to use though – anyone know if we can achieve the same thing using poll()?


Our original plan was to drive down to Bath tomorrow and stay there for 2 nights exploring the area. With all the flooding, we’re probably not going to have much fun with that. Uraeus told me on the phone that they had a rough time on the M5, but made it to Bristol today.

At this stage, I’m inclined to try the drive anyway, but I might change my mind in the morning.

For Ronald

It happens that I was (re)installing Ubuntu Dapper on my laptop this afternoon too, like Ronald was on his new Mac.
The laptop has been dist-upgraded since I first installed warty on it, and on the development repositories for most of the way. It had accumulated a bit of cruft because of all the hacks I had to do when things broke during the dev cycles, and I’d never been able to get either hibernation or suspend to work at all.
I copied /home, /etc and /var off to another machine and did a complete reinstall, which worked flawlessly.

Here’s some notes from the post-install. There aren’t many, because by and large things worked flawlessly for me:

  • X had the wrong resolution by default. I had to edit /etc/X11/xorg.conf and change it from 1024×768 to 1400×1050.
  • The 3D stuff was set up automatically, using the open source DRI drivers. For better DRI performance (xorg doesn’t get them right on its own), I added these lines to the xorg.conf:
    • Option “AGPMode” “4”
    • Option “AGPSize” “64”
  • Multimedia.
    • plugins – the usual story – too many formats are encumbered for one reason or another and can’t be included in the default install, so playing anything from Apple Trailers wouldn’t work. Following the instructions in the Ubuntu wiki, I added the universe and multiverse repositories to my apt configuration in /etc/apt/sources.list and installed the packages for restricted formats:
      • gstreamer0.10-ffmpeg gstreamer0.10-pitfdll gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse
    • After that, I could play trailers on the Apple trailers site, and Rhythmbox could play the few m4a files I have access to.
    • I only partially agree with Ronald about the ‘cryptic fd://0 errors’ when trying to play things in totem’s mozilla playing.The title of the box was poorly labelled ‘Could not play fd://0’ (this should be fixable in totem to show the URL instead), but the text of the dialog box clearly stated something along the lines of ‘You do not have a decoder installed to handle this file’ and put me on the right track. However, it doesn’t say what format or plugins to look for, which sucks. There’s been some discussions in the past about building a mechanism for telling the user exactly which plugin/package they should look for, but noone has ever built it.
    • software – totem & rhythmbox are the main multimedia apps I use. Both were installed, but the totem-gstreamer-firefox-plugin wasn’t for some reason. Simple enough to install that.
    • ALSA set itself up nicely with DMIX, so no worries about sharing the sound card between applications. Unfortunately, that tends to cause shuddery video playback in GStreamer and (to a lesser extent) xine for reasons that are best explained in another post.
  • Hibernation:
    • Encouragingly, worked the first time I tried it – immediately after installing and logging in, before doing any configuration or anything.
    • Sadly, failed in the old way an hour later after I’d installed stuff, copied some pieces of my home directory back across, and had a few apps running it failed in the same way as before the reinstall: by getting as far through the hibernate as ‘turn the screen off’ and then sitting there flashing the hard-disk-access light for the next 10 minutes until I switch it off via the power button.
    • I really wish I knew how to fix it – the hard disk in this machine is so slow it takes a good 4-5 minutes to boot up, so I hate turning it off, and carrying it around still turned on isn’t much better.

Guadec roundup

Had a ball of a time at Guadec last week. It was nice and close to Barcelona, which made a nice change – traditionally it’s been held in the opposite hemisphere from me and I haven’t been able to get there.

Got to meet a bucket-load of interesting people that have previously just been hackergotchis on various planets, or nicks on IRC channels.

Socially, it was an absolute blast.

Technologically, wonderful: loads of great hacks being shown off, and heaps of in-depth technical talks.

Business-wise, it was nice to feel love from the community about the things we’re trying to do with multimedia at Fluendo, and great to see everyone wearing our hats ๐Ÿ™‚

The Elisa guys did some very nice demos, and for anyone that missed it, all the source code for Elisa is available in Subversion. Check out the project page for more details.

Edward gave a nice lightning talk showing off some of the things he’s been doing in Pitivi. It came off nicely, but I think made it hard for the audience to see just how cool it was – appending video files in multiple source formats and resolutions to a timeline and then scrubbing through them in realtime like it was nothing really shows how much polish he’s added to GStreamer and Gnonlin since last year. I know the Jokosher guys certainly appreciate it, since it’s made things considerably easier for them.

Andy and Wim gave a talk on some new things in GStreamer 0.10, including a neat ~20 lines-of-python demo of distributed synchronised playback using GStreamer’s network clock implementation.

It was also nice to meet a bunch of new (to me) people that are doing fabulous things with GStreamer 0.10: MDK (from Diva), the Jokosher gang, Nokia N770 people, Opened-Hand, Iain Holmes (from Marlin), MacSlow, Burger, Zeenix, and the Elisa guys – even though they work for Fluendo too, this was the first time I actually got to meet Phil and Loic. And of course, Bastien and Tim – always good guys to hang out with.

We (the GStreamer team) have worked really hard to make GStreamer 0.10 as good as it is, and it’s nice to see people reaping the benefit of our collective labour.

I found Jim Getty’s talk on the OLPC project quite exciting – a project that really has a chance to make a big difference to the world. We took possession of one of the dev boards, and I’m looking forward to seeing what we can do to help them in the multimedia area.

We had a good audio BOF on Friday afternoon, talking about Pulseaudio and how it can be more than a better-ESD for the Gnome desktop. Monty from talked about writing a FUSD based wrapper for OSS, and possibly even ALSA devices. This would replace the existing system of LD_PRELOAD hacking using padsp and esddsp, and make pulseaudio a serious candidate for The Linux Desktop Audio Solution. I’m really keen to see whether we can make it fly, since the Linux Audio swamp has needed draining since forever.

I need to write up a summary of some of the things I’ve been hacking on lately, but I might leave that for another night. I’m also looking forward to putting in some GStreamer related talk proposals for LCA next year.