See you later!
Some time in 2012, the GStreamer team was busy working toward the GStreamer 1.0 major release. Along the way, I did my part and ported the DVD playback components from 0.10. DVD is a pretty complex playback scenario (let’s not talk about Blu-ray)
I gave a talk about it at the GStreamer conference way back in 2010 – video here. Apart from the content of that talk, the thing I liked most was that I used Totem as my presentation tool
With all the nice changes that GStreamer 1.0 brought, DVD playback worked better than ever. I was able to delete a bunch of hacks and workarounds from the 0.10 days. There have been some bugs, but mostly minor things. Recently though, I became aware of a whole class of DVDs that didn’t work for a very silly reason. The symptom was that particular discs would error out at the start with a cryptic “The stream is in the wrong format” message.
It turns out that these are DVDs that begin with a piece of video that has no sound.
Sometimes, that’s implemented on a disc as a video track with accompanying silence, but in the case that was broken the DVDs have no audio track for that initial section at all. For a normal file, GStreamer would handle that by not creating any audio decoder chain or audio sink output element and just decode and play video. For DVD though, there are very few discs that are entirely without audio – so we’re going to need the audio decoder chain sooner or later. There’s no point creating and destroying when the audio track appears and disappears.
Accordingly, we create an audio output pad, and GStreamer plugs in a suitable audio output sink, and then nothing happens because the pipeline can’t get to the Playing state – the pipeline is stuck in the Paused state. Before a pipeline can start playing, it has to progress through Ready and Paused and then to Playing state. The key to getting from Paused to Playing is that each output element (video sink and audio sink) in our case, has to receive some data and be ready to output it. A process called Pre-roll. Pre-rolling the pipeline avoids stuttering at the start, because otherwise the decoders would have to race to try and deliver something in time for it to get on screen.
With no audio track, there’s no actual audio packets to deliver, and the audio sink can’t Pre-roll. The solution in GStreamer 1.0 is a GAP event, sent to indicate that there is a space in the data, and elements should do whatever they need to to skip or fill it. In the audio sink’s case it should handle it by considering itself Pre-rolled and allowing the pipeline to go to Playing, starting the ring buffer and the audio clock – from which the rest of the pipeline will be timed.
Everything up to that point was working OK – the sink received the GAP event… and then errored out. It expects to be told what format the audio samples it’s receiving are so it knows how to fill in the gap… when there’s no audio track and no audio data, it was never being told.
In the end, the fix was to make the dummy place-holder audio decoder choose an audio sample format if it gets a GAP event and hasn’t received any data yet – any format, it doesn’t really matter as long as it’s reasonable. It’ll be discarded and a new format selected and propagated when some audio data really is encountered later in playback.
That fix is #c24a12 – later fixed up a bit by thiagoss to add the ‘sensible’ part to format selection. The initial commit liked to choose a samplerate of 1Hz
If you have any further bugs in your GStreamer DVD playback, please let us know!
Hi world! It’s been several years since I used this blog, and there’s been a lot of things happen to us since then. I don’t even live on the same continent as I did.
More on that in a future post. Today, I have an announcement to make – a new Open Source company! Together with fellow GStreamer hackers Tim-Philipp MÃ¼ller and Sebastian DrÃ¶ge, I have founded a new company: Centricular Ltd.
From 2007 until July, I was working at Oracle on Sun Ray thin client firmware. Oracle shut down the project in July, and my job along with it – opening up this excellent opportunity to try something I’ve wanted for a while and start a business, while getting back to Free Software full time.
Our website has more information about the Open Source technologies and services we plan to offer. This list is not complete and we will try to broaden it over time, so if you have anything interesting that is not listed there but you think we can help with, get in touch
As Centricular’s first official contribution to the software pool, here’s my Raspberry Pi Camera GStreamer module. It wraps code from Raspivid to allow direct capture from the official camera module and hardware encoding to H.264 in a GStreamer pipeline – without the shell pipe and fdsrc hack people have been using to date. Take a look at the README for more information.
Sebastian, Tim and I will be at the GStreamer Conference in Edinburgh next week.
I upgraded my laptop to Ubuntu Karmic recently, which brought with it the ability to use my Bluetooth A2DP headphones natively. Getting them running is now as simple as using the Bluetooth icon in the panel to pair the laptop with the headphones, and then selecting them in the Sound Preferences applet, on the Output tab.
As soon as the headphones are connected, they show up as a new audio device. Selecting it instantly (and seamlessly) migrates my sounds and music from the laptop sound device onto the headphones. The Play/Pause, Next Track and Previous Track buttons all generate media key keypresses – so Rhythmbox and Totem behave like they’re supposed to. It’s lovely.
If that we’re all, it would already be pretty sweet in my opinion, but wait – there’s more!
A few days after starting to use my bluetooth headphones, my wife and I took a trip to Barcelona (from Dublin, where we live for the next few weeks… more on that later). When we got to the airport, the first thing we learned was that our flight had been delayed by 3 hours. Since I occasionally hack on multimedia related things, I typically have a few DVDs
with me for testing. In this case, I had Vicky Christina Barcelona on hand, and we hadn’t watched it yet – a perfect choice for 2 people on their way to Barcelona.
Problem! There are four sets of ears wanting to listen to the DVD, and only 2 audio channels produced. I could choose to send the sound to either the in built sound device, and listen on the earbuds my wife had, or I could send it to my bluetooth headphones, but not both.
Pulseaudio to the rescue! With a bit of command-line fu (no GUI for this, but that’s totally do-able), I created a virtual audio device, using Pulseaudio’s “combine” module. Like the name suggests, it combines multiple other audio devices into a single one. It can do more complex combinations (such as sending some channels hither and others thither), but I just needed a straight mirroring of the devices. In a terminal, I ran:
pactl load-module module-combine sink_name=shared_play adjust_time=3 slaves=”alsa_output.pci-0000_00_1b.0.analog-stereo,bluez_sink.00_15_0F_72_70_E1″
Hey presto! Now there’s a third audio device available in the Sound Preferences to send the sound to, and it comes out both the wired ear buds and my bluetooth headphones (with a very slight sync offset, but close enough for my purposes).
Also, for those interested – the names of the 2 audio devices in my pactl command line came from the output of ‘pactl list’.
This kind of seamless migration of running audio streams really isn’t possible to do without something like Pulseaudio that can manage stream routing on the fly. I’m well aware that Pulseaudio integration into the distributions has been a bumpy ride for lots of people, but I think the end goal justifies the painful process of fixing all the sound drivers. I hope you do too!
Lennart points out that the extra paprefs application has a “Add virtual output device for simultaneous output on all local sound cards” check-box that does the same thing as loading the combine module manual, but also handles hot-plugging of devices as they appear and disappear.
Actually, the credit for the entire talk goes to Vincent and friends, who gave the GNOME 3.0 overview during the GUADEC opening at GCDS and to Owen for his GNOME Shell talk. I stole content from their slides shamelessly.
My intention with the talk was to present some of the history and development of the GStreamer project as a means to look at where we might go next. I talked briefly about the origins of the project, its growth, and some of my personal highlights from the work we’ve done in the last year. To prepare the talk, I extracted some simple statistics from our commit history. In those, it’s easy to see both the general growth of the project, in terms of development energy/speed, as well as the increase in the number of contributors. It’s also possible to see the large hike in productivity that switching to Git in January has provided us.
The second part of the talk was discussing some of the pros and cons around considering whether to embark on a new major GStreamer release cycle leading up to a 1.0 release. We’ve successfully maintained the 0.10 GStreamer release series with backwards-compatible ABI and API (with some minor glitches) for 3.5 years now, and been very successful at adding features and improving the framework while doing so.
After 3.5 years of stable development, it’s clear to me that when we made GStreamer 0.10, it really ought to have been 1.0. Nevertheless, there are some parts of GStreamer 0.10 that we’re collectively not entirely happy with and would like to fix, but can’t without breaking backwards compatibility – so I think that even if we had made 0.10 at that point, I’d want to be doing 1.2 by now.
Some examples of things that are hard to do in 0.10:
- Replace ugly or hard to use API
- ABI mistakes such as structure members that should be private having been accidentally exposed in some release.
- Running out of padding members in public structures, preventing further expansion
- Deprecated API (and associated dead code paths) we’d like to remove
There are also some enhancements that fall into a more marginal category, in that they are technically possible to achieve in incremental steps during the 0.10 cycle, but are made more difficult by the need to preserve backwards compatibility. These include things like adding per-buffer metadata to buffers (for extensible timestamping/timecode information, pan & scan regions and others), variable strides in video buffers and creating/using more base classes for common element types.
In the cons category are considerations like the obvious migration pain that breaking ABI will cause our applications, and the opportunity cost of starting a new development cycle. The migration cost is mitigated somewhat by the ability to have parallel installations of GStreamer. GStreamer 0.10 applications will be able to coexist with GStreamer 1.0 applications.
The opportunity cost is a bit harder to ignore. When making the 0.9 development series, we found that the existing 0.8 branch became essentially unmaintained for 1.5 years, which is a phenomenon we’d all like to avoid with a new release series. I think that’s possible to achieve this time around, because I expect a much smaller scope of change between 0.10 and 1.0. Apart from the few exceptions above, GStreamer 0.10 has turned out really well, and has become a great framework being used in all sorts of exciting ways that doesn’t need large changes.
Weighing up the pros and cons, it’s my opinion that it’s worth making GStreamer 1.0. With that in mind, I made the following proposal at the end of my talk:
- We should create a shared Git playground and invite people to use it for experimental API/ABI branches
- Merge from the 0.10 master regularly into the playground regularly, and rebase/fix experimental branches
- Keep developing most things in 0.10, relying on the regular merges to get them into the playground
- After accumulating enough interesting features, pull the experimental branches together as a 0.11 branch and make some released
- Target GStreamer 1.0 to come out in time for GNOME 3.0 in March 2010
This approach wasn’t really possible the last time around when everything was stored in CVS – it’s having a fast revision control system with easy merging and branch management that will allow it.
On Thursday, we’re having a GStreamer summit in one of the rooms at the university. We’ll be discussing my proposal above, as well as talking about some of the problems people have with 0.10, and what they’d like to see in 1.0. If we can, I’d like to draw up a list of features and changes that define GStreamer 1.0 that we can start working towards.
Please come along if you’d like to help us push GStreamer forward to the next level. You’ll need to turn up at the university GCDS venue and then figure out on your own which room we’re in. We’ve been told there is one organised, but not where – so we’ll all be in the same boat.
The summit starts at 11am.
We’re leaving tomorrow afternoon for 11 days holiday in New York and Washington D.C. While we’re there, I’m hoping to catch up with Luis and Krissa and Thom May. It’s our first trip to either city, so we’re really excited – there’s a lot of fun, unique stuff to do in both places and we’re looking forward to trying to do all of it in our short visit.
On the GStreamer front, I just pushed a bunch of commits I’ve been working on for the past few weeks upstream into Totem, gst-plugins-base and gst-plugins-bad. Between them they fix a few DVD issues like multiangle support and playback in playbin2. The biggest visible feature though is the API that allowed me to (finally!) hook up the DVD menu items in Totem’s UI. Now the various ‘DVD menu’, ‘Title Menu’ etc menu items work, as well as switching angles in multiangle titles, and it provides the nice little ‘cursor switches to a hand when over a clickable button’ behaviour.
I actually had it all ready yesterday, but people told me April 1 was the wrong day to announce any big improvements in totem-gstreamer DVD support
If you’re in Dublin tomorrow (Saturday 28th March), and you’re interested in Open Source, feel free to come along to the OSSbarcamp at DIT Kevin St and enjoy some of the (completely free!) talks and demos. I’ll be presenting an introduction to the GStreamer multimedia framework in the afternoon.
Details of times and the talk schedule are at http://www.ossbarcamp.com/
I’d like to offer congratulations to all the Xiph folks, but especially Monty, Ralph and Tim on the theora-1.1alpha1 release from the Thusnelda encoder branch.
To try it out, I transcoded a short (1m40s) 720p trailer video from H.264+AAC to Theora+Vorbis, at libtheora quality setting of 32, using the 1.0 theora encoder, and again with 1.1. It’s not a very rigorous experiment, but enlightening nonetheless.
Both encoders produced output of comparable visual quality. With the 1.0 theora encoder the output file was 24383125 bytes. Of that, the video portion is about 1.835 Mbit/s. With libtheora 1.1alpha1, the same quality is reached with only 19088009 bytes (video at 1.416 Mbit/s)!
That’s a pretty easy 20-ish% compression improvement for no loss in quality. As an added bonus, the transcoding time dropped from 2m10s to 1m52s. That’s not quite real-time for this frame size in either case, but an impressive step closer.
There’s some degradation from the visual quality of the original file. That’s to be expected when taking a 5.61 Mbit/s video stream down to less than a third of the original size – even on the same codec.
For comparison, here’s a random frame from the output of each encoder, along with the same frame from the original file:
Sweeeet. Well done, guys!
Nearly 4 months after the fact, my birthday is finally complete – my “Netherlands and Architecture” “Open Source” coin finally arrived from the Royal Dutch Mint:
The delay was caused by transmission errors introduced somewhere while communicating our delivery address.
Obligatory GStreamer bit
The other nice thing from today is this script Luis and I put together to convert any supported video into a format suitable for playback on his new BlackBerry Storm after he had trouble with encoding errors trying to use FFmpeg for the task.
It’s a simple shell script that uses GStreamer’s gst-launch utility to do 2 pass conversion to H.264 and AAC in an MPEG-4 container. You can find it here if you’re interested.
As an added bonus, Luis reports that the GStreamer conversion is noticeably faster than the erroneous FFmpeg one.