Success!

Posted by stephen on Tuesday, 31st January, 2006 @ 15:11

I thought I'd best place this warning here having just come out with the load of nonsense below, this has become a long post. Not because it is full of great and detailed information but because I've padded it with an unnecessary and pleonastic narrative, for which I can only apologise. I'm approaching the end of Stephen Fry's autobiography at the moment and I fear it's started to influence my own writings.

As by now you're aware, one of my tasks for our work in Scotland is the live streaming of terrestrial TV, transcoded from it's native MPEG2 into MPEG4 (H.264).

It's something I've been rather dreading ever since I realised that hardware encoder cards are not that widely available, and even those that are won't be just plugin and go with the VLC streaming solution we currently employ.

My eventual conclusion was that we needed to be able to transcode in software. Increased processing power is actually cheaper than buying hardware encoders, and the lack of rackspace available to us means we don't really have the PCI slots to spare anyway.

So on we trundled, assuming that software transcoding in realtime was going to be possible. I think I blocked out of my mind an almost certain knowledge that it wouldn't be possible. Based on my experiences with QuickTime Pro, H.264 encoding is a long and difficult task, even for my trusty PowerMac G5.

Last night I received a phone call from Richard telling me I must come in tomorrow (now today). The guys from Scotland would be down on Wednesday with their wireless kit for some testing and we must have some MPEG4 content available and a server to stream it ready and waiting for them.

My biggest task right now is finishing off the documentation for RegenTV, a task I was told I must complete by Wednesday if we were to be paid again anytime soon. So spending all day in Stratford, not being at my most productive because of the many distractions was not an appealing prospect. Add to that the 2h30m best case scenario for driving to work and back. I say best case, it only really occurs at odd hours of the day. Last Thursday I spent 4h30m on my way to and from work. Thus is the nature of British motorways.

So I decided that I must complete the MPEG4 task last night. Not to produce a realtime transcode, but to simply have some MPEG4 content from TV and the mechanism to send it as a multicast stream across a network.

From previous experience I've discovered that QuickTime encoded H.264 files can't be encapsulated in an MPEG2 Transport Stream and multicast by VLC. So the task is to get VLC itself to do the encoding. I grabbed a line from the VLC documentation that transcodes and streams in a single pass, modified it for MPEG4 output and fed it a file ripped from my TiVo.

Segmentation Fault

Bugger. All the log messages until that point seemed to indicate everything was working perfectly. So what happened? I was using the packaged version of VLC for Ubuntu. Based on previous experience of VLC crashing I thought it best to try and compile from source instead.

After many rounds of ./configure and make and apt-get install somelibrary-dev I finally got a complete build. Then I noticed that x264, the H.264 encoder wasn't included. So off the to the VLC developers site to download the apparently very experimental code. It built first time without a hitch, great! Rebuild VLC and I'm away.

Now for the tricky bit, feed in the magic runes...

/usr/local/vlc/bin/vlc -vvv FatherTed.mpg --sout \
'#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=96,deinterlace}:
standard{access=udp,mux=ts,url=239.192.255.1}'

And success it would seem! Lots of warnings about dodgy MPEG data and timing problems, but a running process none the less. As long as there was both audio and video in the Transport Stream I'd be safe. I'd just feed it into a file instead of onto the network and the computer can have all the time in the world to do it's thing.

So I fired up VLC on the G5 and tuned to the appropriate multicast address. And something unexpected happened. There in front of me was full size, good quality video with only the occasional stutter. A quick change from multicast to unicast and I tuned in again. Perfect picture!

I'm not sure what saintly deed I'd performed in a previous existence to deserve such fortune. But that wasn't the end of my surprises that evening. I thought it best to check on the load being placed on the streaming server. Blimey! 20% CPU usage for a single stream! And the server in question is 2 years old.

I could only begin to imagine what strides in CPU performance in the past two years would allow me to achieve with a more modern system.

The next test, run another stream in parallel. So a few more runes and Kevin McCloud and one of his Grand Designs was being flung across my network eagerly seeking a viewer. So up was fired VLC on my PowerBook and sure enough, two streams from one server to two wireless clients running perfectly well, using about half the bandwidth available on my entirely not-optimised network (experience learned available bandwidth rather than theoretical available bandwidth). And a pair of VLC processes taxing barely half of their host's thinking powers.

And so an overly excited Stephen got on to the phone to his boss who doesn't answer, sent a message on Skype, only for him to disappear and to then receive a 'Message could not be sent' dialogue box, most probably sent from the bowls of hell to torment me. The final attempt, a frustrated and quickly typed text of "Good news!" was sent as I bubbled away desperate to tell someone of my exploits.

Within moments my call is returned and Richard makes the assumption that good news must mean that I've taken the test and I'm not pregnant after all, despite 24 years without a period. Apparently this is the usual good news he gets.

But no, I eagerly relay my even better news and create a much excited and happy boss. Not quite as much excited and happy as myself I suspect. But that can only be a positive thing for anyone who wishes it to be known that they have an interesting and full life.

The call is concluded and I returned to the scene of my triumph. Only to realise that I've been premature in my declaration of victory. I've not encoded to MPEG4-10 (H.264), I've encoded to the inferior MPEG4-2. Bugger. So a quick change to the magic runes and lets see if all hope is shattered...

Joy of joys, I still had Father Ted streaming in real time. Albeit a little jerkily, but ultimately watchable. A quick check on the server and the VLC process is consuming as much of my processing powers as it can lay it's little French paws on.

But that's OK. I've done enough. My processor is 2 years behind current designs which should be more adept at handling the task. We'll be most likely limited to one stream per server, or two if we're lucky enough to acquire some dual-core technology. But that's OK, that's all we intended to deliver in the first place.

A few more tweaks of the runes and I've got it down to 446Kbit of video bandwidth and still have a very watchable picture, at the very least equivalent to the 1Mbit MPEG4-2 stream.

So I think we've arrived at a useful place. If the wireless bandwidth is going to be a problem we can do a small number of streams in H.264. If there is more bandwidth available we can do perhaps three or four times as many channels per server with MPEG4-2, assuming our as yet non-existant set top boxes support such a thing.

And the best news of all. I'm at home today!