Notes on Speex

I've been working on improving the Speex support in FFmpeg. I pretty much have everything completed except for getting it all through review to SVN.

I started out just making an encoder for libavcodec based on libspeex, along with adding support for Speex in the Ogg muxer. This was fairly simple. Art Clarke of Xuggler also submitted similar patches around the same time, so there was some good discussion as well as some shared code.

The Ogg muxer is obviously needed first, so I tested it using stream copy from Ogg-to-Ogg and FLV-to-Ogg. In both cases I ran into problems with a small pop near the beginning of the output file when decoded using speexdec. After about a full day of debugging, I managed to figure out the problem. The first frame is supposed to have skipped samples due to transmission delay. The way this is communicated is through Ogg granule positions. However, the first granule position reflects the last sample of the last frame in the first Ogg page. In order to determine how much of the first frame to skip, one must calculate what that granule position would normally be without skipping, then calculate the difference. I submitted a patch to ffmpeg-devel to handle this in the Ogg demuxer. As a side note, it seems that Adobe Flash Media Server does not take this into account, and therefore the timestamps skip samples in the middle of the stream instead of at the start. So stream copy of those files from FLV-to-Ogg will always produce incorrect granule positions.

The libspeex decoder should also be modified to skip these samples in the first frame. I have a patch ready for that. Cutting off samples from the last frame is not possible though because you never know if you're encoding the last frame. That's not as big a deal though.

Speex-in-FLV muxing support is a pretty simple and straightforward patch, but should not be committed until the Ogg timestamp issues are fixed to prevent more broken files in the wild. The same goes for the Speex-in-Ogg muxing.

Today I have been cleaning up the libspeex encoder for lavc. Speex is weird in that it can use either a quality setting or bitrate to set the CBR rate. My proposal will be to add a flag2 for explicitly enabling VBR. That way it will not just depend on CODEC_FLAG_QSCALE like in most other audio encoders. For setting frames-per-packet, the user can set AVCodecContext.frame_size to a multiple of the Speex frame size (e.g. 960 in wideband mode would be 3 frames-per-packet). Complexity is set using AVCodecContext.compression_level.

After I get all this through review (hopefully it won't take terribly long) I think I'll try adding support for vorbiscomment muxing in lavf.

wow, great quote

"The first question I ask myself when something doesn't seem to be beautiful is why do I think it's not beautiful. And very shortly you discover that there is no reason." -- John Cage

Remember the Milk

I stumbled across a pretty neat TODO list website called Remember the Milk. I started to use it for work, but work soon became too crazy to have time to keep up with a TODO list in addition to doing the things on it. But I do like it for keeping up with my personal long-term and short-term programming tasks. I've made my list public so anyone can see what I'm working on.

more fail-safe FLAC parser

I submitted my FLAC parser (as mentioned in my last post) to the ffmpeg-devel mailing list and got a suggestion (a.k.a. requirement) from Michael for decreasing the likelihood of false-positive frame detection. Basically it would analyze a sequence of potential frame headers and pick the most likely sub-sequence based on concurrent non-overlapping CRC matches. It would also take header parameter changes into account. I am currently working on an implementation of this idea.

more FLAC parsing

As noted in my previous post, I was able to make a working FLAC parser, but I was not completely happy with it. I just had a feeling if I kept trying I could get it to work without buffering max_frame_size bytes and use a state variable instead.

Well, after many hours I finally got it working! At least it works for all samples I've tried so far. I still want to do more tests just to be sure though. Here is how it works:

• The FLACParseContext has a 31-byte state buffer and state size.
• I had to modify ff_combine_frame() to pad the parser buffer with 16 bytes instead of 8 bytes to accommodate negative values for start of frame less than -8.
  

1. If state buffer has data, copy data from current buffer to fill state buffer (or as much as possible). Search the first 16 bytes of the state buffer for start of frame and CRC of previous frame. If it passes, return start position within the buffer (will be negative).
   
2. Search the current buffer, up to 16 bytes before the end, for start of frame and CRC of previous frame. If the next frame is found, return the start position with the buffer. If none is found, check last 16 bytes for a frame sync word and if found, store in the state buffer. Return END_NOT_FOUND.
3. There are 2 exceptions to the CRC check. One is if an inline header is found since it will not have a CRC match. The other is if there are 10 CRC mismatches in a single frame. The reason for that is to abort the frame search for damaged files. It is extremely unlikely that a valid frame will have 10 CRC mismatches.
   

passing off work

I've been wanting to take some time to write an MPEG-4 ALS decoder for a while now, but haven't had the motivation. So this year I decided to propose the project for GSoC '09. Hopefully a good student will qualify and want to do the project.

FFmpeg Summer Of Code 2009 - MultimediaWiki

After writing a raw FLAC parser, then looking at the raw ALS format, I've decided to aggresively push for not touching raw .als files with a 10-foot pole. We just need to stick to MP4 first, and other containers like NUT and MKV second.

FLAC parsing

Let me start by saying that I very much dislike the raw FLAC format! After a lot of failed attempts I finally got a parser working. Part of the frustration was due to FLAC, and the other part was the way FFmpeg's parsing works.

First issue: reliable frame detection

• Using only the 16-bit frame header gives many many false positive frame start matches.
• Using the frame header + frame CRC-16 gives a handful of false positives.
• The only reliable way seems to be a full frame header validation including the header CRC-8, along with checking the CRC-16 for the whole frame.

Second issue: using FFmpeg's parser to combine pieces of frames from chunks of data

• Frame headers are variable-sized from 6 bytes to 16 bytes.
• The whole frame can be as small as 11 bytes, and silent frames are often less than 16 bytes.
  
• ff_combine_frame() only keeps up to 8 bytes of state information

So, the way I finally ended up with a working parser was to basically use the same strategy as the buffering that's done currently in the FLAC decoder, but do it with ff_combine_frame().

1. read first header to get starting point for block size/channels/bps
2. estimate maximum frame size
3. buffer enough data for maximum frame
4. detect next header
5. update maximum frame size with header info
   
6. update buffer
   
7. return frame start and size
8. goto 3

There are special cases for when there is no more data to read and for checking up to 11 bytes from the end of the buffer to make sure there isn't a small frame that's not being detected due to the 16-byte state size.

TODO list

My TODO list is getting long, so I'm writing it down here. I'll keep updating this post if I decide to add/remove items or when I complete an item.

1. finish simplifying Aften's asm structure
2. FFmpeg AC3 decoder channel reordering (after channel order patch is applied)
   
3. add FLAC 24-bit encoding to Flake and FFmpeg
4. finish Alsophila bit allocation improvement and port to Aften
5. port Alsophila frame encoding structure to Aften
   
6. add a trellis mode for variable frame size in Flake
7. FFmpeg FLAC parser
   
8. better FFmpeg raw FLAC muxer
9. better FFmpeg raw FLAC demuxer
10. MP4ALS decoder for FFmpeg
11. clean up MP4ALS encoder for FFmpeg
12. AC3 metadata editor
    

ALAC update

Jai's ALAC encoder for FFmpeg SoC is going very well. Entropy coding and prediction are working. Adaptive prediction order and stereo decorrelation have been implemented, but are still works-in-progress. There is still an odd bug which generates non-lossless output when decoded. It's very rare from what I've found, and hopefully the cause will be tracked-down and fixed.

Even with the encoder as it is right now, it could be very easily cleaned-up for inclusion in FFmpeg SVN. The compression isn't great, but a decent working ALAC encoder is better than no encoder, and it's definitely on the right path to being a great encoder.

MPEG-4 ALS encoder

Last weekend, I started working on an mp4als encoder for FFmpeg. Here is a summary of what I have implemented so far.

• rice coding (slightly different from jpeg-ls)
• random access (progressive prediction order)
• lpc prediction with fixed order
• joint stereo
• entropy block partitioning
• constant and zero block modes

I have quite a bit left to do though for a complete encoder. What I have so far gets comparable compression to the RM20 reference encoder with default settings, and is 3x faster.

Here is a patch against FFmpeg SVN.