AC-3 Variable Bandwidth

Originally, the FFmpeg/Libav AC-3 encoder used a fixed bandwidth cutoff of 21kHz for 48kHz sample rate content.  For low-to-moderate bitrates (80kbps to 160kbps, stereo) this was way too high and ended up with audible, annoying high frequency artifacts and dull/tinny sounding cymbals.  It also made the lower frequencies sound worse at the lower settings.  As a remedy for this, I changed the default bandwidth setting to adapt to the bitrate and number of channels.  The values I chose matched content produced by professional AC-3 encoders.

This fixed the issue of high-frequency artifacts, but the downside was that some content that would sound much better with the higher frequencies was getting cutoff much lower than necessary.  The way I tried to solve this problem previously in Aften was to implement a variable bandwidth mode that would adjust the bandwidth per-frame to adapt to changing content.  The idea was ok, but the result ended up sounding weird in some cases because of large fluctuations in bandwidth in a short period of time.  It turns out that the human ear is quite sensitive to changes in bandwidth.  If you start out at a low bandwidth, your ears don't know what they're missing, but if you go from high bandwidth to low bandwidth it is very noticeable.

For the Libav AC-3 encoder, I decided to revisit the Aften variable bandwidth (VBW) encoding mode to see if I could improve it.  What I wanted to do was to somehow smooth out the transitions so that the changes are not as noticeable.  The solution I ended up with works pretty well.  I started by defining the minimum bandwidth as the existing default bandwidth and then creating a new table for maximum bandwidth.  For high and very low bitrates the max and min are the same, which disables the VBW mode.  For low-to-moderate bitrates the VBW mode selects a value between min and max that will result in a decent quality (fixed SNR offset of -1dB).  The rate of change between frames is moderated using an exponential moving average with a half-life of 2.2 seconds.  This allows for decent transition speed without being too volatile.

The VBW mode is the first step toward adaptive channel coupling.  The goal will be to adaptively turn coupling on/off and to adapt the coupling start/end bands based on content.

AC-3 exponent strategy

I'm having some issues with coming up with an ideal exponent strategy decision algorithm for the FFmpeg AC-3 encoder. I thought I would post some of my results so far...

Problem 1: The current algorithm is far too biased toward frequency resolution over time resolution.

Problem 2: Adjusting the exp diff threshold solves some of those problems but gives slightly worse results at high bitrates. These are likely not a problem since they're in the "inaudible differences" range for PEAQ ODG, but it's still interesting.

Problem 3: Regardless of exp diff threshold, the algorithm uses too many bits for the exponents for lower bitrates.

For some reason Aften's exponent strategy decision, which I have ported to FFmpeg, is not as good as I expected. Limiting the search to low bitrate strategy sets did allow me to compare bit usage to the FFmpeg algorithm, but overall it is not the best choice. I'm trying to come up with either a hybrid solution or modify the FFmpeg algorithm to make better choices.

Ideally I can dynamically adjust the bit usage (and/or the threshold) in the FFmpeg algorithm based on the content of the current frame in order to have a good balance for most input samples.

Update:
Ideas so far based on lots and lots of testing:
1) use current algorithm that determines new exponents based on sad threshold
2) lower the threshold from 1000 to 500
3) dynamically adjust bit usage level based on a pre-processing bit allocation run. vbr would probably be fastest. run bit allocation with original exponents at a good target snr offset. determine how many bits are left for exponents and adjust exp strategy accordingly. (if final bit allocation is also vbr, adjust based on maximum frame size)

Update2:
The complex solution is not necessarily the best solution...
Now my plan is:
1) lower EXP_DIFF_THRESHOLD to a good value after testing with a variety of samples
2) implement adaptive constant bandwidth depending on bitrate
3) implement VBR bit allocation
4) revisit exponent strategy decision, possibly just using 1 additional algorithm w/ significantly lower bit usage

Ideas for AC-3 Bit Allocation

First I guess I should explain the AC-3 basic bit allocation process.

1) run window+MDCT on input samples
2) extract exponents (range 0-24) from MDCT coefficients
3) compress exponents for efficient encoding (part of the compression is not optional)
4) calculate approximate PSD for each frequency bin from compressed exponents
5) group frequency bins into critical bands and merge PSD for each band
6) spreading function + ATH comparison to calculate masking curve
7) allocate bits for each bin based on the PSD, masking curve, and an SNR offset value

Stages 6 and 7 use parameters from a pre-defined list for the spreading function and bit allocation. The SNR offset parameter in stage 7 is chosen by the encoder so that the number of allocated bits will fit in the frame.

I have several ideas to try to improve the accuracy of the bit allocation.

1) Adjust fast gain value in the spreading function based on the exponent strategy. This is a very rough estimate of noisiness, so it could lead to a more accurate masking curve.

2) If I decide to port VBR mode, try using the original exponents to calculate the masking curve and bap based on the target quality. Then when using the compressed exponents, select a final SNR offset that ensures that each bin gets at least as many bits as determined by the more accurate calculation.

3) Same as #2, but use in CBR mode and use delta bit allocation to adjust the masking curve to more closely match the one that used uncompressed exponents.

4) Same as #2, but use in CBR mode and vary the bandwidth first if needed in order to allocate more bits to bins that need them based on the more accurate masking curve.

5) Same as #2, but calculate a more accurate PSD based on original coefficients rather than exponents. Would probably have to modify low_comp1() in ac3.c to use a threshold comparison instead of an equivalency.

6) Run a separate psychoacoustic model using the original input samples to determine better parameters and masking curve. Adjust parameters to better match the more accurate masking curve. Use delta bit allocation if necessary.

7) Same as #2, but adjust SNR offsets per-block and per-channel to possibly achieve desired quality with fewer bits. The cost of re-transmitting SNR offsets in each block would need to be taken into consideration.

FFmpeg AC-3 encoder

About a month ago, I started a project to improve the FFmpeg AC-3 encoder. The first part consists of porting improvements from Aften to FFmpeg. The second part consists of further improving the encoder by adding some features that are not present in Aften. The third part is adding E-AC-3 encoding support.

So far I have done quite a few speed and structural improvements. The metadata and floating-point support have been ported, but are still going through review on ffmpeg-devel. Once those are approved, there are only a few things left to port from Aften.

I'm excited to get to work on this. I keep coming up with new ideas that I want to implement. I just have to stay focused and keep working to get the changes to the main FFmpeg repo. I'll try to post more details on this blog for the various random ideas I have.

ALS encoder

For the last several months, Thilo and I have been working on an ALS encoder. I'm quite happy with the results so far. It is nearly finished, minus a few optional features. It is generally much faster than the ISO reference encoder (twice as fast in default mode) and usually gets slightly better compression. Once we get the encoder to the point where we have all of the features implemented that we want and as many bugs worked out as we can, we will do massive amounts of benchmarking. I'll try to post any interesting results here.

Notes on Speex

I've been working on improving the Speex support in FFmpeg. I pretty much have everything completed except for getting it all through review to SVN.

I started out just making an encoder for libavcodec based on libspeex, along with adding support for Speex in the Ogg muxer. This was fairly simple. Art Clarke of Xuggler also submitted similar patches around the same time, so there was some good discussion as well as some shared code.

The Ogg muxer is obviously needed first, so I tested it using stream copy from Ogg-to-Ogg and FLV-to-Ogg. In both cases I ran into problems with a small pop near the beginning of the output file when decoded using speexdec. After about a full day of debugging, I managed to figure out the problem. The first frame is supposed to have skipped samples due to transmission delay. The way this is communicated is through Ogg granule positions. However, the first granule position reflects the last sample of the last frame in the first Ogg page. In order to determine how much of the first frame to skip, one must calculate what that granule position would normally be without skipping, then calculate the difference. I submitted a patch to ffmpeg-devel to handle this in the Ogg demuxer. As a side note, it seems that Adobe Flash Media Server does not take this into account, and therefore the timestamps skip samples in the middle of the stream instead of at the start. So stream copy of those files from FLV-to-Ogg will always produce incorrect granule positions.

The libspeex decoder should also be modified to skip these samples in the first frame. I have a patch ready for that. Cutting off samples from the last frame is not possible though because you never know if you're encoding the last frame. That's not as big a deal though.

Speex-in-FLV muxing support is a pretty simple and straightforward patch, but should not be committed until the Ogg timestamp issues are fixed to prevent more broken files in the wild. The same goes for the Speex-in-Ogg muxing.

Today I have been cleaning up the libspeex encoder for lavc. Speex is weird in that it can use either a quality setting or bitrate to set the CBR rate. My proposal will be to add a flag2 for explicitly enabling VBR. That way it will not just depend on CODEC_FLAG_QSCALE like in most other audio encoders. For setting frames-per-packet, the user can set AVCodecContext.frame_size to a multiple of the Speex frame size (e.g. 960 in wideband mode would be 3 frames-per-packet). Complexity is set using AVCodecContext.compression_level.

After I get all this through review (hopefully it won't take terribly long) I think I'll try adding support for vorbiscomment muxing in lavf.

wow, great quote

"The first question I ask myself when something doesn't seem to be beautiful is why do I think it's not beautiful. And very shortly you discover that there is no reason." -- John Cage

Remember the Milk

I stumbled across a pretty neat TODO list website called Remember the Milk. I started to use it for work, but work soon became too crazy to have time to keep up with a TODO list in addition to doing the things on it. But I do like it for keeping up with my personal long-term and short-term programming tasks. I've made my list public so anyone can see what I'm working on.

more fail-safe FLAC parser

I submitted my FLAC parser (as mentioned in my last post) to the ffmpeg-devel mailing list and got a suggestion (a.k.a. requirement) from Michael for decreasing the likelihood of false-positive frame detection. Basically it would analyze a sequence of potential frame headers and pick the most likely sub-sequence based on concurrent non-overlapping CRC matches. It would also take header parameter changes into account. I am currently working on an implementation of this idea.

more FLAC parsing

As noted in my previous post, I was able to make a working FLAC parser, but I was not completely happy with it. I just had a feeling if I kept trying I could get it to work without buffering max_frame_size bytes and use a state variable instead.

Well, after many hours I finally got it working! At least it works for all samples I've tried so far. I still want to do more tests just to be sure though. Here is how it works:

• The FLACParseContext has a 31-byte state buffer and state size.
• I had to modify ff_combine_frame() to pad the parser buffer with 16 bytes instead of 8 bytes to accommodate negative values for start of frame less than -8.
  

1. If state buffer has data, copy data from current buffer to fill state buffer (or as much as possible). Search the first 16 bytes of the state buffer for start of frame and CRC of previous frame. If it passes, return start position within the buffer (will be negative).
   
2. Search the current buffer, up to 16 bytes before the end, for start of frame and CRC of previous frame. If the next frame is found, return the start position with the buffer. If none is found, check last 16 bytes for a frame sync word and if found, store in the state buffer. Return END_NOT_FOUND.
3. There are 2 exceptions to the CRC check. One is if an inline header is found since it will not have a CRC match. The other is if there are 10 CRC mismatches in a single frame. The reason for that is to abort the frame search for damaged files. It is extremely unlikely that a valid frame will have 10 CRC mismatches.