E-AC-3 Spectral Extension

So, I've moved on from AHT now, and it's on to Spectral Extension (SPX).  I got the full syntax working yesterday, now I just need to figure out how to calculate all the parameters.  I have a feeling this will help quality quite a bit, especially when used in conjunction with variable bandwidth/coupling.  My vision for automatic bandwidth adjustment is starting to come together.

SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement.  Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors.  The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream).  For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors.  Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.

Once SPX encoding is working properly, I'll revisit variable bandwidth.  However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate.  This could also be used in a VBR mode with constrained bitrate limits.

If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx

E-AC-3 Adaptive Hybrid Transform

I finally got the complete AHT syntax working properly.  Unfortunately, the quality seems to be lower at all bitrates than with the normal AC-3 quantization.  I'm hoping that I just need to pick better gain values, but I have a suspicion that some of the difference is related to vector quantization, which the encoder has no control over (a basic 6-dimensional VQ minimum distance search is the best it can do).

My first step is to find out for sure if choosing better gain values will help.  One problem is that the bit allocation model is saying we need X number of bits for each mantissa.  Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions.  So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations.  Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT).  This is probably the point where I need a more accurate bit allocation model...

edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range.  They are nearly identical.  So the difference is clearly in VQ.  Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.

edit2: dct+VQ is nearly always worse than linear quantization...  I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off.  I don't know what to do at this point... *sigh*

note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.

edit3: It turns out that the majority of the quality difference was in the 6-point DCT.  If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better.  I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation...  If not then I'm at another dead-end.

edit4: I'm giving up on AHT for now.  The DCT is definitely correct and is very certainly causing the quality decrease.  If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this.  Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.