For the past few weeks I've been working on a new project sponsored by FFMTech. The entire project involves reworking much of the existing audio framework in libavcodec.
Part 1 is changing the audio decoding API to match the video decoding API. Currently the audio decoders take packet data from an AVPacket and decode it directly to a sample buffer supplied by the user. The video decoders take packet data from an AVPacket and decode it to an AVFrame structure with a buffer allocated by AVCodecContext.get_buffer(). My project will include modifying the audio decoding API to decode audio from an AVPacket to an AVFrame, as is done with video.
AVCODEC_MAX_AUDIO_FRAME_SIZE puts an arbitrary limit on the amount of audio data returned by the decoder. For example, each FLAC frame can hold up to 65536 samples for 8 channels at 32-bit sample depth, which is 2097152 bytes of raw audio, but AVCODEC_MAX_AUDIO_FRAME_SIZE is only 192000. Using get/release_buffer() for audio decoding will solve this problem. It will, however, require changes to every audio decoder. Most of those changes are trivial since the frame size is known prior to decoding the frame or is easily parsed. Some of the changes are more intrusive due to having to determine the frame size prior to allocating and writing to the output buffer.
As part of the preparation for the new API, I have been cleaning up all the audio decoder, which has been quite tedious. I've found some pretty surprising bugs along the way. I'm getting close to finishing that part so I'll be able to move on to implementing the new API in each decoder.
Thursday, October 27, 2011
Wednesday, July 27, 2011
E-AC-3 Spectral Extension
So, I've moved on from AHT now, and it's on to Spectral Extension (SPX). I got the full syntax working yesterday, now I just need to figure out how to calculate all the parameters. I have a feeling this will help quality quite a bit, especially when used in conjunction with variable bandwidth/coupling. My vision for automatic bandwidth adjustment is starting to come together.
SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement. Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors. The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream). For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors. Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.
Once SPX encoding is working properly, I'll revisit variable bandwidth. However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate. This could also be used in a VBR mode with constrained bitrate limits.
If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx
SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement. Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors. The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream). For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors. Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.
Once SPX encoding is working properly, I'll revisit variable bandwidth. However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate. This could also be used in a VBR mode with constrained bitrate limits.
If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx
Thursday, July 21, 2011
E-AC-3 Adaptive Hybrid Transform
I finally got the complete AHT syntax working properly. Unfortunately, the quality seems to be lower at all bitrates than with the normal AC-3 quantization. I'm hoping that I just need to pick better gain values, but I have a suspicion that some of the difference is related to vector quantization, which the encoder has no control over (a basic 6-dimensional VQ minimum distance search is the best it can do).
My first step is to find out for sure if choosing better gain values will help. One problem is that the bit allocation model is saying we need X number of bits for each mantissa. Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions. So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations. Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT). This is probably the point where I need a more accurate bit allocation model...
edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range. They are nearly identical. So the difference is clearly in VQ. Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.
edit2: dct+VQ is nearly always worse than linear quantization... I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off. I don't know what to do at this point... *sigh*
note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.
edit3: It turns out that the majority of the quality difference was in the 6-point DCT. If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better. I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation... If not then I'm at another dead-end.
edit4: I'm giving up on AHT for now. The DCT is definitely correct and is very certainly causing the quality decrease. If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this. Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.
My first step is to find out for sure if choosing better gain values will help. One problem is that the bit allocation model is saying we need X number of bits for each mantissa. Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions. So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations. Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT). This is probably the point where I need a more accurate bit allocation model...
edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range. They are nearly identical. So the difference is clearly in VQ. Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.
edit2: dct+VQ is nearly always worse than linear quantization... I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off. I don't know what to do at this point... *sigh*
note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.
edit3: It turns out that the majority of the quality difference was in the 6-point DCT. If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better. I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation... If not then I'm at another dead-end.
edit4: I'm giving up on AHT for now. The DCT is definitely correct and is very certainly causing the quality decrease. If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this. Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.
Friday, June 17, 2011
E-AC-3 encoder
Well, I finally got a working E-AC-3 encoder committed to Libav. The bitstream format does save a few bits here and there, but the overall quality difference is minimal. However, it will be the starting point for adding more E-AC-3 features that will improve quality.
The first feature I completed was support for higher bit rates. This is done in E-AC-3 by using fewer blocks per frame. A normal AC-3 frame has 6 blocks of 256 samples each, but E-AC-3 can reduce that to 1, 2, or 3 blocks. This way a small range can be used for the per-frame bit rate, but it still allow for increasing the per-second bit rate. For example, 5.1-channel E-AC-3 content on HD-DVDs was typically encoded at 1536 kbps using 1 block per frame.
Currently I am working on implementing AHT (adaptive hybrid transform). The AHT process uses a 6-point DCT on each coefficient across the 6 blocks in the frame. It basically uses the normal AC-3 bit allocation process to determine quantization of each DCT-processed "pre-mantissa" but it uses a finer resolution for quantization and different quantization methods. I have the 6-point DCT working and one of the two quantization methods. Now I just need to finish the other quantization method and implement mantissa bit counting and bitstream output.
The first feature I completed was support for higher bit rates. This is done in E-AC-3 by using fewer blocks per frame. A normal AC-3 frame has 6 blocks of 256 samples each, but E-AC-3 can reduce that to 1, 2, or 3 blocks. This way a small range can be used for the per-frame bit rate, but it still allow for increasing the per-second bit rate. For example, 5.1-channel E-AC-3 content on HD-DVDs was typically encoded at 1536 kbps using 1 block per frame.
Currently I am working on implementing AHT (adaptive hybrid transform). The AHT process uses a 6-point DCT on each coefficient across the 6 blocks in the frame. It basically uses the normal AC-3 bit allocation process to determine quantization of each DCT-processed "pre-mantissa" but it uses a finer resolution for quantization and different quantization methods. I have the 6-point DCT working and one of the two quantization methods. Now I just need to finish the other quantization method and implement mantissa bit counting and bitstream output.
Friday, May 6, 2011
AC-3 Variable Bandwidth
Originally, the FFmpeg/Libav AC-3 encoder used a fixed bandwidth cutoff of 21kHz for 48kHz sample rate content. For low-to-moderate bitrates (80kbps to 160kbps, stereo) this was way too high and ended up with audible, annoying high frequency artifacts and dull/tinny sounding cymbals. It also made the lower frequencies sound worse at the lower settings. As a remedy for this, I changed the default bandwidth setting to adapt to the bitrate and number of channels. The values I chose matched content produced by professional AC-3 encoders.
This fixed the issue of high-frequency artifacts, but the downside was that some content that would sound much better with the higher frequencies was getting cutoff much lower than necessary. The way I tried to solve this problem previously in Aften was to implement a variable bandwidth mode that would adjust the bandwidth per-frame to adapt to changing content. The idea was ok, but the result ended up sounding weird in some cases because of large fluctuations in bandwidth in a short period of time. It turns out that the human ear is quite sensitive to changes in bandwidth. If you start out at a low bandwidth, your ears don't know what they're missing, but if you go from high bandwidth to low bandwidth it is very noticeable.
For the Libav AC-3 encoder, I decided to revisit the Aften variable bandwidth (VBW) encoding mode to see if I could improve it. What I wanted to do was to somehow smooth out the transitions so that the changes are not as noticeable. The solution I ended up with works pretty well. I started by defining the minimum bandwidth as the existing default bandwidth and then creating a new table for maximum bandwidth. For high and very low bitrates the max and min are the same, which disables the VBW mode. For low-to-moderate bitrates the VBW mode selects a value between min and max that will result in a decent quality (fixed SNR offset of -1dB). The rate of change between frames is moderated using an exponential moving average with a half-life of 2.2 seconds. This allows for decent transition speed without being too volatile.
The VBW mode is the first step toward adaptive channel coupling. The goal will be to adaptively turn coupling on/off and to adapt the coupling start/end bands based on content.
This fixed the issue of high-frequency artifacts, but the downside was that some content that would sound much better with the higher frequencies was getting cutoff much lower than necessary. The way I tried to solve this problem previously in Aften was to implement a variable bandwidth mode that would adjust the bandwidth per-frame to adapt to changing content. The idea was ok, but the result ended up sounding weird in some cases because of large fluctuations in bandwidth in a short period of time. It turns out that the human ear is quite sensitive to changes in bandwidth. If you start out at a low bandwidth, your ears don't know what they're missing, but if you go from high bandwidth to low bandwidth it is very noticeable.
For the Libav AC-3 encoder, I decided to revisit the Aften variable bandwidth (VBW) encoding mode to see if I could improve it. What I wanted to do was to somehow smooth out the transitions so that the changes are not as noticeable. The solution I ended up with works pretty well. I started by defining the minimum bandwidth as the existing default bandwidth and then creating a new table for maximum bandwidth. For high and very low bitrates the max and min are the same, which disables the VBW mode. For low-to-moderate bitrates the VBW mode selects a value between min and max that will result in a decent quality (fixed SNR offset of -1dB). The rate of change between frames is moderated using an exponential moving average with a half-life of 2.2 seconds. This allows for decent transition speed without being too volatile.
The VBW mode is the first step toward adaptive channel coupling. The goal will be to adaptively turn coupling on/off and to adapt the coupling start/end bands based on content.
Monday, January 10, 2011
AC-3 exponent strategy
I'm having some issues with coming up with an ideal exponent strategy decision algorithm for the FFmpeg AC-3 encoder. I thought I would post some of my results so far...
Problem 1: The current algorithm is far too biased toward frequency resolution over time resolution.
Problem 2: Adjusting the exp diff threshold solves some of those problems but gives slightly worse results at high bitrates. These are likely not a problem since they're in the "inaudible differences" range for PEAQ ODG, but it's still interesting.
Problem 3: Regardless of exp diff threshold, the algorithm uses too many bits for the exponents for lower bitrates.
For some reason Aften's exponent strategy decision, which I have ported to FFmpeg, is not as good as I expected. Limiting the search to low bitrate strategy sets did allow me to compare bit usage to the FFmpeg algorithm, but overall it is not the best choice. I'm trying to come up with either a hybrid solution or modify the FFmpeg algorithm to make better choices.
Ideally I can dynamically adjust the bit usage (and/or the threshold) in the FFmpeg algorithm based on the content of the current frame in order to have a good balance for most input samples.
Update:
Ideas so far based on lots and lots of testing:
1) use current algorithm that determines new exponents based on sad threshold
2) lower the threshold from 1000 to 500
3) dynamically adjust bit usage level based on a pre-processing bit allocation run. vbr would probably be fastest. run bit allocation with original exponents at a good target snr offset. determine how many bits are left for exponents and adjust exp strategy accordingly. (if final bit allocation is also vbr, adjust based on maximum frame size)
Update2:
The complex solution is not necessarily the best solution...
Now my plan is:
1) lower EXP_DIFF_THRESHOLD to a good value after testing with a variety of samples
2) implement adaptive constant bandwidth depending on bitrate
3) implement VBR bit allocation
4) revisit exponent strategy decision, possibly just using 1 additional algorithm w/ significantly lower bit usage
Problem 1: The current algorithm is far too biased toward frequency resolution over time resolution.
Problem 2: Adjusting the exp diff threshold solves some of those problems but gives slightly worse results at high bitrates. These are likely not a problem since they're in the "inaudible differences" range for PEAQ ODG, but it's still interesting.
Problem 3: Regardless of exp diff threshold, the algorithm uses too many bits for the exponents for lower bitrates.
For some reason Aften's exponent strategy decision, which I have ported to FFmpeg, is not as good as I expected. Limiting the search to low bitrate strategy sets did allow me to compare bit usage to the FFmpeg algorithm, but overall it is not the best choice. I'm trying to come up with either a hybrid solution or modify the FFmpeg algorithm to make better choices.
Ideally I can dynamically adjust the bit usage (and/or the threshold) in the FFmpeg algorithm based on the content of the current frame in order to have a good balance for most input samples.
Update:
Ideas so far based on lots and lots of testing:
1) use current algorithm that determines new exponents based on sad threshold
2) lower the threshold from 1000 to 500
3) dynamically adjust bit usage level based on a pre-processing bit allocation run. vbr would probably be fastest. run bit allocation with original exponents at a good target snr offset. determine how many bits are left for exponents and adjust exp strategy accordingly. (if final bit allocation is also vbr, adjust based on maximum frame size)
Update2:
The complex solution is not necessarily the best solution...
Now my plan is:
1) lower EXP_DIFF_THRESHOLD to a good value after testing with a variety of samples
2) implement adaptive constant bandwidth depending on bitrate
3) implement VBR bit allocation
4) revisit exponent strategy decision, possibly just using 1 additional algorithm w/ significantly lower bit usage
Subscribe to:
Posts (Atom)