Well, I've finished the new audio decoding API, which has been merged into Libav master. The new audio encoding API is basically done, pending a (hopefully final) round of review before committing.
Next up is audio timestamp fixes/clean-up. This is a fairly undefined task. I've been collecting a list of various things that need to be fixed and ideas to try. Plus, the audio encoding API revealed quite a few bugs in some of the demuxers. Today I started a sort of TODO list for this stage of the project. I'll be editing it as the project continues to progress.
Friday, January 13, 2012
Thursday, October 27, 2011
Libav Audio Project
For the past few weeks I've been working on a new project sponsored by FFMTech. The entire project involves reworking much of the existing audio framework in libavcodec.
Part 1 is changing the audio decoding API to match the video decoding API. Currently the audio decoders take packet data from an AVPacket and decode it directly to a sample buffer supplied by the user. The video decoders take packet data from an AVPacket and decode it to an AVFrame structure with a buffer allocated by AVCodecContext.get_buffer(). My project will include modifying the audio decoding API to decode audio from an AVPacket to an AVFrame, as is done with video.
AVCODEC_MAX_AUDIO_FRAME_SIZE puts an arbitrary limit on the amount of audio data returned by the decoder. For example, each FLAC frame can hold up to 65536 samples for 8 channels at 32-bit sample depth, which is 2097152 bytes of raw audio, but AVCODEC_MAX_AUDIO_FRAME_SIZE is only 192000. Using get/release_buffer() for audio decoding will solve this problem. It will, however, require changes to every audio decoder. Most of those changes are trivial since the frame size is known prior to decoding the frame or is easily parsed. Some of the changes are more intrusive due to having to determine the frame size prior to allocating and writing to the output buffer.
As part of the preparation for the new API, I have been cleaning up all the audio decoder, which has been quite tedious. I've found some pretty surprising bugs along the way. I'm getting close to finishing that part so I'll be able to move on to implementing the new API in each decoder.
Part 1 is changing the audio decoding API to match the video decoding API. Currently the audio decoders take packet data from an AVPacket and decode it directly to a sample buffer supplied by the user. The video decoders take packet data from an AVPacket and decode it to an AVFrame structure with a buffer allocated by AVCodecContext.get_buffer(). My project will include modifying the audio decoding API to decode audio from an AVPacket to an AVFrame, as is done with video.
AVCODEC_MAX_AUDIO_FRAME_SIZE puts an arbitrary limit on the amount of audio data returned by the decoder. For example, each FLAC frame can hold up to 65536 samples for 8 channels at 32-bit sample depth, which is 2097152 bytes of raw audio, but AVCODEC_MAX_AUDIO_FRAME_SIZE is only 192000. Using get/release_buffer() for audio decoding will solve this problem. It will, however, require changes to every audio decoder. Most of those changes are trivial since the frame size is known prior to decoding the frame or is easily parsed. Some of the changes are more intrusive due to having to determine the frame size prior to allocating and writing to the output buffer.
As part of the preparation for the new API, I have been cleaning up all the audio decoder, which has been quite tedious. I've found some pretty surprising bugs along the way. I'm getting close to finishing that part so I'll be able to move on to implementing the new API in each decoder.
Wednesday, July 27, 2011
E-AC-3 Spectral Extension
So, I've moved on from AHT now, and it's on to Spectral Extension (SPX). I got the full syntax working yesterday, now I just need to figure out how to calculate all the parameters. I have a feeling this will help quality quite a bit, especially when used in conjunction with variable bandwidth/coupling. My vision for automatic bandwidth adjustment is starting to come together.
SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement. Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors. The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream). For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors. Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.
Once SPX encoding is working properly, I'll revisit variable bandwidth. However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate. This could also be used in a VBR mode with constrained bitrate limits.
If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx
SPX encoding/decoding is fairly straightforward, so I expect this won't take too long to implement. Similar to channel coupling, the encoder writes coarsely banded scale factors for frequencies above the fully-encoded bandwidth, along with noise blending factors. The decoder copies lower frequency coefficients to the upper bands, multiplies them by the scale factors, and blends them with noise (which has been scaled according to the band energy and the blending factors in the bitstream). For the encoder, I just need to make the reconstructed coefficients match the original coefficients as closely as possible by calculating appropriate spectral extension coordinates and blending factors. Also, like coupling coordinates, the encoder can choose how often to resend the parameters to balance accuracy vs. bitrate.
Once SPX encoding is working properly, I'll revisit variable bandwidth. However, instead of adjusting the upper cutoff frequency (which is somewhat complex to avoid very audible attack/decay), it will adjust the channel coupling and/or spectral extension ranges to keep the cutoff frequency constant while still adjusting to changes in signal complexity to keep a more stable quality level at a constant bitrate. This could also be used in a VBR mode with constrained bitrate limits.
If you want to follow the development, I have a separate branch at my Libav github repository.
http://github.com/justinruggles/Libav/commits/eac3_spx
Thursday, July 21, 2011
E-AC-3 Adaptive Hybrid Transform
I finally got the complete AHT syntax working properly. Unfortunately, the quality seems to be lower at all bitrates than with the normal AC-3 quantization. I'm hoping that I just need to pick better gain values, but I have a suspicion that some of the difference is related to vector quantization, which the encoder has no control over (a basic 6-dimensional VQ minimum distance search is the best it can do).
My first step is to find out for sure if choosing better gain values will help. One problem is that the bit allocation model is saying we need X number of bits for each mantissa. Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions. So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations. Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT). This is probably the point where I need a more accurate bit allocation model...
edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range. They are nearly identical. So the difference is clearly in VQ. Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.
edit2: dct+VQ is nearly always worse than linear quantization... I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off. I don't know what to do at this point... *sigh*
note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.
edit3: It turns out that the majority of the quality difference was in the 6-point DCT. If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better. I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation... If not then I'm at another dead-end.
edit4: I'm giving up on AHT for now. The DCT is definitely correct and is very certainly causing the quality decrease. If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this. Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.
My first step is to find out for sure if choosing better gain values will help. One problem is that the bit allocation model is saying we need X number of bits for each mantissa. Using mode=0 (all zero gains) gives exactly X number of bits per mantissa (with no overhead for encoding the gain values), but the overall quality is lower than with normal AC-3 quantization or even GAQ with simplistic mode/gain decisions. So I think that means there is some bias built-in to the AHT bit allocation table that assumes GAQ will appropriately fine-tune the final allocations. Additionally, it could be that AHT should not always be turned on when the exponents are reused in blocks 1 through 5 (the condition required to use AHT). This is probably the point where I need a more accurate bit allocation model...
edit: After analyzing the bit allocation tables for AC-3 vs. E-AC-3, it seems there is no built-in bias in the GAQ range. They are nearly identical. So the difference is clearly in VQ. Next step, try a direct comparison of quantized mantissas using VQ vs. linear quantization and consider that in the AHT mode decision.
edit2: dct+VQ is nearly always worse than linear quantization... I also tried turning AHT off for a channel if the quantization difference was over a certain threshold, but as the threshold approached zero, the quality approached that with AHT turned off. I don't know what to do at this point... *sigh*
note: analyzation of a commercial E-AC-3 sample using AHT shows that AHT is always turned on when the exponent strategy allows it.
edit3: It turns out that the majority of the quality difference was in the 6-point DCT. If I turn it off in both the encoder and decoder (but leave the quantization the same) the quality is much better. I hope it's a bug or too much inaccuracy (it's 25-bit fixed-point) in my implementation... If not then I'm at another dead-end.
edit4: I'm giving up on AHT for now. The DCT is definitely correct and is very certainly causing the quality decrease. If I can get my hands on a source + encoded E-AC-3 file from a commercial encoder that uses AHT then I will revisit this. Until then, I have nothing to analyze to tell me how using AHT can possibly produce better quality.
Friday, June 17, 2011
E-AC-3 encoder
Well, I finally got a working E-AC-3 encoder committed to Libav. The bitstream format does save a few bits here and there, but the overall quality difference is minimal. However, it will be the starting point for adding more E-AC-3 features that will improve quality.
The first feature I completed was support for higher bit rates. This is done in E-AC-3 by using fewer blocks per frame. A normal AC-3 frame has 6 blocks of 256 samples each, but E-AC-3 can reduce that to 1, 2, or 3 blocks. This way a small range can be used for the per-frame bit rate, but it still allow for increasing the per-second bit rate. For example, 5.1-channel E-AC-3 content on HD-DVDs was typically encoded at 1536 kbps using 1 block per frame.
Currently I am working on implementing AHT (adaptive hybrid transform). The AHT process uses a 6-point DCT on each coefficient across the 6 blocks in the frame. It basically uses the normal AC-3 bit allocation process to determine quantization of each DCT-processed "pre-mantissa" but it uses a finer resolution for quantization and different quantization methods. I have the 6-point DCT working and one of the two quantization methods. Now I just need to finish the other quantization method and implement mantissa bit counting and bitstream output.
The first feature I completed was support for higher bit rates. This is done in E-AC-3 by using fewer blocks per frame. A normal AC-3 frame has 6 blocks of 256 samples each, but E-AC-3 can reduce that to 1, 2, or 3 blocks. This way a small range can be used for the per-frame bit rate, but it still allow for increasing the per-second bit rate. For example, 5.1-channel E-AC-3 content on HD-DVDs was typically encoded at 1536 kbps using 1 block per frame.
Currently I am working on implementing AHT (adaptive hybrid transform). The AHT process uses a 6-point DCT on each coefficient across the 6 blocks in the frame. It basically uses the normal AC-3 bit allocation process to determine quantization of each DCT-processed "pre-mantissa" but it uses a finer resolution for quantization and different quantization methods. I have the 6-point DCT working and one of the two quantization methods. Now I just need to finish the other quantization method and implement mantissa bit counting and bitstream output.
Friday, May 6, 2011
AC-3 Variable Bandwidth
Originally, the FFmpeg/Libav AC-3 encoder used a fixed bandwidth cutoff of 21kHz for 48kHz sample rate content. For low-to-moderate bitrates (80kbps to 160kbps, stereo) this was way too high and ended up with audible, annoying high frequency artifacts and dull/tinny sounding cymbals. It also made the lower frequencies sound worse at the lower settings. As a remedy for this, I changed the default bandwidth setting to adapt to the bitrate and number of channels. The values I chose matched content produced by professional AC-3 encoders.
This fixed the issue of high-frequency artifacts, but the downside was that some content that would sound much better with the higher frequencies was getting cutoff much lower than necessary. The way I tried to solve this problem previously in Aften was to implement a variable bandwidth mode that would adjust the bandwidth per-frame to adapt to changing content. The idea was ok, but the result ended up sounding weird in some cases because of large fluctuations in bandwidth in a short period of time. It turns out that the human ear is quite sensitive to changes in bandwidth. If you start out at a low bandwidth, your ears don't know what they're missing, but if you go from high bandwidth to low bandwidth it is very noticeable.
For the Libav AC-3 encoder, I decided to revisit the Aften variable bandwidth (VBW) encoding mode to see if I could improve it. What I wanted to do was to somehow smooth out the transitions so that the changes are not as noticeable. The solution I ended up with works pretty well. I started by defining the minimum bandwidth as the existing default bandwidth and then creating a new table for maximum bandwidth. For high and very low bitrates the max and min are the same, which disables the VBW mode. For low-to-moderate bitrates the VBW mode selects a value between min and max that will result in a decent quality (fixed SNR offset of -1dB). The rate of change between frames is moderated using an exponential moving average with a half-life of 2.2 seconds. This allows for decent transition speed without being too volatile.
The VBW mode is the first step toward adaptive channel coupling. The goal will be to adaptively turn coupling on/off and to adapt the coupling start/end bands based on content.
This fixed the issue of high-frequency artifacts, but the downside was that some content that would sound much better with the higher frequencies was getting cutoff much lower than necessary. The way I tried to solve this problem previously in Aften was to implement a variable bandwidth mode that would adjust the bandwidth per-frame to adapt to changing content. The idea was ok, but the result ended up sounding weird in some cases because of large fluctuations in bandwidth in a short period of time. It turns out that the human ear is quite sensitive to changes in bandwidth. If you start out at a low bandwidth, your ears don't know what they're missing, but if you go from high bandwidth to low bandwidth it is very noticeable.
For the Libav AC-3 encoder, I decided to revisit the Aften variable bandwidth (VBW) encoding mode to see if I could improve it. What I wanted to do was to somehow smooth out the transitions so that the changes are not as noticeable. The solution I ended up with works pretty well. I started by defining the minimum bandwidth as the existing default bandwidth and then creating a new table for maximum bandwidth. For high and very low bitrates the max and min are the same, which disables the VBW mode. For low-to-moderate bitrates the VBW mode selects a value between min and max that will result in a decent quality (fixed SNR offset of -1dB). The rate of change between frames is moderated using an exponential moving average with a half-life of 2.2 seconds. This allows for decent transition speed without being too volatile.
The VBW mode is the first step toward adaptive channel coupling. The goal will be to adaptively turn coupling on/off and to adapt the coupling start/end bands based on content.
Monday, January 10, 2011
AC-3 exponent strategy
I'm having some issues with coming up with an ideal exponent strategy decision algorithm for the FFmpeg AC-3 encoder. I thought I would post some of my results so far...
Problem 1: The current algorithm is far too biased toward frequency resolution over time resolution.
Problem 2: Adjusting the exp diff threshold solves some of those problems but gives slightly worse results at high bitrates. These are likely not a problem since they're in the "inaudible differences" range for PEAQ ODG, but it's still interesting.
Problem 3: Regardless of exp diff threshold, the algorithm uses too many bits for the exponents for lower bitrates.
For some reason Aften's exponent strategy decision, which I have ported to FFmpeg, is not as good as I expected. Limiting the search to low bitrate strategy sets did allow me to compare bit usage to the FFmpeg algorithm, but overall it is not the best choice. I'm trying to come up with either a hybrid solution or modify the FFmpeg algorithm to make better choices.
Ideally I can dynamically adjust the bit usage (and/or the threshold) in the FFmpeg algorithm based on the content of the current frame in order to have a good balance for most input samples.
Update:
Ideas so far based on lots and lots of testing:
1) use current algorithm that determines new exponents based on sad threshold
2) lower the threshold from 1000 to 500
3) dynamically adjust bit usage level based on a pre-processing bit allocation run. vbr would probably be fastest. run bit allocation with original exponents at a good target snr offset. determine how many bits are left for exponents and adjust exp strategy accordingly. (if final bit allocation is also vbr, adjust based on maximum frame size)
Update2:
The complex solution is not necessarily the best solution...
Now my plan is:
1) lower EXP_DIFF_THRESHOLD to a good value after testing with a variety of samples
2) implement adaptive constant bandwidth depending on bitrate
3) implement VBR bit allocation
4) revisit exponent strategy decision, possibly just using 1 additional algorithm w/ significantly lower bit usage
Problem 1: The current algorithm is far too biased toward frequency resolution over time resolution.
Problem 2: Adjusting the exp diff threshold solves some of those problems but gives slightly worse results at high bitrates. These are likely not a problem since they're in the "inaudible differences" range for PEAQ ODG, but it's still interesting.
Problem 3: Regardless of exp diff threshold, the algorithm uses too many bits for the exponents for lower bitrates.
For some reason Aften's exponent strategy decision, which I have ported to FFmpeg, is not as good as I expected. Limiting the search to low bitrate strategy sets did allow me to compare bit usage to the FFmpeg algorithm, but overall it is not the best choice. I'm trying to come up with either a hybrid solution or modify the FFmpeg algorithm to make better choices.
Ideally I can dynamically adjust the bit usage (and/or the threshold) in the FFmpeg algorithm based on the content of the current frame in order to have a good balance for most input samples.
Update:
Ideas so far based on lots and lots of testing:
1) use current algorithm that determines new exponents based on sad threshold
2) lower the threshold from 1000 to 500
3) dynamically adjust bit usage level based on a pre-processing bit allocation run. vbr would probably be fastest. run bit allocation with original exponents at a good target snr offset. determine how many bits are left for exponents and adjust exp strategy accordingly. (if final bit allocation is also vbr, adjust based on maximum frame size)
Update2:
The complex solution is not necessarily the best solution...
Now my plan is:
1) lower EXP_DIFF_THRESHOLD to a good value after testing with a variety of samples
2) implement adaptive constant bandwidth depending on bitrate
3) implement VBR bit allocation
4) revisit exponent strategy decision, possibly just using 1 additional algorithm w/ significantly lower bit usage
Thursday, December 30, 2010
Ideas for AC-3 Bit Allocation
First I guess I should explain the AC-3 basic bit allocation process.
1) run window+MDCT on input samples
2) extract exponents (range 0-24) from MDCT coefficients
3) compress exponents for efficient encoding (part of the compression is not optional)
4) calculate approximate PSD for each frequency bin from compressed exponents
5) group frequency bins into critical bands and merge PSD for each band
6) spreading function + ATH comparison to calculate masking curve
7) allocate bits for each bin based on the PSD, masking curve, and an SNR offset value
Stages 6 and 7 use parameters from a pre-defined list for the spreading function and bit allocation. The SNR offset parameter in stage 7 is chosen by the encoder so that the number of allocated bits will fit in the frame.
I have several ideas to try to improve the accuracy of the bit allocation.
1) Adjust fast gain value in the spreading function based on the exponent strategy. This is a very rough estimate of noisiness, so it could lead to a more accurate masking curve.
2) If I decide to port VBR mode, try using the original exponents to calculate the masking curve and bap based on the target quality. Then when using the compressed exponents, select a final SNR offset that ensures that each bin gets at least as many bits as determined by the more accurate calculation.
3) Same as #2, but use in CBR mode and use delta bit allocation to adjust the masking curve to more closely match the one that used uncompressed exponents.
4) Same as #2, but use in CBR mode and vary the bandwidth first if needed in order to allocate more bits to bins that need them based on the more accurate masking curve.
5) Same as #2, but calculate a more accurate PSD based on original coefficients rather than exponents. Would probably have to modify low_comp1() in ac3.c to use a threshold comparison instead of an equivalency.
6) Run a separate psychoacoustic model using the original input samples to determine better parameters and masking curve. Adjust parameters to better match the more accurate masking curve. Use delta bit allocation if necessary.
7) Same as #2, but adjust SNR offsets per-block and per-channel to possibly achieve desired quality with fewer bits. The cost of re-transmitting SNR offsets in each block would need to be taken into consideration.
1) run window+MDCT on input samples
2) extract exponents (range 0-24) from MDCT coefficients
3) compress exponents for efficient encoding (part of the compression is not optional)
4) calculate approximate PSD for each frequency bin from compressed exponents
5) group frequency bins into critical bands and merge PSD for each band
6) spreading function + ATH comparison to calculate masking curve
7) allocate bits for each bin based on the PSD, masking curve, and an SNR offset value
Stages 6 and 7 use parameters from a pre-defined list for the spreading function and bit allocation. The SNR offset parameter in stage 7 is chosen by the encoder so that the number of allocated bits will fit in the frame.
I have several ideas to try to improve the accuracy of the bit allocation.
1) Adjust fast gain value in the spreading function based on the exponent strategy. This is a very rough estimate of noisiness, so it could lead to a more accurate masking curve.
2) If I decide to port VBR mode, try using the original exponents to calculate the masking curve and bap based on the target quality. Then when using the compressed exponents, select a final SNR offset that ensures that each bin gets at least as many bits as determined by the more accurate calculation.
3) Same as #2, but use in CBR mode and use delta bit allocation to adjust the masking curve to more closely match the one that used uncompressed exponents.
4) Same as #2, but use in CBR mode and vary the bandwidth first if needed in order to allocate more bits to bins that need them based on the more accurate masking curve.
5) Same as #2, but calculate a more accurate PSD based on original coefficients rather than exponents. Would probably have to modify low_comp1() in ac3.c to use a threshold comparison instead of an equivalency.
6) Run a separate psychoacoustic model using the original input samples to determine better parameters and masking curve. Adjust parameters to better match the more accurate masking curve. Use delta bit allocation if necessary.
7) Same as #2, but adjust SNR offsets per-block and per-channel to possibly achieve desired quality with fewer bits. The cost of re-transmitting SNR offsets in each block would need to be taken into consideration.
FFmpeg AC-3 encoder
About a month ago, I started a project to improve the FFmpeg AC-3 encoder. The first part consists of porting improvements from Aften to FFmpeg. The second part consists of further improving the encoder by adding some features that are not present in Aften. The third part is adding E-AC-3 encoding support.
So far I have done quite a few speed and structural improvements. The metadata and floating-point support have been ported, but are still going through review on ffmpeg-devel. Once those are approved, there are only a few things left to port from Aften.
I'm excited to get to work on this. I keep coming up with new ideas that I want to implement. I just have to stay focused and keep working to get the changes to the main FFmpeg repo. I'll try to post more details on this blog for the various random ideas I have.
So far I have done quite a few speed and structural improvements. The metadata and floating-point support have been ported, but are still going through review on ffmpeg-devel. Once those are approved, there are only a few things left to port from Aften.
I'm excited to get to work on this. I keep coming up with new ideas that I want to implement. I just have to stay focused and keep working to get the changes to the main FFmpeg repo. I'll try to post more details on this blog for the various random ideas I have.
Saturday, April 17, 2010
ALS encoder
For the last several months, Thilo and I have been working on an ALS encoder. I'm quite happy with the results so far. It is nearly finished, minus a few optional features. It is generally much faster than the ISO reference encoder (twice as fast in default mode) and usually gets slightly better compression. Once we get the encoder to the point where we have all of the features implemented that we want and as many bugs worked out as we can, we will do massive amounts of benchmarking. I'll try to post any interesting results here.
Subscribe to:
Posts (Atom)