Professor of Audio Technology University of Salford Dukinfield, England, United Kingdom
Theoretical audio signal processing assumes that the signal is represented with infinite precision. In practice this is not the case. Real audio is processed with a finite amount of precision. There is a limit to the number of bits that our processors can have. This affects both the resolution as well as the maximum signal levels in our processing. So how does the finite precision of our processing affect what we do, and what can we do to achieve the best processing results possible? The purpose of this tutorial is to examine the effect of finite precision in DSP. At the end the listener will understand the effects of finite precision and possible solutions to it.
In particular: we will examine number formats in audio DSP; Fixed point (Integer versus Fractional); Floating point and block floating point. The effect of different operations on the number representing the signal’s size and resolution. The need for Quantisation, Rounding and Truncation and the problems they may cause. We will examine dither solutions (Rpdf, Tpdf, and self dithering due to signal noise. Overload and saturation and their effects and possible solutions also will be examined.
Finite precision effects in the filtered signal path such as; noise enhancement, limit cycles, and overflow oscillations will be examined and solutions presented.
In addition finite precision effects on the filter’s frequency response will be covered: The effect of filter coefficient quantisation on frequency response irregularities in equalisers and filters, the effect of oversampling on coefficient accuracy requirements, and the how effect of filter structure may offer possible solutions.
Finally we will look at noise shaping and system structure as possible solutions to these issues.
The tutorial will be suitable for both novice and intermediate levels and will contain both visual and audible examples, and where relevant code examples.
