Improvement of an autonomous digital dynamic range compressor. Name : Lucas Doméjean Supervisor : Dr. Josh D. Reiss

Transcription

1 Improvement of an autonomous digital dynamic range compressor Name : Lucas Doméjean Supervisor : Dr. Josh D. Reiss 14th September

2 Dedication A big thanks to my supervisor Josh Reiss and all research group occupying the room 112 for their support during this wonderful rainy summer. I would also like to thanks all the participants to my test for their time. 2

3 Preamble This report describes the project that I undertook as a student researcher at the Centre For Digital Music, Queen Mary University of London. This is a research project in the area of automatic multi- track audio mixing and will investigate new ways of automatically setting compressor parameters. Automatically setting compressor parameters dynamically adapts to the signal's features and may give better results than static human preferences. Automating such parameters does not aim at replicating artistic choices, nor replacing sound engineers, but will save amateur users and amateur musicians a lot of trouble in setting correctly the parameters and avoiding sound artifacts. In this report, we expand on work previously done in the field of automating compressor parameters and try to keep to a minimum the user- adjustable controls. We try to optimize the methods proposed by M. Massberg, D. Giannoulis and J. Maddams and investigate new ways of calculation based on subjectives listening tests. 3

4 Table of content Part A. Overview of dynamic range compression A.1 Compressor parameters and associated artefacts... 5 A.1.1 Compressor parameters/ User controls... 5 A.1.2 Associated artefacts... 6 A.2- Compression process... 8 A.2.1 Topology... 8 A.2.2 Compression stages - digital implementation... 8 A.2.3 Compression summary A.3- Parameters automation A.3.1 Time constants automation A.3.2 Knee Width automation A.3.3 Make- up gain automation Part B. Parameters improvement B.1 Theoric parameters improvement B.1.1 Time constants B.1.2 Knee width B.2 Subjective Listening Tests B.2.1 Listening test Framework B.2.2 Results extraction Part C. Development and plugin implementation C.1 Real time implementation C.2 Single Track Implementation C.3 Multitrack implementation C.4 Informal testing Part D. Discussion and further work References

5 Part A. Overview of dynamic range compression A.1 Compressor parameters and associated artefacts A.1.1 Compressor parameters/ User controls As explained in [5], a compressor has a set of controls directly linked to compressor parameters through which one can set up the effect. The most commonly used compressor parameters may be defined as follows: - A Threshold defines the level above which compression starts. Any signal overshooting the threshold will be reduced in level. - A Ratio controls the input/output ratio for signals overshooting the threshold level. It determines the amount of compression applied. - Attack and release times provide a degree of control over how quickly a compressor acts. They are also know as time constants, although the latter is usually referenced to db, denoting the gain decrease in db that the compressor will apply for the given attack time and the opposite for the release time. The attack time defines the time it takes the compressor to decrease the gain to the level determined by the ratio once the signal overshoots the threshold (as an example, if the attack time is set to 100ms, the compressor will start to compress a signal which overshoot the threshold 100ms after the overshooting). The release time defines the time it takes to bring the gain back up to the normal level once the signal has fallen below the Threshold (as an example, if the release time is set to 1s, the compressor will compress the signal during 1s after his level fall below the Threshold). - A Make- Up Gain control is usually provided at the compressor output. The compressor reduces the level (gain) of the signal, so that feeding back a make- up gain to the signal allows for matching the input and output loudness level. - A Knee Width option controls whether the bend in the response curve has a sharp angle or has a rounded edge. The Knee is the threshold- determined point where the input- output ratio changes from unity to a set ratio. A sharp transition is called a Hard Knee and provides a more noticeable compression. A softer transition where the ratio gradually grows from 1:1 to a set value in a transition region on both sides of the threshold is called a Soft Knee. It makes the compression effect less perceptible. Depending on the signal one can use hard or soft knee, with the latter being preferred when we want less obvious (transparent) compression. 5

6 Figure 1 : Static compression characteristic with make- up gain (10 db) and hard (0 db) or soft (30 db) knee for a 5:1 ratio A.1.2 Associated artefacts All these parameters need to be set properly, in order to avoid undesirable effects. Threshold, Ratio and Knee Width almost define the amount of compression applied to the signal and can't cause unpleasant artefacts unlike time constants. These time constants (attack and release times) can introduce unpleasant artefacts, so setting up these parameters need to be done properly. Very short attack and release times should be avoided because they can introduce pumping, breathing, low frequency distortion and alter the attack of an instrument (which can lead to less punch and clarity). - Pumping is caused when the gain reduction is obvious to the listener. This is usually because of quick noticeable level variations such as fast level drops after short transients that surpass the threshold level. - Breathing is less extreme than pumping and is caused by varying the noise level of a signal with high noise content. This may cause an audible airy sound similar to breathing. Very long attack and release times should also be avoided because they can introduce timbre alteration (in the case of a long attack time) and dropouts after short transient (in the case of a long release time) (see figure 3) 6

7 Figure 2 : Different attack times on a piano key hit - the envelope reshaping phenomenon - taken from [5] Figure 3 : Different release times on a snare drum hit - the phenomenon of introducing dropouts after transients All these artefacts are known to sound engineers. They can also be used as creative effects, e.g., very short attack time for a snare drum is a good way to soften the hits and to make it less dominant in the mix, and low frequency distortion can be used to add warmth to a mix. Controlling these parameters offer us a certain degree of freedom, in the different possible ways it allows us to bring instrument forward or backward in the mix. The goal of such an automated compressor is to apply dynamic range compression, making loud sounds quieter. It will be impossible for us to guess artistic intention, which includes more subjective aspects. This shows the limit of automated parameters. 7

8 A.2- Compression process A.2.1 Topology In order to understand compression, we need to look at the main stages within a compressor and its internal building blocks. The signal entering the compressor is split in two copies. One is sent to a variable- gain amplifier and the other to a side- chain where a circuit controlled by the level of the input signal applies the required gain reduction to the gain stage. Two possible topologies exist : a feedback type and a feed- forward type. It has been shown in [5] that the feed- forward topology is more accurate in a digital way of compressor implementation, even if feedback topology was traditionally used in early compressors and had the benefit that the side- chain could rectify possible inaccuraies of the gain stage. Therefore a feed- forward implementation will have to be accurate over the whole signal's dynamic range as opposed to a feedback type compressor where it will have to be accurate over a reduced dynamic range since the side- chain is fed with the compressor's output. Figure 4 : Feed- forward and Feedback type compressor designs in [6] A.2.2 Compression stages - digital implementation Different placement of each compression stages are proposed in different bibliography [19] In [5] Massberg performed a thorough study on the different designs and arrived at the conclusion that the most preferred design was as it is shown in the block diagram below. 8

9 Figure 5 : Block Diagram of the compressor configuration A The log converter The log converter is the stage where the input signal decibels (db). His digital implementation is done as follows : x[n] is converted into x G [n] = 20 log x[n] (Equation 1) A The gain computer The gain computer is the compressor stage which determines the gain reduction to be applied to the signal. This stage involves the compressor's Threshold, Ratio and Knee Width parameters. Once the signal level exceeds the threshold value, it is attenuated according to the ratio and the Knee Width (as shown in Figure 1). His digital implementation is done as follows : # % y G [n] = $ % &% x G if 2(x G!T) <!W x G + (1/ R!1)(x G!T +W / 2) 2 / (2W ) if 2 (x G!T) " W T + (x G!T) / R if 2(x G!T) > W (Equation 2) 9

10 A The level detector In that stage, the input - which is a non- smooth copy of the control voltage - is smoothed according to the values of Attack and Release times. Indeed the gradual change of gain of the input is due to the attack and release times that are introduced in the compressor's circuit through a smoothing detector filter. His digital implementation is done as follows : y L [n] =! A y L [n!1]+ (1!! A )y 1 [n] (Equation 3) where : y 1 [n] = max(x L [n],! R y 1 [n!1]+ (1!! R )x L [n] x L [n] = x G [n]! y G [n]! A = e!1/(" A f S )! R = e!1/(" R f S ) A The Make- up gain stage The purpose of this stage is to add a gain back to the signal in order to match output and input levels since the output will have a decrease gain level because of the compression. Without the make- up gain, the output signal (compressed) will always sound quieter than the input signal (uncompressed). In a digital compressor we can easily implement a make- up gain as follows : c db [n] = M[n]! y L [n] (Equation 4) where : c db [n] is the control voltage (in db) M[n] is the make- up gain A The gain stage The gain stage is responsible for attenuating the input signal by a varied amount over time of decibels (db) determined by the side- chain. The most widely used type of analogue compressor gain stage is a solid- state voltage- controlled amplifier (VCA), because they provide the most accurate and controllable gain manipulation. 10

11 An ideal VCA is in a digital design implemented as follows : y[n] = c[n].x[n] (Equation 5) where : c[n] is the control voltage in a linear scale A.2.3 Compression summary The following figures show the different stage on a particular audio sample, which is 2 seconds of a bass on a slap style. A Input Figure 6 : Compressor input - (a) linear scale - (b) db scale 11

12 A Gain computer Figure 7 : Gain computer input and output A Level Detector Figure 8 : Level detector input and output (attack and release times are varying over time) 12

13 A Make- up gain stage Figure 9 : Make- up gain stage input and output signal A Control voltage Figure 10 : Control voltage - (a) linear scale - (b) db scale 13

14 A Compression input/output A.3- Parameters automation Figure 11 : Compressor input - output and control voltage Research in automating the dynamic range compressor goes back many years [1] and is still active [2]. Compressors with partly automated parameters (like auto release) have already found their way to production both as analogue and digital design. In some existing designs, the automation of the time constants is performed by observing the difference between the peak and RMS levels of signal fed in the side- chain [3] or by observing Crest Factor and Spectral Flux [6]. In [4] an RMS measurement was used to scale the release time constant. The RMS measurement, however, is always an absolute one and dependent on overall signal level. It does not directly take into account the transient nature of the signal. That's why in [6] the authors considered this transient nature, choosing a relatively short attack and release time if the current input signal contained a considerable share of transients and a relatively long time if it were a more steady state signal (calculating Crest Factor or Spectral Flux). The concept of replacing a user- controlled ratio and knee width with an infinite ratio has been used before in both analogue and digital compressors [2], but all of these models only feature a static knee width. Similarly, automatic make- up gain can be found in some compressor designs, but only as signal- independent static compensation. The methods do not take into account loudness, even though the main purpose of make- up gain is to achieve the same loudness between the uncompressed and compressed signals. 14

15 A.3.1 Time constants automation The value of the time constants need to be strongly related to the transient nature of a signal in order to minimise artefacts. Indeed we can summarize the relative action of the time constants considering extreme transient nature of the signal as follows : - If the release time is too long for hard transients, we will experience perceivable dropouts after these transient (see section A.1) and if the release time is too short for steady- state signal (slow variation function of time) we will experience perceivable distortion because of excessive gain modulation. - If the attack time is too long for hard transients, the ompressor will not be able to catch them efficiently and if the attack time is too short for steady- state signal we will experience distortion and loss of clarity. if the signal is a steady- sate signal and if the signal is a high transient signal. That's why we need to be able to catch this transient nature, which is one of the biggest challenge in compression, the transient being most of the time underestimate with the current computations. And since most the signals are not static, dynamically changing time constants are preferred. An automated attack would therefore have a relatively low value when the current input signal contains a considerable share of transients and a relatively high value if it is a more steady- state signal. It has been shown in [7] that Crest factor and spectral Flux are both useful measure for determining the transient nature of a signal. Indeed both value of these short term signal measure tend to get low if the signal is a steady state signal and get much higher if the signal contains transient (as we can see in the academic example below see figure 12). We will then outline these two methods that give a control and flexibility over the selection of suitable time constants. 15

16 A Short term signal measurement - Crest Factor and Spectral Flux A a Crest factor The Crest factor is defined as the ratio of peak to RMS. In order to measure the short- term crest factor of a signal, we can combine a peak detector and an averaging RMS detector. A discrete time implementation with averaging coefficient! = e!1/(" f s ) calculated from time constant! and sampling frequency f s, is the following : y C [ ] [ ] [ n] = y peak n n y RMS (Equation 6) where : n y peak [ ] = max( x[ n],!y peak [ n!1] + (1!!) x[ n] ) is the peak detector 2 y RMS [ n] =!y RMS [n!1]+ (1!!) x 2 [ n] is the averaging RMS detector The time constant! for the two detectors determines the integration time of the crest factor measurement, and was at 200ms based on informal testing. Though the crest factor of a steady- state signal is relatively low and it increases once he signal contains transient. Maximum time constants are chosen for the attack and release times. These are divided by a function of crest factor to calculate a signal dependent time constant measurement. This significantly shortens the times on high tansient content like the atack part of a note. On the other hand, the steady- state signal part needs longer time constants to avoid reshaping the envelope of the notes. In order to avoid dropouts (see figure 3) and pumping (see section A.1), the effect of a high crest factor on the release time needs to be extreme. For this reason we divide the release time constant by the square of the crest factor then multiply by 2 to ensure we get the maximum time constant for sinusoidal input signals (crest factor for a pure sine wave is 2 ). Finally to compensate for the influence of the attack trajectory on the release trajectory in the decoupled peak detector [5] we substract the attack time from the release.! A [n] = 2! Amax y c 2 [n] (Equation 7)! R [n] = 2! Rmax y c 2 [n]!! A [n] (Equation 8) 16

17 where the maximum attack time is set to time to! Rmax =1s.! Amax =80ms and the maximum release A b Spectral flux The Spectral Flux measures how quickly the power spectrum of a signal changes and offers detection based on amplitude or energy information of the signal. The spectral flux method makes use of the short time Fourier transform : X(n, k) = N /2!1 " m=!n /2 x(nh + m)!(m)e! j2"mk/n (Equation 9) Then Spectral Flux (SF) is calculated from the change in magnitude of the short time Fourier transform : SF(n) = N /2!1 " k=!n /2 H( X(n, k)! X(n!1, k) N /2!1 " k=!n /2 X(n, k) (Equation 10) where H(x) = (x + x ) / 2 The attack and release time constants play an important role close to the onsets of notes, since onsets will probably cross the threshold level and trigger the compression. Therefore, we correlate the time constants to the peaks of the spectral flux, which in turn are closely related to note onsets. And because spectral flux peak values are a lot higher compared to their corresponding crest factor values, we do not have to use the square of these values in order to achieve short enough times after transients. Instead we use an instantaneous attack peak detecto with a release time of 2ms for calculating attack times and 9 ms for the release times to smooth the SF curve.! A [n] = 2! Amax SF smooth [n] (Equation 11)! R [n] = 2! Rmax SF " smooth [n]!! A [n] (Equation 12) 17

18 where : n SF smooth [ ] = max(x[n],!sf smooth [ n!1] + (1!!)SF[n])! Amax where the maximum attack time is set to =80ms and the maximum release time to! Rmax =1s. The parameter! was set to 0.8 to provide a less intense change of release times as opposed to attack times. As we can see in the figure below, the spectral flux is more sensitive than the crest factor and this enable it to detect more subtle changes to the signal. For the spectral flux calculation a N=1024 points window for the Fourier transform is used, with a hop size between adjacent windows h=512, ie 50% overlap between windows. These setting were chosen in [7] because they produce narrow peaks for the spectral fluw function, at the same time instances as the crest factor. Figure 12 : (a) Sine wave with varied amplitude and frequency. (b) Crest Factor and Spectral Flux measurement on the sine wave taken in [7] 18

19 Figure 13 : (a) Drums audio sample (b) Crest Factor and Spectral Flux measurement on the sample A.3.2 Knee Width automation Automated the Knee width set the ratio into the gain computer stage at! :1. So the signal will be perfectly limited once it exceeds T +W / 2, will reach 2 : 1 exactly at T and it will keep decreasing until T!W / 2 where it will become 1 : 1 (no compression at all). So by setting the ratio to infinity and varying the knee width one can access the whole range of compression ratios. Figure 14 : Compression Input/output curves with various knee widths for a set threshold at - 30dB taken in [7] 19

20 Automating the knee is based on the following assumption : if the compression is applied for short periods of time, so only a few peaks are trimmed and then average gain reduction is small, then the compressor should act as a limiter for more efficient result, but if the signal is heavily compressed then the compression effect should appear less obviously. So an adaptative method based on the average gain reduction appears in [5] to be relevant. A Non input dependent method A control voltage estimate is used to bias the averaging filter, by substracting the estimate before the filtering and adding it back in afterwards : c Dev [n] =!c Dev [n!1]+ (1!!)(c[n]! c Est ) (Equation 13) W[n] = 2.5(c Dev [n]+ c Est ) (Equation 14) where : c Est = T(1!1/ R) / 2 is an estimated value of the gain reduction (based on the parameter settings).! =!1/ ( f s ln!) has been chosen to 2s in order to do not match the value the gain reduction to closely (ie quickly) and to do not interfere with the release envelope. The 2.5 scale factor has been decided empirically during informal listening tests. The weakness of this method is that the knee is not directly related to information and characteristics of the input signal. A Input dependent method using Spectral Flux Automating the Knee using information and characteristics of the input signal has been done in [7] using the spectral flux computation. Indeed, signals with extensive transient content will have their spectral flux values above a certain level, considerably higher compared to that of a signal with fewer transients. So if the spectral flux minima values remain above a certain value, then the Knee width has to vary greatly with the gain reduction while if the spectral flux minima reach lower values then the Knee width has to vary less with the gain reduction (to follow the previous assumption). This is achieved using the following digital implementation : SF min [n] = min( SF[n],!SF min [n!1]+ (1!!)SF[n]) (Equation 15) SF min,avg [n] =! 2 SF min,avg [n!1]+ (1!! 2 )SF min [n]) (Equation 16) 20

21 where! and! 2 are based on time constants! =2ms and! 2 =1 ms to closely match the value of the spectral flux. Thanks to listening tests, [7] show that the relationship between the average gain reduction and the preferred knee width is non linear and instrument independent. So a polynomial of order k was used to describe this relation : W[n] = 2.5c k avg [n] (Equation 17) Then based on the average of the spectral flux minima, k was set as follows : " $ k = # %$ 0.6 if SF min,avg > if SF min,avg! 0.1 Figure 15 : Spectral Flux of a drum sample (top) and a bass sample (bottom) - taken in [7] 21

22 A.3.3 Make- up gain automation The goal for an automatic make- up gain can be describe as the followed assumption : we want to achieve that the output signal has the same perceived volume as the input signal. In [7] the idea was to estimate the Make- up gain using the average amount of applied compression : c Make!up gain [n] =!(c Dev [n]+ c Est ) (Equation 18) In [11], [12] and [13] more recent standard for loudness, based on a thresholded implementation of the ITU 1170 standard, can be used to measure loudness of the uncompressed and the compressed signal. The Make- up gain is then easily extracted using the difference of the loudness values between the two signals : c Make!up gain [n] = L(x[n])! L(y[n]) (Equation 19) where : L(X[n]) represent the loudness value of X[n] according to the ITU 1170 standard. It has been shown in [6] and [7] that such an automation can give good results and can closely match human preferencies, thanks to subjective listening tests. The weakness we can find to these previous studies is the small number of subjects considered and the small number of samples tested to set the automation rules. The challenge of finding new ways of automation will hurt the difficulty to catch the transient nature of a signal, the difficulty to get even closer to human preferencies (in what concern the Ratio, the Knee width, the Attack time, the Release time and the Threshold) and to human feelings (in what concern the Make- up gain) and the difficulty to try to find general rules in a field where the artistic intention of the sound engineer features prominently. The final aim of such an automation being real time implementation, heavy CPU use due to heavy computation in calculating the parameters also need to be avoided. As much as we could Crest Factor method which use only basic operation will be preferred instead of spectral flux whose computation is much more heavy and complex. 22

23 Part B. Parameters Improvement B.1 Theoric parameters improvement B.1.1 Time constants As explained in A.3.1, being able to catch the transient nature of a signal is the biggest issue in the compression process, due to the fact that most audio material that needs to be compressed is highly transient in nature. This section will demonstrate that Crest Factor and Spectral Flux catch efficiently this transient nature, thanks to tests on lots of audio materials and are more appropriate than other methods. B Transient Detection according to "a sub- band approach to modification of musical transient" In [16] a new, high performance transient detector and modifier were developed. It has been evaluated on various audio materials and confronted to Crest Factor and Spectral Flux efficiency. Different methods have been analyzed to perform this high performance transient detector; onset detection, spectral models, speech modeling and speech enhancement [16] have been used. The most suitable method in terms of identifying the transient parts reliably, with low computational complexity, high temporal precision, and employing a suitable signal representation for high quality transient modification has been tried to find. A full description of the tested approaches can be found in [16]. The complex domain onset detection function [16] was chosen since it fulfilled the requirements mentioned above. This function uses a Short Time Fourier Transform (STFT) and retains both phase and amplitude information. The STFT of the input signal x(n) is defined as : X(n, k) = N /2!1 " x(nh + m)w(m)e!2i!mk/n (Equation 20) m=!n /2 where n and k are the time and frequency index, i is the imaginary unit, w(m) is a N- point window and h is the hop size between adjacent windows. In the detection function each Fourier coefficient is given as a combination of its magnitude and phase X k (n) = X k (n) e i! k (n) where X k (n) is the magnitude and! k the phase of the k th bin at time n. The predicted target Fourier coefficient is 23

24 X^ k(n) = X^ k(n) e i! ^ k (n) where the target magnitude X^ k(n) is the magnitude of the ^ previous STFT frame X k (n!1) and the target phase! k(n) is calculated from the phase of the two previous STFT frames according to the phase vocoder principle [20] : ^ (n) = princarg[2!k (n!1)!! k (n! 2)] (Equation 21)! k where :! k corresponds to the unwrapped phase of the k th frequency bin and princarg maps the values of the deviation in the range of [!!,! ]. The latter is defined as a modulo operation where the divisor and remainder, per convention, share the same sign : princarg(!! k ) = mod(!! k + ","2" )+ " (Equation 22) The transient detection function is then given by the Euclidean distance between the predicted and actual measured complex Fourier coefficient for each frequency bin k. A frame- by- frame detection function is defined by : N T 0 (n) =! D k (n) (Equation 23) k=1 where D k (n) = {[!(X " k(n))#!(x k (n))] 2 +[$(X " k(n))# $(X k (n))] } 2 (Equation 24) where!(.) and!(.)denote the real and the imaginary part, respectively. The transient values can be bounded between [0,1] using a simple peak follower T(n) = T 0 (n) /!(n) where!(n) = (T 0 ( j)) capable of real time operation. max j!{0,1,...,n} For locally steady state regions, the frequency and amplitude should remain constant, T(n)! 0, and for highly transient regions, T(n)!1. The combination of phase and magnitude information enables accurate detection of pronounced (percussive) and non- pronounced (pitched) transients for both multi- voiced and single instrument signals. 24

25 B Comparison This method has been compared to the Crest factor and Spectral flux methods, as you can see in the following figures with different samples. Figure 16 : Steady- state bass sample Figure 17 : Slap style bass 25

26 Figure 18 : Drums Figure 19 : Drums 26

27 Figure 20 : Electric guitar Figure 21 : Acoustic guitar 27

28 Figure 22 : Mandolin Figure 23 : Piano 28

29 Figure 24 : Violin Figure 25 : Low pitch vocals 29

30 Figure 26 : High pitch vocals Then the time constants are correlated to the peaks of the non- normalized transient detection function (which are closely related to note onsets), as done with the crest factor and spectral flux method :! A [n] = 2! Amax T 0 (n) 2 (Equation 25)! R [n] = 2! Rmax T 0 (n)!! 2 A [n] (Equation 26) 30

31 The transient detection function as presented above gives wider peak than the spectral flux and it is less efficient in the calculation of the time constants. On the one hand if such a transient detection function is used, once the signal will be below the threshold, the compressor will still act - due to high value of the transient detection function as presented above - dropouts and pumping will not be avoided. On the other hand once the signal will be above the threshold, the compressor will be less responsive to the signal and introduce the enveloppe reshaping phenomenon. Figure from 16 to 26 show that the Spectral flux and Crest Factor are more efficient than the high performance transient detector function for our application, and that the Crest factor (whose calculation involve less heavy computation) can give really satisfactory results. That's why in the next real time implementation in section we will then consider Crest factor method instead of Spectral flux. B.1.2 Knee width Avoiding heavy computation in automating the knee using information and characteristics of the input signal, as done in [7] using the spectral flux computation, can be done using time constants information. Indeed, signals with extensive transient content will have (using the Crest factor method) a relative short attack time which will be considerably higher for a signal with fewer transients. So if the signal has extensive transient content the knee width will adapt faster to the input signal than if the signal is a more steady- state signal. This can be done considering the following digital implementation : c Dev [n] =!c Dev [n!1]+ (1!!)(c[n]! c Est ) W[n] =! A W[n!1]+ (1!! A ).2.5(c Dev [n]+ c Est ) (Equation 27) considering the previous notation. It results in an input dependent method much more efficient in term of computation. 31

32 B.2 Subjective Listening Tests B.2.1 Listening test Framework In order to find a correlation between signal features and preferred parameters some listening test will be performed [7]. In [5] Massberg based his work only on 4 short audio tracks : one of drums, one of vocals, one of guitar and one of bass. Our challenge here is to based these listening tests on a largest number of samples in order to determine more general rules. The listening tests will follow a "method of adjustment" style in order to determine how humans set up and use a dynamic range compressor in their environment and with their own equipment. They will be performed by experienced listeners, including professional sound engineers. B Samples selection 12 samples about 20 seconds have been carefully selected among 200 samples extracted from a multi- track database [15] covering a large range of instrument and being representative of the dynamic range of each instrument. The samples can be listed as follow : 1) Steady- state bass sample 2) Slap style bass 3) Drums sample 4) Drums sample 5) Electric guitar 6) Acoustic guitar 7) Mandoline 8) Piano 9) Trumpet 10) Violin 11) Low pitch vocals 12) High pitch vocals 32

33 B Framework The participants will be asked to set the values of attack time, release time and knee width on different tracks and for different amounts of compression - using combinations of threshold and ratio. Setting both threshold and ratio will set the amount of compression and the participants will be asked to choose their preferred parameters for this specific amount of compression. The values of Ratio and Threshold were chosen among the most usual values of ratio we can find in traditional harware and more recent plugins : 1.5, 2, 3, 4, 6, 8 and infinity for the Ratio and the [- 10;- 30] interval for the Threshold. Appendix II presents the test as it has been presented to the participants. B Results organization These results will aim to calculate in real time the appropriate values of the compressor parameters in a dynamic way. The analysis will be divided into two different sections : - Attack time, release time and knee width function of ratio for different windows of threshold [- 10;- 15], [- 15;- 20], [- 20;- 25], [- 25;- 30] and for each instrument. - Attack time, release time and knee function of Crest Factor and Spectral Flux and Knee Width function of control voltage to corroborate with a larger number of samples and participants the conclusions done in [7] B Number of participants To be able to extract significant results with such a listening test, the number of testers (data set) need to be very large and as far as we can representative of a large class. A number of 40 participant has been considered as sufficient to obtain enough values of parameters for each Threshold windows. B Results issues This listening test has been designed trying to avoid a lot of debating and questioning. But even with these precautions, this test suffers from some possible issues and peculiarities that might come up from it [2]. The most important issues can be listed as follow: The test was performed on individual tracks that were not part of a general mix. As a result what was tested was not compression of tracks in order to nicely fit into a mix, but rather individual track compression. This is very hard for someone to decide on, since it is unknown what he or she actually wants to 33

34 achieve using compression. That's why combination of Threshold and Ratio which set the amount of compression has been imposed to try to avoid imposing any artistic intention (as "Do I want a heavy compression or not?" "Do I want to soften the sound?" "Do I want to add some punch or some soft peak trimming?"). The test was performed by the testers in a non controlled environment without supervising and using their own systems and equipment (speakers, etc.). The validity of the results cannot be guaranteed since mistakes might have occurred. Furthermore, it is impossible for one to predict the influence the different listening environments and equipment had on each tester's choice of favored settings. Finally, the preferred human choices for each setting will be compared against the compressor's automation method. But while the first ones are single, static values, the automation is an adaptive method, producing different values for each sample instead of a fixed value. - The plugin, which has been used by the participant to perform the listening test, was a feed- forward monaural compressor with a smoothed de- coupled peak detector, which cannot represent the whole range of compressor designs. Thus the test may be considered limited to a narrow range of commercially relevant designs. All these issues are to be considered when studying the results of the evaluation. But this subjective test, based on ranking various audio samples and large number of participants should allow us to still achieve meaningful results.. B.2.2 Results extraction Unfortunately within the given time, and considering the fact that we needed a large number of participants to have a successful result we have not yet been able to extract a meaningful result. The test has been send to 49 participants, but at the time being only 12 results have been submitted and extracted. Due to the fact that some combinations of Ratio and Threshold have not been tested on any participants, no meaningfull conclusions can be done and extraction remains pending But the general extraction method will be presented below to help further analysis of the results (which should be done some time in september). To be able to considerer the two different points of the analysis previously presented, figures like the figures presented below will be displayed to corroborate in the same time the automation method by the Crest factor and to try to extract the compressor parameters as a function of Ratio. 34

35 Figure 27 : Created figures - listening test results after extraction for each different Threshold window. '+' : attack time value calculated using Crest Factor method and Knee width average of all participants / 'o' : mean value / the boxes represent the 25th and 75th percentile Comment : Such figures would be displayed for attack and release time. 35

36 Part C. Development and Plug- in implementation Most of the compressor building blocks discussed in the background section were initially implemented as Matlab functions. This allowed testing of the individual components and to generate the various plots seen in the previous sections. Then, to evaluate the automation methods on real music signals, a real- time audio plug- in implementation is much more suitable, since that allows testing the compressor under real world mixing conditions. Steinberg s VST 2.4 software development kit (SDK) [21] has been chosen, because in contrast to other formats like Audio Unit, RTAS, LADSPA or the more recent VST 3 it is widely supported by many different DAWs across both the Microsoft Windows and Mac OS- X platforms. Usually adapters (as FXPansion) are available for the very few hosts that do not support the interface natively (like Digidesign ProTools and Apple Logic). Furthermore the VST interface itself is platform- agnostic since it does not require any system- specific libraries. The same code can simply be compiled on Windows and Mac without any modifications. Juce (Jules Utilitary class extension) has been used to implement all C++ projects, and provide three different tools: - 'Introjucer' application that provides programming interface (API) and define the type of the project (plugin, etc.) and the type of target (Windows, Linux, Mac, etc.). This application will create all the files and the link compulsory to the Integrated Development Environment (IDE) - An interface dedicated to graphical aspect (GUI) which generate C++ codes and create the framework and supply widgets for labels, text fields, knobs, sliders, meters, etc. - A large number of classes linked to various objects, which provide large numbers of data and methods to use these data. 36

37 C.1 Real time implementation As shown in the previous sections, automation methods related to Crest Factor method present various beneficial aspects: a very efficient computation which does not involve fast Fourier transform calculation (FFT), the possibility of catching efficiently the transient nature of a signal, and a real time implementation thanks to sample base calculation. That's why the following implementation has been done using Crest factor method to automate the compressor parameters. C.2 Single Track Implementation Figure 28 shows the final VST plug- in implementation of the automatic compressor. The compressor parameters, except for Threshold and Make- up gain, have an automatic mode which can be activated or not. We choose to allow users to decide to have or not a manual control on all the parameters. That's why if further refinement is desired, the usr can disengage the automation methods and set the respective parameters manually. Concerning the automatic modes the users can choose, two different automatic mode can be chosen singly : 'Auto Knee' mode and 'Auto Time Constants' mode. When 'Auto Knee' mode is activated, the Ratio is set to infinity and the Knee mode is calculated using the method proposed in section B.1. It doesn't really make any sense to automate singly the attack time and the release time, that's why when the 'Auto Time Contants' mode is activated both attack time and release time are automated using the Crest factor method presented in section A.3 and B.1. When the automatic modes are activated, each sliders display the current value calculated by the automation method in relation with the automatic mode. A gain reduction meter reflects the amount of gain reduction before the make- up gain is added, so that the user also has visual feedback on how much compression is being applied. Appendix III presents the user guide in relation with the single track Automatic Compressor. Figure 28 : VST interface of the automated single track compressor 37

38 C.3 Multitrack implementation Figure 29 shows the final VST plug- in implementation of the automatic multitrack compressor. The compressor parameters are only Threshold and Make- up gain, by default both automatic mode ('Auto Knee' and 'Auto Time Constants') are activated and the users can't choose to have a manual control on Ratio, Knee width, Attack time and Release time. The Threshold slider control all the compression process, that's why it should represent more a level of compression for the user than a standard Threshold button. The automatic modes 'Auto Knee' mode and 'Auto Time Constants' are implemented as in the single track automatic compressor plugin. Appendix III presents the user guide in relation with the multitrack Automatic Compressor. Figure 29 : VST interface of the automated multitrack compressor 38

39 C.4 Informal testing During all this study, a lots of multitracks had been tested and had been provided by Tandem Launch [ a Canadian firm connected to Center For Digital Music (C4DM), in order to explore commercial opportunities related to the 'Development of Automatic mixing tools for audio and music production' research project. Seven songs have been fully mixed using the tools developed in C4DM and have been compared to manual rough- mixes (ie non mastered mixes) done by professional sound engineers. The automatic mixing chain was composed of: - Automatic EQ - Automatic compressor - Automatic faders - Automatic stereo positioning (panning) The results of this different mixes do not make up a proper listening test, but represented a way, all along this study, to test these implementations, to exchange suggestions with professional sound engineer and to be sure that these implementations can be applied real time and can give good results. The results of such informal testing seem to be quite successful and can challenge human preferences. 39

40 Part D. Discussion and further work The work done during this study was an answer to the weakness of all the previous works done in the field of setting automatically compressor parameters and especially works previously done in the 'Development of Automatic mixing tools for audio and music production' research project. A number of different modes of automation have been tested on a lot of samples in order to confirm the efficiency of the methods used. The final aim of such a project being real time implementation and uses in real mixing condition, automation method using Crest factor should be preferred and can give as efficient results as more high level features (heavier in computation) for this specific implementation. VST has been developed during this project and the method it used seem to work quite well and seem to match efficiently human preferences. But that's also where all this study show its limits, subjective listening test result need to be extracted in order to extract a lot of preferred parameters on a lot of samples, without what this study could not be totally complete, and all automation method could not be totally confirmed. A few implementation, for which we had the theory to achieve significant good results have not been done because of the limited time available for this internship. Automating the make- up gain on the single track and multitrack compressor as implemented and shown in section C.2 and C.3 considering loudness models as presented in [13] and applied off- line in [6] and in this project seems to be in the short term the most important improvement to add. It will save the user from permanently re- adjusting the gain whenever they made a significant change to any other parameters. Introducing relation between the different tracks in term of loudness and coded efficiently off- line during this project could be in the next months fully imaginable and will improve a lot the automatic multitrack compressor. The efficiency of this code multitrackcompressorautomated.m (see appendix V) has been tested on few files provided by Tamden Launch. In general, setting properly the parameters should become a much easier task if we knew what type of signal it is going to be used on. An auto compressor that only has to work on drums for instance can make many more assumptions about its input signal than a compressor taht is expected to sound well on a arbitrary tracks. That's why being able of making tracks different, creating as an example automatically submixes will turn the task on setting automatically the compressor parameters much easier. 40

41 This project is a step forward for the field of automating digital audio effects and dynamic range compression. In this sense, we could provide a summary of main recommendations for further investigation: - Using Crest Factor instead of Spectral Flux or Transient detection function seems to be more realistic in what concern real time implementation and can achieve efficient (and sometimes better) results than these others methods, as shown in this report. - Optimization on loudness calculation, which still represents ongoing research, seems to be a good and compulsory approach to be even closer to human preferences. Better loudness models would allow better make- up gain computation and more efficient transient nature understanding. The idea of considering additional features as Crest factor to weight the current loudness value and catch more efficiently the transient nature of a signal could represent a good way of further investigation. - Loudness range detection needs to be used to get information of the different tracks from one to another and to perform a related tracks dependent automatic compression as presented in [6]. Real time implementation need to be done in this sense to reduce the number of user parameters, who could only control the preferred amount of compression needed for the whole mix. It has been shown in [6] that an adaptation of the EBU definition of LRA considering a variation in a 400 ms windows (rather than 3 second) is preferred. But the idea of weighting louder part of the signal, e.g. using different percentiles windows in the loudness range calculation according to EBU models, has not been explored and can be a way of investigation, which could give good results. We could also think about additional features, linked to LRA calculation, which could give alternative way to understand if the LRA value is underestimated or overestimated. - Threshold could also be automated using RMS characteristics of the input signal with a slow moving average in order to avoid dropouts or pumping after or before short transient. Once the Threshold automated, no user parameters will be needed, but I am not still not sure this can represent a good approach to automate a compressor: the user need to have a parameter to decide the level of compressor he wants. And the way we automated the compressor in this project tends to present the Threshold as the only user parameter to decide the amount of preferred applied compression. Ratio could also be the only user control parameter, but some rules need to be find to define the ratio function of the threshold calculated using RMS characteristics of the signal. 41

42 - Another interesting idea - suggested by one of the professional sound engineer of Tamden Launch - would be to work simultaneously with a non compressed version of a tracks and a highly compressed version (with really hard parameters) of the same tracks and to apply the compression finding the right balance between the two version. This approach could represent an easier approach to the problem and could reduce the complexity of setting correctly the parameters. The preferred mix between the uncompressed and the compressed signal could then be provide thanks to a target LRA value using pre- Dynamic range compression LRA and post- Dynamic range compression LRA values as presented in [6]. - As presented in this report, listening test results extraction could corroborate conclusions on how good is the Crest factor feature to catch the transient nature of the signal. Once these conclusions confirmed they could provide a way to extract the preferred Ratio function of the time constants, directly linked to the Crest factor. The preferred Ratio could then be extracted directly from the Crest factor and could provide a way to set it automatically instead of using an infinite ratio in the automation methods. If some rules, which linked the preferred Ratio to the preferred time constants, can be extracted and if we decide to set the Ratio as the only user parameters (calculating the Threshold via RMS method), Crest factor computation will become useless and it will increase a lot the speed of real time implementation for a future automatic compressor. 42

43 Appendix I Final project organisation Here is a final project organisation which follow the study advance. 43

44 Appendix II. Listening test instructions " Setting Compressor Parameters Participant : X During these listening tests, different amounts of compression (using different combinations of threshold and ratio) will be used in order to understand how attack time, release time and knee width are set for different tracks. Not all settings are expected to sound good. For hard compression this exercise will be about finding the preferred settings and minimising the artefacts. Please open the compressor_mac.zip or compressor_win.zip and copy the VST plug- in into your VST plug- ins folder, start up your host (Logic, Pro- tools, Cubase,...) and load in the audio samples. Then send them the "Auto Compressor". It is suggested you loop the samples. Please turn off all of the buttons on the Auto Compressor (none of the blue LEDs must be lit), do not use extra plug- ins and leave the faders at 0dB (as gain changes, extra plug- ins etc would interfere with the results). This test will probably take about 1 hour, so you are allowed to take a break whenever you like. To save time, write down your choices and read the values of the parameters I suggest you divide your screen between your host and these instructions during the experiment (as you can see in the picture below) : 44

45 You can also change the values by typing them directly instead of using the buttons (as you can see in the picture below) : For all the experiments, you will have to set the values defined by the number of the experiment. If it is written : " Experiment 1 Select the track number 9 Set the Ratio at 8:1 Set the Threshold at - 11 db " Then you will have to set the thereshold at - 11 db, the Ratio at 8 and play the track number > Then (for each experiments) set the make- up gain so that you can hear the signal comfortably without it clipping. These settings should generate different amounts of compression, sometimes only a little, on the audio samples. You can easily listen to the uncompressed audio sample by turning off the "comp. active" button (see the picture above), this can allow you to hear the effect of the current compression that you are applying to the signal. - - > Now vary the attack time, the release time and the knee width until they fit to your taste (you are allowed to vary the make- up gain whenever you want to). You are not required to spend a lot of time on each experiment. - - > Try to find the values that make the compresssion sound best and write down your choices for each experiment : 45