Chapter 15 Other Modifications

Transcription

1 Chapter 15 Other Modifications We have aready seen ways to modify a sound through either edition (see Chap. 6) or fitering (see Chap. 14). Some other changes in ampitude, time, and/or frequency might be required for anaysis or to prepare fies for broadcast. To go through some functions that change the ampitude, time, and frequency parameters of sound, we wi use the song of the bird Zonotrichia capensis, the sound of the A-fat tuning fork, the voice of chid saying heo, the vocaizations of the South-American dart poison frog Aobates femorais, and the caing song of the cicada Cicada orni: data(tico) tuningfork <- readwave("sampe/tuning-fork.wav") heo <- readwave("sampe/heo.wav") femo <- readwave("sampe/aobates_femorais.wav") data(orni) 15.1 Setting the Ampitude Enveope The ampitude enveope can be changed through edition functions, as detaied in Chap. 6, but a fancy change coud be to create a sort of sound chimera by appying the ampitude enveope of one sound to another one, such that the second sound woud have the ampitude enveope of the first sound. This kind of mix, which can be usefu in psychoacoustic or bioacoustic experiments, can be performed with the seewave function setenv(). In the foowing exampe, we appy the enveope of tico to tuningfork so that the origina ampitude enveope of tuningfork is drasticay changed with the four-note ampitude enveope of tico. In this particuar case, this manipuation coud be used in a payback experiment where the frequency properties of the tico Springer Internationa Pubishing AG, part of Springer Nature 2018 J. Sueur, Sound Anaysis and Synthesis with R, UseR!, 465

2 Other Modifications sound woud be tested. The code starts with a downsamping of tuningfork so that both objects have a simiar samping frequency f s = 22,050 Hz: f <- tico@samp.rate tuningfork.u <- resamp(tuningfork, g=f) We then ca setenv() which has two main arguments: the first argument, wave1, is the input sound, and the second argument, wave2, is the sound which ampitude enveope wi be used as a reference. The resut is visuaized with spectro() (Fig. 15.1): Fig Changing the ampitude enveope with setenv(). The ampitude enveope of tico was appied totuningfork. Fourier window size = 512 sampes, overap = 0%, Hanning window

3 15.2 Echoes and Reverberation 467 Ampitude Time (s) Fig Changing the ampitude enveope with drawenv(). The ampitude enveope of tico was modified graphicay using the mouse cursor res <- setenv(tuningfork.u, tico, f=f, pot=false, output="wave") spectro(res, osc=true) As we did with the transfer function of a frequency fiter (see Sect ), we coud need to modify the ampitude enveope by drawing it on the graphica device. The interactive graphica function drawenv() dispays the osciogram of the origina sound and ets the user draw a new enveope profie by cicking severa times on the waveform (Fig. 15.2): res <- drawenv(tuningfork, output="wave") 15.2 Echoes and Reverberation The exporation of echoocation and reverberation might require the artificia addition of echoes to a signa. The seewave function echo() adds severa echoes to an input sound. This function, which is based on a simiar principe to the FIR fiter (see Sect. 14.6), processes a convoution between the input wave and a puse echo fiter. There are two parameters defining an echo: its reative ampitude compared to the input signa and its time position or deay in reference to the beginning of the input signa. These parameters are set with the arguments ampitude and deay, respectivey. For instance, choosing deay=1 and amp=0.8 resuts in the addition of an echo starting 1 s after the beginning of the input fie with an ampitude scaed by a factor of 0.8 compared to the maximum of the input fie. In the foowing exampe, some reverberation is generated on heo using a simpe echo with a high reative ampitude (0.9) and very brief deay (0.01 s). The

4 Other Modifications use of isten=true aows to isten to the resut directy: echo(heo, amp=0.9, deay=0.01, output="wave", isten=true) In this second exampe, three non-overapping echoes decreasing in ampitude by a power of 2 and increasing in duration by a factor of 1, 2, and 3 are added to the same dataset heo: echo(heo, amp=1/(2^(2:4)), deay=duration(heo)*1:3, output="wave", isten=true) 15.3 Ampitude Fitering A way to attempt ceaning a sound is to repace any signa sampe which ampitude absoute vaue fas beow a threshod by a 0 vaue. Such ampitude fiter is easy to impement and fast because it reies on the test of a simpe condition: s f itered [n] = { 0 if s[n] θ s[n] if s[n] >θ where s[n] is the origina signa and θ the threshod. The seewave function afiter() does this easy job by referring to a threshod expressed in percentage reative to the maximum of the ampitude enveope. Here is a test with a threshod set with the argument threshod at 2% and 5%, respectivey (Fig. 15.3): res1 <- afiter(femo, threshod=2, output="wave", pot=false) res2 <- afiter(femo, threshod=5, output="wave", pot=false) The function afiter() is actuay parsed by severa functions which incudes an argument threshod (e.g., dfreq(), ifreq(), timer()). The function afiter() is far to be cever since it converts every sampe beow the threshod into 0, even if these sampes are part of the main signa to be anayzed here the frog vocaizations. It is thereby necessary to check whether the

5 15.3 Ampitude Fitering 469 Fig Ampitude fiter with afiter(). The origina femo recording (eft) was passed through an ampitude fiter with a threshod of 3% (midde) and 5% (right). Fourier window size = 512 sampes, overap = 0%, Hanning windows modified sound was not too much atered in its ampitude and frequency content. A simpe test consists in istening to the resuts: isten(res1) isten(res2) Here, the sounds appear sighty distorted. Such distortion may introduce some artifacts when tracking the frequency feature, as the dominant frequency. This was the case for the sheep dataset as shown in Fig Here, in the case of femo,the impact on frequency tracking is negigibe as iustrated in Fig. 15.4: df1 <- dfreq(femo, pot=false) df2 <- dfreq(res1, pot=false) df3 <- dfreq(res2, pot=false)

6 Other Modifications 8 no fiter afiter(..., threshod=3) afiter(..., threshod=5) Frequency (khz) Time (s) Fig Use of afiter() on dominant frequency tracking. The graphic shows the resuts of tracking the dominant frequency of femo after having fitered the sound using afiter() with different settings 15.4 Modifications Using the ISTDFT We have seen in Sect that we can use the inverse short-time Fourier transform (ISTFT/ISTDFT) to appy a band-pass or a band-stop fiter to a particuar time frequency section of sound. The principe was based on the production a STDFT matrix, the incusion of 0 vaues in the STDFT matrix, and then a return in the time domain using the ISTDFT. The changes of the STDFT matrix are not imited to the introduction of 0 vaues. Any modification can be appied to a time frequency section. For instance, we can increase the ampitude of the first harmonic of the fourth note of femo by a simpe mutipication by 20 (Fig. 15.5, top-eft). We first compute the STDFT matrix and get time and frequency imits of the harmonic to be manipuated as presented in Sect : # STDFT matrix f <- femo@samp.rate ; w <- 512; ovp <- 75; wn <- "hanning" data <- spectro(femo, w=w, ovp=ovp, wn=wn, pot=false, norm=false, db=null, compex=true) # time and frequency imits of the harmonic tmin <- which.min(abs(data$time-0.44)) tmax <- which.min(abs(data$time-0.51)) fmin <- which.min(abs(data$freq-2.77)) fmax <- which.min(abs(data$freq-4.21))

7 15.4 Modifications Using the ISTDFT 471 Fig Modifications using the ISTDFT. Four exampes of sound modifications on femo based on the function istft(). The second harmonic of the first harmonic of the fourth note was ampified (top-eft), reversed in frequency (top-right), repaced by a pure tone (bottom-eft), and repaced by noise (bottom-right). Fourier window size = 512 sampes, overap = 0%, Hanning windows

8 Other Modifications We then appy the modification with (Fig. 15.5, top-eft): data1 <- data$amp data1[fmin:fmax, tmin:tmax] <- 20*data1[fmin:fmax, tmin:tmax] res1 <- istft(data1, w=w, ovp=ovp, wn=wn, We can aso invert the harmonic in frequency (Fig. 15.5, top-right): data2 <- data$amp data2[fmin:fmax, tmin:tmax] <- data2[fmax:fmin, tmin:tmax] res2 <- istft(data2, w=w, ovp=ovp, wn=wn, We may aso ike to repace the harmonic by a pure tone (Fig. 15.5, bottom-eft): data3 <- data$amp data3[fmin:fmax, tmin:tmax] <- 0 data3[fmin+(fmax-fmin)/2, tmin:tmax] <- compex(rea=max(re(data3))) res3 <- istft(data3, w=w, ovp=ovp, wn=wn, Eventuay we can repace the harmonic with white noise (Fig. 15.5, bottomright) data4 <- data$amp re <- im <- rnorm(ength(fmin:fmax)*ength(tmin:tmax)) data4[fmin:fmax, tmin:tmax] <- compex(rea=re, imaginary=im) res4 <- istft(data4, w=w, ovp=ovp, wn=wn, Figure 15.5 was produced with the foowing code. The graphic function named pot.istft() is designed to faciitate the repetition of the pots: coeves <- seq(-80,0,1) pot.istft <- function(x){ spectro(x, coeves=coeves, fim=c(0,10), scae=false, tab="", fab="") rect(xeft=0.44, ybottom=2.77, xright=0.51, ytop=4.21, border="red", wd=3) } par(mfrow=c(2,2), mar=c(3,3,1,1), oma=c(2,2,0,0)) pot.istft(res1) (continued)

9 15.4 Modifications Using the ISTDFT 473 pot.istft(res2) pot.istft(res3) pot.istft(res4) mtext("time (s)", side=1, outer=true) mtext("frequency (khz)", side=2, outer=true, as=0) We can aso take advantage of the ISTDFT to shift positivey or negativey the frequencies of a sound to be used, for instance, in payback experiments. The function fs() for inear frequency shift of seewave can appy such drastic change. The main argument is shift that contros the frequency shift in Hz. If we take the orni song, we first appy a high-pass FIR fiter to remove the wind noise that coud generate undesired frequency bands: orni.fitered <- fir(orni, from=150, output="wave") We afterward use fs() to appy a positive and a negative frequency shift of 1000 Hz (Fig. 15.6): orni.higher <- fs(orni.fitered, shift=1000, output="wave") orni.ower <- fs(orni.fitered, shift=-1000, output="wave") The process of frequency shift can be easiy incuded in a for oop to generate a succession of modified sounds as iustrated in the DIY box DIY 15.1 How to generate a series of sounds with different inear frequency shifts It can be necessary for the purpose of an experiment to generate a series of stimui that resut of a inear frequency shift. We can, for instance, think that in the case of the cicada Cicada orni, a payback experiment coud aim at testing the importance of frequency in the encoding-decoding process of species recognition. To run such a test, it coud be necessary to appy a series of reguar shift between 2000 and Hz with a step of 500 Hz. We first prepare a numeric vector containing the shift vaues: shift <- seq(-2000, 2000, by=500) (continued)

10 Other Modifications DIY 15.1 (continued) We remove the 0 vaue: shift <- shift[shift!=0] so that we obtain the vector: shift [1] We use a FIR fiter to remove ow-frequency noise due to wind: orni.fitered <- fir(orni, from=150, output="wave") We write a for oop that cas the function fs() to appy the frequency change and the function savewav() to export the modified sounds into individua.wav fie fies named orni_fs_-2000.wav, orni_fs_-1500.wav, orni_fs_-1000.wav, etc.: for(i in 1:ength(shift)){ tmp <- fs(orni.fitered, shift=shift[i], output="wave") savewav(tmp, fie=paste("orni_fs_", shift[i], ".wav", sep="")) } 15.5 Modifications Using the Hibert Transform We have seen in Sects and that the Hibert transform can be used to obtain the ampitude enveope and the instantaneous frequency of monotona sounds. If we ca the seewave functions env() and ifreq(), we can store the Hibert ampitude enveope and the instantaneous frequency of tico in two

11 15.5 Modifications Using the Hibert Transform 475 Fig Linear frequency shift using the ISTDFT. The song of orni was shifted toward ow or high frequencies with the function fs() that uses the ISTDFT in background. Fourier window size = 512 sampes, overap = 0%, Hanning window distinct objects: env.tico <- env(tico, pot=false) ifreq.tico <- ifreq(tico, pot=false) seewave incudes a function to synthesize sound, synth2(), thatusesas input the Hibert ampitude enveope and the instantaneous frequency. This function, which is detaied in Sect. 18.5, can be used to recover the origina sound of tico. We just extract and mutipy by 1000 the second coumn of ifreq.tico that contains the instantaneous frequency in Hz: f <- tico@samp.rate ifreq.tico <- ifreq.tico$f[,2]*1000

12 Other Modifications We use these parameters to feed the function synth2(): res <- synth2(env=env.tico, ifreq=ifreq.tico, The object res is simiar to tico so that the manipuation is not that interesting. However, we can pay with enveope and instantaneous frequency independenty to modify the sound in a more fancy way. First we coud inverse the ampitude enveope but not the frequency moduations 1 : res <- synth2(env=rev(env.tico), ifreq=ifreq.tico, Second, we may reverse the instantaneous frequency but not the enveope (Fig. 15.7,eft): res <- synth2(env=env.tico, ifreq=rev(ifreq.tico), We can appy any arithmetic operation as squaring the ampitude enveope: res <- synth2(env=env.tico^2, ifreq=ifreq.tico, shifting the instantaneous frequency by adding 1000 Hz: res <- synth2(env=env.tico, ifreq=ifreq.tico+1000, 1 A simpe time reversion of the sound can be obtained with the function revw() as seen in Sect

13 15.5 Modifications Using the Hibert Transform 477 Fig Modifications using the Hibert transform. Three exampes of sound modifications on tico based on the function synth2(). The frequency moduation was inverted according to time (eft), the frequencies were mutipied by 2 (midde), and the frequency moduation was repaced by 4000 Hz pure tone (right). Fourier window size = 512 sampes, overap = 0%, Hanning window or mutipying the instantaneous frequency by a factor of 2 (Fig. 15.7, midde): res <- synth2(env=env.tico, ifreq=ifreq.tico*2, We can aso generate sighty more compex modifications. For instance, we can seectivey change the second note which position can be estimated automaticay using the function timer() (see Sect. 8.3): # timer() ca tres <- timer(tico, threshod=5, msmooth=c(50,0), pot=false) # start position of the second note (continued)

14 Other Modifications start <- tres$s.start[2] # end position of the second note end <- tres$s.end[2] # assignation of env.tico object in a new object env.tico.mod <- env.tico We repace the sampe vaues of the second note by the maximum of the ampitude enveope with: env.tico.mod[foor(start*f):foor(end*f)] <- max(env.tico) that we use with synth2(): res <- synth2(env=env.tico.mod, ifreq=ifreq.tico, The second note now appears as a rectanguar signa but sti containing the origina frequency moduation. This suggests that the ampitude moduation coud be removed without touching to the frequency moduation, a manipuation often carried out in anima payback experiments. Such manipuation is aso directy accessibe with the seewave function rmam(): res <- rmam(tico, pot=false, output="wave") At the opposite, we might intend to keep the ampitude moduation but to remove any frequency moduation. Here is a possibe soution by repacing the frequency moduation by a 4000 Hz pure tone (Fig. 15.7, right): res <- synth2(env=env.tico, ifreq=rep(4000, times=ength(ifreq.tico)),