Audio watermarking using spikegram and a two-dictionary approach

Transcription

1 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 1 Audio watermarking using spikegram and a two-ditionary approah Yousof Erfani, Student Member, IEEE, Ramin Pihevar, Member, IEEE, and Jean Rouat, Senior Member, IEEE Abstrat This paper introdues a new audio watermarking tehnique based on a pereptual kernel representation of audio signals (spikegram). Spikegram is a reent method to represent audio signals. It is ombined with a ditionary of gammatones to onstrut a robust representation of sounds. In traditional phase embedding methods, the phase of oeffiients of a given signal in a speifi domain (suh as Fourier domain) is modified. In the enoder of the proposed method (two-ditionary approah), signs and phases of gammatones in the spikegram are hosen adaptively to maximize the strength of the deoder. Moreover, the watermark is embedded only into kernels with high amplitudes where all masked gammatones have been already removed. The effiieny of the proposed spikegram watermarking is shown via several experimental results. First, robustness of the proposed method is shown against kbps MP with an embedding rate of 5.5 bps. Seond, we showed that the proposed method is robust against unified speeh and audio ode (4 kbps USAC, linear preditive and Fourier domain modes) with an average payload of 5-15 bps. Third, it is robust against simulated small real room attaks with a payload of roughly 1 bps. Lastly, it is shown that the proposed method is robust against a variety of signal proessing transforms while preserving quality. Index Terms Copyright protetion, Watermarking, Spikegram, Gammatone filter bank, Sparse representation, Multimedia seurity I. INTRODUCTION An analysis by the Institute for Poliy Innovation onludes that every year global musi piray is making 1.5 billion of eonomi losses, 71 U.S. jobs lost, a loss of.7 billion in workers earnings and a loss of 4 million in tax revenues, 1 million in personal inome tax and 11 million in lost orporate inome and prodution taxes. Most of the musi piray is beause of rapid growth and easiness of urrent tehnologies for opying, sharing, manipulating and distributing musial data [1]. As one promising solution, audio watermarking has been proposed for post-delivery protetion of audio data. Digital watermarking works by embedding a hidden, inaudible watermark stream into the host audio signal. Generally, when the embedded data is easily removed by manipulation, the watermarking is said to be fragile whih is suitable for authentiation appliations, whereas for opyright appliations, the watermark needs to be robust against manipulations []. Watermarking has also many other appliations suh as opy Copyright () 1 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. All authors are with NECOTIS group, Department of Eletrial and Computer Engineering, Université de Sherbrooke, Sherbrooke, QC, 5 boul. de l Université, Sherbrooke, ({yousof.erfani, ramin.pihevar, jean.rouat}@usherbrooke.a). Manusript reeived Month,DD,YYYY; revised Month DD, YYYY. ontrol, broadast monitoring and data annotation [], [4], [5]. For audio watermarking, several approahes have been reently proposed in the literature. These approahes inlude audio watermarking using phase embedding tehniques [], ohlear delay [7], spatial masking and ambisonis [8], eho hiding [], [1], [11], pathwork algorithm [1], wavelet transform [1], singular value deomposition [14] and FFT amplitude modifiation [15]. State of the art methods introdue phase hanges in the signal representation (i.e., from the phase of the Fourier representation) [], [1], while we adopt a more original strategy by using two ditionary of kernels and by shifting the sinusoidal term of the gammatones [17], [18]. In this paper, the watermarking is of multi-bit type [1] and ould be used for data annotation. Multiple ditionaries for sparse representation has already drawn the attention of researhers in signal proessing [], [1], [], []. For example, in [], a two-ditionary method is proposed for image inpainting where one deomposed image serves as the artoon and the other as the texture image. Also, a watermark detetion algorithm was proposed by Son et al. [1] for image watermarking where two ditionaries are learned for horizontally and vertially lustered dots in the half tone ells of images. In [], authors propose an audio denoising algorithm using a sparse audio signal regression with a union of two ditionaries of modified disrete osine transform (MDCT) bases. They use long window MDCT bases to model the tonal parts and short window MDCT bases to model the transient parts of the audio signals. In [5], two random ditionaries are used to improve the ryptographi seurity of spread spetrum (SS) image watermarking. In all mentioned methods, the goal is to have an effiient representation of the signal. However for audio watermarking, one goal is to manipulate the signal representation in a way to find adaptively the spetro-temporal ontent of the signal for effiient transmission of watermark bits. In this paper, we propose an embedding and deoding method for audio watermarking whih jointly uses two type of gammatone ditionaries (inluding gammasines and gammaosines) and a spikegram of the audio signal. It is shown in [4] that in omparison to blok based representations, spikegram is time-shift invariant, where the signal is deomposed over a ditionary of gammatones. To generate the spikegram, we use the Pereptual Mathing Pursuit (PMP) [5]. PMP is a bio-inspired approah that generates a sparse representation and takes into aount the auditory masking at the output of a gammatone filter bank (the gammatone ditionary is obtained by dupliating the gammatone filter bank at different time samples). The proposed method is blind, as the original signal is not () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

2 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY Center frequeny (Hz) (A) An artifial audio signal (B) Spikes in the spikegram A masker kernel found by PMP A masked kernel removed by PMP ms Time Figure 1. A D plane of gammatone kernels of a spikegram generated from PMP [5], [5] oeffiients. The D plane is generated by repeating N = 4 gammatones at different hannels (enter frequenies) and at eah time samples. A gammatone with non-zero oeffiient is alled a spike. required for deoding. Also, the only information needed to be shared between the enoder and the deoder inlude the key, whih is used as the initial state for a Pseudo Noise (PN) sequene, the type and parameters for ditionary generation. To evaluate the performane of the proposed method, extensive experimental results are done with a variety of attaks. Robustness against lossy pereptual odes is a major requirement for a robust audio watermarking, thus we deided to evaluate the robustness of the method against kps MP (although not used that often anymore, it is still a powerful attak whih an be used as an evaluation tool). The proposed method is robust against kbps MP ompression with the average payload of 5.5 bps while the state of the art robust payload against this attak is lower than 5. bps []. In this paper, for the first time, we evaluate the robustness of the proposed method against USAC (Unified Speeh and Audio Coding) [7], [8], []. USAC is a strong ontemporary ode (high quality, low bit rate), with dual options both for audio and speeh. USAC applies tehnologies suh as spetral band repliation, CELP ode and LPC. Experiments show that the proposed method is robust against USAC for the two modes of linear preditive domain (exeuted only for speeh signals) and frequeny domain (exeuted only for audio signals), with an average payload of 5-15 bps. The proposed method is also robust against simulated small real room attaks [], [1] for the payload of roughly 1 bps. Lastly, the robustness against signal proessing transforms suh as resampling, re-quantization, low-pass filtering is evaluated and we observed that the quality of signals an be preserved. In this paper, the sampled version of any time domain signal is onsidered as a olumn vetor with a bold fae notation. II. SPIKEGRAM KERNEL BASED REPRESENTATION A. Definitions With a sparse representation, a signal x[n],n =1:N (or x in vetor format) is deomposed over a ditionary = {g i [n]; n = 1 : N,i = 1 : M} to render a sparse vetor = { i ; i =1:M} whih inludes only a few non-zero oeffiients, having the smallest reonstrution error for the host signal x [4], [5]. Hene, x[n] MX i g i [n], n =1,,..,N (1) i=1 where i is a sparse oeffiient. A D time-hannel plane is generated by dupliating a bank of N gammatone filters (having respetively different enter frequenies) on eah time sample of the signal. Also, all the gammatone kernels in the mentioned D plane form the olumns of the ditionary (Hene, M = N N). Thus g i [n] is one base of the ditionary whih is loated at a point orresponding to hannel i {1,..,N }, and time sample i {1,,..,N} inside the D time-hannel plane (Fig.1). The spikegram is the D plot of the oeffiients at different instants and hannels (enter frequenies). The number of non-zero oeffiients in i per signal s length N is defined as the density of the representation (note that sparsity = 1-density). To ompute the sparse representation in (1), many solutions have been presented in the literature inluding Iterative Thresholding [], Orthogonal Mathing Pursuit (OMP) [], Alternating Diretion Method (ADM) [4], Pereptual Mathing Pursuit (PMP) [5]. Here, we use PMP for three different reasons: PMP is not omputationally expensive, it is a high resolution representation for audio signals, and it generates auditory masking thresholds and removes the inaudible ontent under the masks [5]. PMP is a reent approah whih solves the problem in (1) for audio and speeh using a gammatone ditionary [5], [5]. PMP is a greedy method and an improvement over Mathing Pursuit []. PMP finds only audible kernels for whih the sensation level is above an iteratively updated masking threshold and neglets the rest. A kernel is onsidered as a masked kernel if it is under the masking of (or lose enough in time or hannel to) another masker kernel with larger amplitude. The effiieny of PMP for signal representation is onfirmed in [5] and [5]. The gammatone filter bank (used to generate the gammatone ditionary) is adapted to the natural sounds [4] and is shown to be effiient for sparse representation [5]. A gammatone kernel equation [17] has a gamma part and a tone part as below g[n] =an m 1 e ln os[ (f /f s )n + ],n=1,..,1 () in whih, n is the time index, m and l are used for tuning the gamma part of the equation. f s is the sampling frequeny, is the phase, f is the enter frequeny of the gammatone. The term a is the normalization fator to set the energy of eah gamatone to one. Also, the effetive length of a gammatone is defined as the duration where the envelope is greater than one perent of the maximum value of the gammatone. In this paper, a 5-hannel gammatone filter bank is used (Table I). Their bandwidths and enter frequenies are fixed and hosen to orrespond to 5 ritial bands of hearing. They are implemented at the enoder and the deoder using (). Also, a gammatone is alled a gammaosine when =or a gammasine when = /. In Table I, enter frequenies and effetive lengths for some gammatones, versus their hannel () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

3 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY Table I THE EFFECTIVE LENGTHS AND CENTER FREQUENCIES FOR GAMMATONE KERNELS USED IN THIS WORK. Channel number Effetive length (mse) Center frequeny (Hz) gammaosine Effetive length gammasine mse Figure. A sample gammaosine (blue) and gammasine(red) (for hannel-8) with a enter frequeny of 84 Hz and an effetive length of 1. mse. Gammasines and gammaosines are hosen in the watermark embedding proeess based on their orrelation with the host signal and the input watermark bit. The sampling frequeny is 44.1 khz. numbers are given. In Fig., hannel 8 gammasine and gammaosine are plotted. B. Good harateristis of spikegram for audio watermarking 1) Time shift invariane: In most traditional watermarking tehniques, the signal representation is blok-based, where the signal is divided into overlapping bloks and watermark is inserted into eah blok. The onventional methods have two drawbaks. First, they might misrepresent the transients and periodiities in the signal. Moreover, in the blok-based representation of nonstationary signals, small time shifts in the time domain signal might produe large hanges in the representation, depending on the position of a partiular aousti event in eah blok [4]. The spikegram representation in (1) is time-shift invariant and is suitable for robust watermarking against time shifting de-synhronization attak. ) Low host interferene when using spikegram: In (1), many gammatones have either zero oeffiients or are masked, thanks to PMP. Therefore, ompared to traditional transforms suh as STFT and Wavelet transforms, spikegram is expeted to yield less host interferene at the deoder (see Fig.1 and Fig.14 for experiments regarding the dependene of the error rate and the quality with the sparsity of the spikegrams). ) Effiient embedding into robust oeffiients: The watermark bits are inserted only into large amplitude oeffiients obtained by PMP, where all inaudible gammatones have been a priori removed from the representation. III. TWO-DICTIONARY APPROACH The watermark bit stream is symbolized by b whih is an M 1 vetor (M <M). The goal is to embed the watermark bit stream into the host signal. K, ap 1 vetor (P <M ), is the key whih is shared between the enoder and the deoder of the watermarking system. Also, the sparse representation of the host signal x on the gammaosine ditionary (i.e., i ) is assumed to be known. The proposed method relies on the fat that the hange in signal quality should not be pereived when hanging the phase of speifi gammatone kernels. Moreover, it is alled a two ditionary approah, as a andidate kernel for watermark insertion, is adaptively seleted from a gammaosine or gammasine ditionary. For inserting multiple bits, the host signal x[n] (x in vetor format) is first represented using (1). Then, M gammatones g k [n] from the representation in (1) are seleted (the seletion of watermark kernels is detailed in setion III-D). These gammatones form the watermark ditionary D 1 and arry the watermark bit stream b k,k = 1,,..,M. Other M 1 = M M kernels form the signal ditionary D. The signal and watermark ditionaries are disjoint subsets of the gammatone ditionary used for sparse representation in (1), thus D 1 \ D = ;. Eah watermark bit b k serves as the sign of a watermark kernel. Hene (1) beomes XM1 XM y[n] = i g i [n]+ b k k g k [n] () i=1 k=1 where y[n] is the watermarked signal. In (), if the watermark and signal ditionaries use the same gammatone kernels, the watermarking beomes a one ditionary method. In one ditionary method, the watermark bits are inserted as the sign of gammatone kernels. In two ditionary method, in addition to the manipulation of the sign of gammatone kernels, their phase also might be shifted as muh as /, based on the strength of the deoder. Hene, for the two-ditionary approah, eah watermark kernel is hosen adaptively from a union of two ditionaries, one ditionary of gammaosines and one ditionary of gammasines. The k th watermark kernel in the watermark ditionary is found adaptively and symbolized with f k whih is either a gammasine or a gammaosine. Thus for the two ditionary method, the embedding equation in () beomes XM1 XM y[n] = i g i [n]+ b k k f k [n] (4) i=1 k=1 To deode of the p th watermark bit, we ompute the projetions of the watermarked signal on the p th watermark kernel. < y, f p >= b p p + XM1 i=1,i=p M X k=1,k=p i < g i, f p > + b k k < f k, f p > The number of samples used to ompute the projetion in (5) is equal to the gammatone effetive length. The goal is to deode the watermark bit as the sign of the projetion < y, f p >. We later show how to find the best watermark (5) () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

4 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 4 C h ( l ) C h ( k ) C h ( j ) C h ( i ) ( i) l Figure. Watermark insertion using the two-ditionary method. First, the spikegram of the host signal is found using PMP with a ditionary of 5- hannel gammaosines, loated at eah time sample along the time axis. Then for eah proessing window and eah hannel and based on the embedding bit b, the gammaosine, or gammasine (loated at a blue irle) with maximum strength fator (m or m s) is hosen for the watermark insertion. In this work, gammatone hannels Ch s are seleted in the range of 1-4 and - 1 (odd hannels only) for the watermark insertion. Also, to get the same embedding strength for different embedding hannels, proessing windows of different hannels have the same length. kernels so that the first two terms in the right side of (5) have the same signs as the watermark bit b p. There are two soures of interferene in (5). First, the right term in the right side of (5) is the interferene that the deoder reeives from other watermark bit insertions. To remove this interferene term, the watermark insertion is performed into limited number of hannels so that the watermark gammatones are unorrelated. In fat, to design the watermark ditionary, we hoose a subset of the full overomplete ditionary in suh a way that the watermark kernels are spetro-temporally far enough suh that they are unorrelated. Thus the watermark bits will be deoded independently. Hene, in Fig., for eah hannel and time sample, two neighbor watermark kernels should be separated with at least one effetive length and at least one hannel. With this assumption, the orrelation between watermark gammatones will be less than.. The seond soure of interferene is the left term in the right side of (5) whih originates from the orrelations between watermark and signal gammatones, that is shown in (7). We redue this interferene in the enoder of the system in the next setion, by adaptively searhing for and embedding into the strongest watermark gammatones in the spikegram. As embedding of multiple watermark bits are performed independently, thus in the next setion, only the single bit watermarking using the two ditionary method is explained. A. The proposed informed embedder Equation (1) is used to resynthesize the host signal x from sparse oeffiients and gammaosines. Now, we want to embed one bit b { 1, 1} from the watermark bit stream b by hanging the sign and/or the phase of a gammaosine kernel g p (the p th kernel found by PMP, still to be determined later in this setion) with amplitude p (to be determined) loated at a given hannel and proessing window (eah proessing window is a time frame inluding several effetive lengths of a gammatone, Fig.). To find an effiient watermark kernel f p whih bears the x b k, m, α b k s, m s, α s f p[ n ] = g [ n k ] m > m s α p = α b f [ n ] = g s [ n ] p k s α = α Figure 4. The proposed embedder for a given hannel and proessing window. The gammasine or gammaosine with maximum strength fator is hosen as the watermark kernel and its amplitude is set to its assoiated sparse oeffiient in the spikegram. Finally () is used to resynthesize the watermarked signal y (in vetor format). m s and m are respetively the strength fators for gammasine andidate and gammaosine andidate. greatest deoding performane for the watermark b, we write the 1-bit embedding equation as follows: y[n] = MX i=1,i=p p s i g i [n]+b p f p [n] () where the watermarked kernel f p for a given hannel number an be a gammaosine (g) or a gammasine (gs) whih are zero and / phase-shifted versions of the original gammatone kernel g p, respetively. The orrelation between the watermarked signal y and the watermarked kernel f p, is found as below MX < y, f p >= i < g i, f p > +b p (7) i=1,i=p Hene, to design a simple orrelation-based deoder, the sign of the orrelation in the left side of (7) is onsidered as deoded the watermark bit. In this ase, for orret detetion of the watermark bit b, the interferene term should not hange the desired sign at the right hand side of (7). Moreover, the gammatone ditionary is not orthogonal, hene the left term in the right side of (7) may ause erroneous detetion of b. For a strong deoder, two terms on the right side of (7), should have the same sign with large values. We later show that by finding an appropriate gammaosine or gammasine in the spikegram, the right side of (7) an have the same sign as the watermark bit b. In this ase, the module of orrelation in (7) is alled watermark strength fator m p for the bit b and a greater strength fator means a stronger watermark bit against attaks. In this ase, (7) beomes < y, f p >= bm p (8) For a large value strength fator (and with the same sign of the watermark bit), we searh the peak value of the projetions using (7) when a gammatone andidate is a gammaosine or gammasine. Thus, for a given hannel, a proessing window and watermark bit b, the signal interferene is minimized at the deoder using the informed enoder in (7). We do the following proedure to find the phase, position and the amplitude of the watermarked kernel f p (Fig. 4). For a given hannel, we onsider the watermark gammatone andidate f p (the p th gammatone kernel in the signal representation of (1)) to be a gammaosine g or a gammasine gs. Then, do the following steps: y () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

5 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 5 y k, P [ k ] k P [ k ] P k ] > P [ k ] b =s i g n P [ k ]) s, s s [ s s ( b = s ign P s [ k ]) ( s Figure 5. The proposed one-bit deoder. The projetions with maximum absolute value for gammaosines and gammasines are found as P [k ] and P s[k s] and at time sampes k and k s respetively. The watermark bit ˆb is onsidered as the sign of the projetion with the largest absolute value. Watermark strength fator Two ditionary method, m t =max (m,m s ) One ditionary method, m Shift the watermark gammatone andidate f p alongside all proessing windows, at time shifts equal to multiples of the gammatones effetive length. For eah shift ompute the orrelation of the watermarked signal with the sliding watermark andidate kernel. Then, find the absolute maximum of the orrelation (watermark strength fator) < y, f p > using (7) (Fig.). The result is a strength fator, symbolized as m for gammaosine, loated at time sample k with amplitude and also another strength fator, symbolized as m s for a gammasine kernel loated at k s with the amplitude s. Thus m = < y,g[n k ] >, m s = < y,gs[n k s ] >. Afterwards, the gammaosine or gammasine with greater strength fator is hosen as the final watermark gammatone f p and its time shift (sample), amplitude and phase are registered. Gammatone or gammasine with greater strength fator is hosen as the final watermark gammatone f p with the final watermark strength fator being m t = max(m,m s ). The respetive k or k s, amplitude or s and phases are kept. Therefore, the algorithm finds the optimal watermark gamatone from two ditionaries inluding one ditionary of gammaosines and one ditionary of gammasines. After all, the watermarked signal is synthesized using (), f p with its amplitude set to b p. This is equivalent to finding a time-hannel point in the spikegram ( Fig.) where the optimal position for embedding is found. Then, g p is replaed with the watermark gammatone f p. B. The proposed deoder At the deoder, as the watermark kernels assoiated to the watermark bits are unorrelated, the deoding of one watermark bit does not interfere with the deoding of other bits. Hene, the same searh proedure, used in the embedder to find the watermarked kernel andidate, is applied. Therefore for a given hannel and proessing window, the deoding proedure is shown Fig. 5. For a given hannel, suppose the watermark gammatone andidate f p to be a gammaosine g or a gammasine gs. Then do the following steps: Shift the watermark kernels g and gs alongside the proessing window at time shifts (respetively k or k s ) equal to multiples of the gammatone s effetive length. For eah k (either k or k s ), ompute the orrelation of Channel range Figure. The average watermark strength fator for one-ditionary and twoditionary methods versus embedding hannel range. Simulations are done on 1 signals ( minutes eah) of different musi genres and English voies. In eah embedding experiment, watermark is inserted in a speifi hannel range with a 45 mse proessing windows. The maximum peak of a db AWGN is also plotted as a horizontal dashed line in blak. The 5% onfidene intervals are plotted as vertial lines at the enter of the average results for three onsequensive hannels. Input signals are normalized to unit l norm. the watermarked signal with the sliding watermark kernel andidate. Then, find the absolute maximum of the orrelation P [k] = < y,g[n k] > and P s [k] = < y,gs[n k] > (Fig.5). The result is one absolute maximum orrelation m = max( P [k] ) for a gammaosine loated at the time sample k with the amplitude and also another absolute maximum orrelation m s = max( P s [k] ) for a gammasine kernel loated at k s with the amplitude s. Finally, if m > m s then b = sign(p [k ]) otherwise b = sign(p s [k s ]). The resynhronization of the deoder and the enoder is detailed in setion IV-. In Fig., the watermark strength fator is plotted versus the embedding hannel ranges for the ases of one ditionary (when using only gammaosine kernels) and two ditionaries (when both gammasine and gammaosine kernels are used). As expeted, the strength fators for the two-ditionary method is all the time greater than the one for one ditionary method. The maximum improvement ours for middle hannels between 5 and 1. Also, towards greater embedding hannel ranges, the strength fator beomes smaller. For an insight about the robustness of the methods against db additive white Gaussian noise (AWGN), the average maximum peaks of the noise signal is plotted as a horizontal line. As is seen, the embedding hannel ranges whih are robust against db AWGN are wider for two ditionary method ompared to one ditionary method (for one ditionary method it is below hannel number 1 while for the two-ditionary method, it spans 1-4 hannel range.) As illustrated in Fig., for hannel j, eah proessing window inludes several effetive lengths of the gammatone, l j. Thus, for eah hannel j, the algorithm searhes for a watermark gammatone andidate among d L P lj e gammatones (Fig., vertial red lines in eah proessing window). Thanks to the ontent-based aspet of the approah, phase () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

6 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY P e SNR=1 db SNR=15 db SNR= db SNR=5 db Channel number Figure 7. The error probability of the deoder under additive white Gaussian noise with different signal to noise ratios. The mean and variane of the watermark strength fator are estimated from 1 signals inluding musi and English voies, minutes eah. shifts and sign hanges our adaptively. Therefore, depending on the signal, hange in the sign and phase of the gammatone is not neessary when generating the watermark. In that situation, some watermark gammatones are similar in shape and phase to signal gammatones. This is one of the good features of the proposed method whih ontributes to the quality of the watermarked signals. Note that the spikegram removes the inaudible ontent of the signal and uses the gammatone filterbank whih is adapted to the natural sound more than to noise [4]. Therefore, high value oeffiients in the spikegram are robust against mild inaudible distortion and noises. These properties are essential for the proposed deoder when searhing for the watermark gammatones with high value oeffiients. C. Robustness of the proposed method against additive white Gaussian noise For robust watermarking evaluation, we suppose that an additive white Gaussian noise (AWGN) z[n] is added to the watermarked signal y of (). Therefore, (8) beomes r =< y, f p >= m p b+ < z, f p > () in whih, r is the orrelation omputed for the deoding of bit b, z is the AWGN, with mean zero and variane n. As the gammatone kernel f p has zero mean and unit variane, mean and variane of < z, f p > are respetively and n. It is assumed that for eah hannel number, the strength fator m p are samples from a white Gaussian random proess with a speifi mean and variane (Fig.). Assuming that the deorrelation between the watermark strength fator for hannel k and noise z n, the mean m r b and variane for the orrelation term r in () are, m r = m p b and r = n +. Therefore, by onsidering the deoded watermark bit ˆb = sign(< y, f p >), the error probability of the deoder for hannel is as below p k = Pr{ˆb < b =1} = 1 mr erf( p r ) (1) = 1 erf( p k ) ( n + ) Original signal Watermarked minus original signal Time (se) Figure 8. The time domain waveforms for the original signal (blue) and the watermarked minus the original signal (red). The original signal inludes a solo harpsihord instrument sampled at 44.1 khz. Eah gammatone (a spike) in the differene signal (red) indiates the insertion of one watermark bit. Where erf(.) is the omplementary error funtion. In (1), the probability of error is low when we have larger mean and smaller variane for eah hannel s strength fator. In Fig.7, the estimated error probability of the deoder in (1) is plotted versus the embedding hannel number and different levels of signal to noise ratio (SNR). As is seen, for db SNR, the error probability of the first hannels stay below.4. Moreover, an additive noise with lower SNR has a more destroying effet on hannels with higher enter frequenies. D. Designing effiient ditionaries for high quality robust watermarking To design the watermark ditionary, we onsider three onditions. First, based on our empirial results, we do not add watermarks in hannels 5 to 8, beause of the energy greater sensitivity of the ear in this hannel range. Seond, embedding into lower hannels bears more robustness against AWGN attak (Fig.). Lastly, watermark gammatone kernels should be unorrelated. They should be separated at least by one hannel (along the hannel axis) and one effetive length (along the time axis). The final implementation uses hannels, 4,, 11, 1, 15, 17 and 1 for watermark insertion. IV. EXPERIMENTAL SETUP Table II lists the simulation onditions. 1) Test signals: For the quality test, ABC/HR tests [7] are done on piees of 1 seonds of types of audio signals: Pop, Jazz, Rok, Blues, Speeh (in Frenh) and Classi. Titles of the musial piees are listed in Table III. For the robustness test, simulations are done on 1 audio signals, minutes eah (5 hours in total) inluding different musi genres and English voies. Eah signal is sampled at 44.1 khz and has 1-bit wave format. A sample original signal and the original signal minus the watermarked one are plotted in Fig.8. ) Sparse signal representation using PMP and gammatone ditionary: In all experiments, the sparsity of PMP representation is.5. A bank of 5 gammaosine kernels distributed from Hz to khz is implemented to generate spikegrams aording to the onditions given in Table II. ) Resynhonization and Watermark stream generation: The first 1 bits of the embedded bit stream per eah seond is devoted to the synhronization Barker sequene []. This allows synhronization between the deoder and enoder. The other bits in eah seond are () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

7 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 7 Table II THE DEFAULT COMPUTER SIMULATION CONDITIONS Quality test Audio signals of Table III, 1 seonds eah Quality measure Subjetive differene grade (SDG) [7] Robustness test 1 audio signals, minutes eah 5 speeh signals [54], 5 musi signals [55] Signal harateristis sampled at 44.1 khz, quantized at 1 bits Proessing window 45 mse D spikegram 5-hannel gammatone filter bank [17] repeated eah time sample Sparse representation PMP on 1- seond frames, with, 5% sparsity Synhronization ode 1-bit Barker sequene [] Robustness measure Bit Error Rate (BER) Table III EXCERPTS OF 1 SECONDS FROM THESE AUDIO FILES ARE CHOSEN FOR THE ABC/HR [7] LISTENING TESTS, AND THE ODG RESULTS AFTER WATERMARKED ( =.1) Audio Type Title (author or group Name) ODG POP Power of love (Celine Dion) -.41 Classial Symphony No. 5 (Beethoven) -.85 Jazz We ve got (A tribe Called Quest) -.7 Blues Bent Rules (Kiosk) -. Rok Enter Sad man (Metallia) -. Speeh Frenh Voie (A Female) -.1 devoted to the watermark bit stream b. For robustness against ryptographi attaks [4], at eah frame, the watermark bit stream b is multiplied by a Pseudo Noise (PN) sequene [41] p in whih p i { 1, 1}. Thus eah embedded bit in the watermark stream will be b i p i. For synhronization between the enoder and the deoder, we define a 1 ms retangular sliding window, multiply it to the watermarked signal at the deoder and deode the watermark bits from all suessive proessing windows in the sliding window. Note that eah 1 seond sliding window inludes proessing windows. Thus, if the Barker ode is deoded with more than 75 % auray, then the rest of deoding is performed. If the synhronization Barker ode is not aquired, the sliding window is shifted and the same mentioned proedure is ontinued. Note that the signal is not shifted for synhronization, thanks to the time shift invariane property of spikegram representation [4]. Also, beause of the kernel based representation of the spikegram, the proposed method is robust against mild time saling (Appendix A). Moreover, using the mentioned synhronization approah, all 45 ms proessing windows inside eah 1 ms sliding window are also synhronized. A orruption in one proessing window, might result in at most 45ms 441 = 184 orrupted samples. In this ase, to resynhronize the deoder with the embedder, there might be a need to searh for the barker ode with 184 shifting of the sliding window. As the deoding is done in real time, the resynhronization proedure is not omputationally expensive. For resynhronization of ritially time resaled watermarked signals, a searh approah to find the best time resaling ratio, ould be applied as in []. Finally, for the extration of the watermark bits, the deoder uses the key (K) to generate a PN sequene p. Then eah p i multiplies with its bit stream b i p i to find the watermark bit b i (hint: b i p i p i = b i ). If the watermarked signal is shifted Figure. Generation of the embedded watermark stream from the watermark bits and synhronization ode. A 1-bit synhronization Barker ode is inserted into eah seond of the signal. The watermark bits in eah frame are multiplied by a PN sequene. one sample along the time axis, then the required time for the resynhronization equals the deoding time of one input frame (1 seond) minus the preproessing time. V. EXPERIMENTAL EVALUATION As a preproessing task both for the enoder and the deoder, a linear feedbak shift register (LFSR) should be designed to generate a PN sequene. The key K inludes log (M ) bits as the initial state of the LFSR. It also omprises two deimal digits assoiated to the spikegram generation, inluding, number of hannels N and time shifts q (in this work, N = 5, q =1) meaning two 7-bit ASCII odes. Thus 14 bits are devoted to the spikegram parameters. In total, the key inludes 14+log (M ) bits. The spikegram parameters and the initial state of the LFSR are part of the key whih inreases the robustness of the proposed method against ryptographi attaks. A. Quality The embedding hannels, 4,, 11, 1, 15, 17, 1 are seleted for watermark insertion. In the embedding hannels from to 1, eah watermark bit is inserted through three watermark kernels with the highest strength fators. However, for the first two hannels ( and 4), one watermark bit is inserted in eah proessing window. Therefore, the total number of watermark insertions in eah proessing window L P (in seond) is + bits, and we have 1/L P proessing windows per seond. Hene, the total number of embedded bits per seond is M = /L P, while the number of distint embedded watermark bits per seond is 8/L P. Moreover, the total distortion depends on the quality of PMP representation. As the PMP oeffiients for the silent parts are zero, the proposed method also does not insert watermark into the silent parts of the signal. Thus the watermarking payload is alulated for the non-silent parts of the signal. As the PMP oeffiients hange from signal to signal, the robustness and the quality of the algorithm is also ontent dependent. To assess the quality of the watermark signals, ABC/HR listening tests were onduted based on ITU-R BS.111 [7] on segments of 1 seonds of audio signals given in Table () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

8 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 8 SDG SDG POP γ (a) Jazz SDG SDG Classial γ (b) Blues Table IV PARAMETERS USED FOR THE ATTACK SIMULATIONS Attak Condition set up values No Attak - - Resampling Frequeny (KHz), 1 Requantization bits 8, 1 Low-pass filtering th Butterworth, Cut-off.5,.7.5Hz Additive white noise SNR (db), 5, MP ompression Bit rate (kbps) 4, Random ropping Cropping per total length.15%,.5% Amplitude Saling Sale ratio.5, Pith saling Sale ratio.5, 1.5 Time saling Sale ratio.5, 1.5 SDG γ () Rok γ (e) SDG γ (d) Speeh γ (f) Figure 1. SDG as a funtion of embedding perentage fator for 4 minutes of eah audio lip, desribed in Table III, inluding POP (a), Classial (b), Jazz (), Blues (d), Rok (e), Speeh (f). The bottom ends of the bars indiate SDG means and the vertial red line segments represent the 5% onfidene intervals surrounding them. = indiates the original signals and =.1 is also used to generate watermarked signals for the robustness test. III. Experiments are onduted for different embedding perentage ( is the perentage of embeddings per one seond frames and equals M /441. Note that, is different from payload). For the quality measurement, fifteen random subjets (varying from experts to people with no experiene in audio proessing, aged between -5 inluding male and female) partiipated in 5-sale ABC/HR tests by listening to signals using bayerdynami DT5 headsets in a soundproof listening room. In Fig.1, the average subjetive differene grade (SDG) [7] for several types of test signals in respet with the embedding perentage fator is plotted. The tips of the bar harts and the vertial red line segments on them indiate the mean SDG values and their assoiated 5 % onfidene intervals respetively. The SDG indiates the differene between the average quality grade of the watermarked signal (given by listeners) minus the quality grade of the original signal (whih is zero). The SDG is a quality differene grade between zero and -5 and a SDG stritly smaller than -1 means low quality. As is seen from Fig.1, by inreasing the embedding perentage fator, the quality of watermark signals, exept for the lassial audio, degrades and the onfidene interval widens. For the lassial audio, by inreasing the embedding perentage from.5 to.1, the average SDG, rated by listeners, improves. One reason for this is beause the seleted lassial signal in our test inludes no silent and has a noise like spetrum. Thus adding a small amount of watermark noise might not be exatly pereived by the listeners. In all results of Fig.1, when is not higher than.1, the SDG is greater than.5 and onfidene intervals are smaller than.5 (vertial red lines) and ross the line SDG =. Thus for the robustness test, to ensure high quality for the watermark signals, is set to.1. A sample of 1 seonds of eah original signal type and its assoiated watermarked signal an be downloaded at the link: yousof erfani/ The signals in Table III are watermarked with =.1, their objetive differene grade (ODG) results are omputed using the open soure PEAQ [4] test and reported in Table III. B. Payload and Robustness The payload of the method is defined as the number of watermark bits embedded inside eah seond of the host signal while these bits are deoded aurately at the deoder. For the ase of =.1, the number of watermark kernels is M = = 441. Hene, the proessing window length equals L P = /441 = 45mse and the maximum attainable payload for the proposed method is 8/45mse = 177 bps. Fig.11 shows how many embedded watermark bits are perfetly reoverable under different attaks. We use the same number of watermark kernels both for higher and lower bit rates for the results reported on Fig.11. Hene, in the ase of lower bit rates, we use larger repetitive oding fator. Therefore, the watermark deoder for lower bit rates is stronger ompared to the ase of higher bit rate embeddings. The high quality of the average watermarked signals are onfirmed for a payload of 177 bps ( =.1) in Fig.1 and therefore for other bit rates in Fig.11. The Bit Error Rate (BER) of the deoder is defined as the number of erroneously deteted bits at the deoder per all embedded bits. In Fig. 11, to test the robustness of the proposed method, the BER was omputed for a variety of attaks inluding: noise addition, MP ompression, re-sampling, lowpass filtering and re-quantization. The parameter setting for eah attak is given in Table IV. The audio editing tools used in the experiment are CoolEdit.1 [4] (for re-sampling and re-quantization) and Wavepad [44] Sound Editor for MP ompression. Other attaks of Table IV are written in MATLAB [45]. In addition, for all attaks, frame synhronization is performed using the resynhronization approah mentioned in setion IV.. In Fig.11, the most powerful attaks are the MP kbps (with average robust payload of 5.5 bps) and the MP 4 kbps (with average robust payload of 77 bps). The proposed () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

9 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY No Attak (a) 8 bits 1 bits Embedding Bitrate [bps () 4Kbps Kbps (e) (g) (i) 5dB db 15dB (b) khz 1kHz (d) (f) (h).15%.5% (j) Figure 11. Robustness test results. 5 hours of different musi genres and English speeh (1 signals, minutes eah) with onditions presented in Table II are watermarked. The average BER versus the payload (bps) is plotted when the watermarked signals are exposed to the attaks (with onditions listed in Table IV) inluding (a) No attak (b) Additive noise () Re-quanitization (d) Re-sampling (e) MP ompression (f) Pith saling (g) Amplitude saling (h) Low-pass filtering (i) Time saling (j) Random ropping. =.1, sparsity of the signal oeffiient vetor is fored to.5. Note that, the BER is alulated as the ratio of misdeteted watermark bits to the total number of embedded bits. The robust payload for the no-attak, and amplitude saling onditions is 177 bps. method has robustness against low-pass filtering, with a robust payload greater than 8 bps. Also, the robust payload for all other attaks is around 5 bps. Moreover, random ropping is done by setting to zero,.15 or.5 perent of the signals samples, at random plaes, and in every 1-seond frame. By random ropping, we have 55 or 1 orrupted samples per seond. Note that with this definition of random ropping, the signal length is not hanged. Hene, random ropping hanges the spikegram oeffiients obtained by PMP at the deoder. However, the deoder searhes for high peaks in the spikegram whih are robust to mild modifiations (i.e., very low value oeffiients are very prone to ropping). As we inrease the perentage of ropped samples, we expet more degradation on high value oeffiients and hene more BER. BER khz LPF Cropping Time resaling kHz kHz kHz 1 4 Attak strength level Figure 1. The average BER of the deoder under different attak strengths. As an interesting observation, when a 1 db additive white Gaussian noise was added to the signal, we observed that more than perent of the error ours beause of the misdetetion of the loation and type of orret gammatone (gammaosine versus gammasine) at the deoder. This is beause many peak amplitudes in the projetion searh spae might have lose values. This might lead to the misdetetion of the true peak in the attak situations. Under moderate attaks, it is less probable that the sign of the high amplitude peaks be hanged. In Fig.1, the average BER of the deoder is plotted versus the attak strength level for important attaks inluding time resaling (with resaling fators between 1.5 and 1.1), ropping (with orruption between.4% and.%) and LPF (low pass filtering, th order Butterworth with ut-off frequeny between 4 khz and khz). The payload in these experiments equals 1 bps. The experimental onditions are the same as in Table IV. As is seen, even for low pass filtering (ut-off frequeny greater than khz), the BER remains smaller than 1 %. For ropping attak, (.5 perent of samples are randomly put to zero), the BER is small. The strongest attak is time resaling. With a fator of 1.1, we still have more than 1 perent of BER. Note that our approah does not embed watermark into the 5 most high frequeny hannels leading to a robust audio watermarking method against low pass filtering. Moreover, there is a trade-off between the quality of the signal in (1) and the BER of the deoder. In Fig.1 and Fig. 14, respetively, the average (ODG) of the watermarked signals and BER of the deoder are plotted versus the number of gammatone hannels and the density of the oeffiients (when the payload is 1 bps, and there is no attaks). Results are obtained for 5 hours of different musi genres and English speeh (1 signals, minutes eah). Inreasing the number of gammatone hannels and density means using more oeffiients in the sparse representation. Hene, sparsity imposes a trade-off between quality and BER of the deoder. Using more oeffiients for the sparse representation results in more average ODG in Fig. 1 for the watermarked signals and at the same time, it results in more average BER in Fig.14. () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

10 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 1 ODG 1 Density (.4) Density (.) Density(.8) Bit error rate kbps (LPD) kbps (FD) 4 kbps (LPD) 4 kbps (FD) Number of hannels used in the spikegram Figure 1. Average objetive differene grade versus the number of gammatone hannels in the spikegram and density (density=1-sparsity) Channel number BER Density (.4) Density (.) Density(.8) Figure 15. The BER of the proposed deoder under the unified speeh and audio oding (USAC) [] for different bitrates (4 kps and kps) and different modes (linear predition (LPD) and Fourier domain (FD)). The horizontal axis indiates the embedding hannel. As is seen, only for embedding hannels, 4 and 8, the BER is smaller than.. The proessing window length is mse. Also, the BER for LPD mode is slightly larger than the BER for FD mode. The experiments are done on 1 audio signals, inluding different musi genres and English voies, minutes eah Number of hannels used in the spikegram Figure 14. Average BER [%] of the deoder versus the number of gammatone hannels in the spikegram and density. When density inreases (sparsity is redued), the BER and ODG beomes loser. This is beause, for a greater density, we use more PMP iterations. New gammatones found with the last iterations have smaller oeffiients and therefore have less impat on the quality and the BER of the deoder. C. Robustness of the proposed method against USAC In this setion, the robustness of the proposed method is evaluated against a new generation ode alled unified speeh and audio oding (USAC) []. USAC applies linear predition in time domain (LPD) along with residual oding for speeh signal segments and frequeny domain (FD) algorithms for musi segments. Also, it is able to swith between the two modes dynamially in a signal-responsive manner. In Fig.15, the BER results of the proposed deoder under the USAC attak are plotted. As is seen, for hannels,4 and 8, the BER is smaller than.. As the proessing window length for the USAC experiments is mse, hene 5 bits per hannel is embedded in eah seond. Thus the robust payload against USAC is between 5 bps-15 bps. D. Real-time watermark deoding using the proposed method The omputational omplexity of the proposed sheme was analyzed on a personal omputer with an Intel CPU at a frequeny of.5 GHz and DDR memory of 51 MB using a MATLAB 7 ompiler. The deoding proedure inludes omputing projetions and finding a maximum value between several projetions. Our experiments show that the required time for the deoding of one seond of the watermarked signal is 78 mse. Also the preproessing time that inludes reating the gammaosine and gammasine kernels, the pseudo noise, is around. seond. This indiates that, after the initial preproessing stage, the proposed method an be used for realtime deoding of the watermark bits. E. Comparison to reent audio watermarking tehniques Table V ompares the proposed method with several reent methods in terms of robustness against kbps MP attaks. As is seen, the proposed method has a greater robust payload against kbps MP ompression ompared to the mentioned reent methods. In the proposed method, PMP removes the oeffiients assoiated with inaudible ontent of the signal whih are under the masking thresholds and the watermark bits are inserted into high value oeffiients. Therefore, this helps having more robustness against MP attak in whih the pereptual masking is also used. Note that, the onditions of attaks in the aption of Table V are omparable to the onditions desribed in these referenes. Also, to the author s knowledge, this is the first report on the robustness of an audio watermarking system against next generation ode USAC. A BER smaller than 5% is ahieved with an averaged payload omprised between 5 to 15 bps. F. Prior works based on pereptual representation of signals There are several methods whih might have similarities to the proposed approah. In [4], a speeh watermarking method is proposed that uses pith modifiations and quantization index modulation (QIM) for watermark embedding and is robust against de-synhronizaion attaks. Although [4] is robust against low bit rates speeh odes suh as AMR () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

11 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 11 Table V COMPARISON TO RECENT METHODS. THE AVERAGE RESULTS FOR 5 HOURS OF DIFFERENT MUSIC GENRES AND ENGLISH VOICES ARE COMPARED TO THE AVERAGE REPORTED RESULTS. DURING THE ATTACKS, THE WATERMARKED SIGNALS ARE MODIFIED AS FOLLOWS: FOR THE RANDOM CROPPING, THE NUMBER OF CROPPING PER TOTAL LENGTH EQUALS.15%. FOR RE-SAMPLING, THE SIGNALS ARE RE-SAMPLED AT.5 KHZ. FOR RE-QUANTIZATION, THE SIGNALS ARE RE-QUANTIZED AT 8 BITS. FOR PITCH AND AMPLITUDE SCALING, THE PITCH AND THE AMPLITUDE OF THE SIGNALS ARE SCALED WITH THE.5 AND.5- SCALING RATIOS, RESPECTIVELY. FOR LPF, SIGNALS ARE LOW-PASS FILTERED WITH CUT-OFF FREQUENCY EQUALS TO 11.5 KHZ. NMMEANS NOT MENTIONED. Method Payload MP(kbps), Cropping AWGN Resamp- Requantiz- Pith Amplitude LPF (bps) BER SNR =db ling ation, 8bits Saling Saling Bhat [5] 45.,..... NM NM. Khaldi [] 5., NM NM. Yeo [51] 1,. NM NM.. NM. NM Shaoquan [47] 17,.7. <... NM NM NM Zhang [5] 4.7 4,. NM NM.4.4 Nishimura [8] 1 4,. NM Our Method 5.5, ode, no payload results are given for audio signals. In [], after empirial mode deomposition of the audio signals, the watermarking embedding is done on the extrema of last IMF (intrinsi mode funtion) using QIM. Table V onfirms that our approah outperforms this method in terms of robustness against kbps Mp ompression. In [47], the watermark is inserted into the wavelet oeffiients using QIM. Also, in [48], the spread spetrum (SS) is applied on MDCT oeffiients along with psyhoaousti masking for single-bit watermarking. Long duration audio frames are used along with epstral filtering at the deoder. There are several differenes between our approah and the above-mentioned transform domain methods. First, we evaluate the effiieny of a new transform, alled spikegram, for robust watermarking. We introdue a new framework for audio watermarking alled two-ditionary approah. The enoder and the deoder searh in a orrelation spae to find the maximum projetion (minimum signal interferene). Seond, the proposed approah is a phase embedding method on gammatone kernels with uses of masking. Gammatone kernels are the building bloks to represent the audio signal. Third, the proposed method takes are of effiient embedding into non-masked, high value oeffiients whih make it robust against attaks suh as universal speeh and audio ode (4 kbps USAC) [] and kbps MP ompression. Also, thanks to the use of PMP, by removing many oeffiients under the masks, the signal interferene is further redued at the deoder. G. Robustness against analogue hole experiments Here, the robustness of the proposed method against analogue hole is evaluated in a preliminary experiment. In Fig. 1, the BER of the proposed method against a simulated real room are given using the image soure method for modeling the room impulse response (RIR) [4], [1]. We embed one bit of watermark in eah seond of the host signal (1 bps payload). We use an open soure MATLAB ode [], [1] to simulate the room impulse responses. A asade of RIR of a 4m 4m 4m room with a db additive white Gaussian noise is onsidered as the simulated room impulse response. Also, only one mirophone and loud speaker are modeled. The experiments are done for three distanes d between the loudspeaker and the mirophone inluding d = 1, and BER Robustness under room bakground noise 1 meter meter meter The distane between the loudspeaker and mirophone Figure 1. The BER of the proposed method against a simulated analogue hole in ombination with a db additive noise. meters (d denotes the distane between the mirophone and the speaker). For watermark embedding, all the bits in eah 1-seond frames are generated using a pseudo random number generator. A spread spetrum (SS) orrelation deoder is used. Hene, the 1-seond sliding window is shifted sample by sample until the orrelation of the SS deoder is above.75. Then, the watermark bit is deoded as the sign of the SS orrelation. Results are reported in Fig.1. From Fig.1, the deoder an be robust against the analogue hole, when d =1 meter, with a BER lower than 5 %. While for d =or meters, the BER inreases sharply. The experiments are done on the 5 signals presented in Table III. VI. CONCLUSION A new tehnique based on a spikegram representation of the aoustial signal and on the use of two ditionaries was proposed. Gammatone kernels along with pereptual mathing pursuit are used for spikegram representation. To ahieve the highest robustness, the enoder selets the best kernels that will provide the maximum strength fators at the deoder and embeds the watermark bits into the phase of the found kernels. Results show better performane of the proposed method against kbps MP ompression with a robust payload of 5.5 bps ompared to several reent tehniques. Furthermore, () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

12 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY 1 for the first time, we report robustness result against USAC (unified speeh and audio oding) whih uses a new standard for speeh and audio oding. It is observed that the BER is still smaller than 5% for a payload omprised between 5 and 15 bps. The approah is versatile for a large range of appliations thanks to the adaptive nature of the algorithm (adaptive pereptive masking and adaptive seletion of the kernels) and to the ombination with well established algorithms oming from the watermarking ommunity. It has fair performane when ompared with the state of the art. The researh in this area is still in its infany (spikegrams for watermarking) and there is plenty of room for improvements in future works. Moreover, we showed that the approah an be used for realtime watermark deoding thanks to the use of a projetionorrelation based deoder. In addition, two-ditionary method ould be investigated for image watermarking. APPENDIX A ROBUSTNESS AGAINST TIME RESCALING ATTACK In the ontext of watermark deoding, we first ompute the orrelation between the watermarked signal with a sliding gammasine or gammaosine andidate for the given hannel j and different time samples k =1,,..,N P where N P is the number of time samples in the proessing window. The deoded watermark bit is the sign of the peak orrelation, i.e, g opt = argmax gj,k ( < y[ n],g j,k > ), ˆb = sign(< y[ n],g opt >), where <y[ n],g j,k [n] >=< y[n],g j,k [ n ] >, k =1,,..,N P (11) When is lose to one, the position of peaks in (11) do not hange ompared to the no-attak situation. This results in robustness against mild time resaling for single bit watermarking. In multibit watermarking, the watermark gammatone is inserted in odd hannel numbers. Thus, when is lose to one, odd hannel numbers have weak orrelations. This means that time resaling, with a small resaling fator, does not affet the deoder of the multi-bit watermarking. Figure 1 reports the BER for between 1.5 and 1.1. ACKNOWLEDGMENT This work was initially funded by CRC and then by NSERC and Université de Sherbrooke. Many thanks to Hossein Najaf- Zadeh for the PMP ode, to Malolm Slaney for the auditory modeling toolbox, to Frédérik Lavoie and Roh Lefebvre for managing the listening tests, to Philippe Gournay for providing the USAC ode, to Simon Brodeur and Sean Wood for optimizing the PMP ode and to the partiipants of the listening test. This researh was enabled in part by support provided by Calul Quebe ( and Compute Canada ( Many thanks to the reviewers for their very onstrutive omments that improved the paper. REFERENCES [1] S.E. Siwek, True Cost of Reorded Musi Piray to the U.S. Eonomy, Published by IPI, Aug. 7. [] I. Cox, M. Miller, J. Bloom, J. Fridrih and T. Kalker, Digital Watermarking and Steganography, San Franiso, USA: Morgan Kaufmann Publishers In., nd ed., 7. [] M. Steinebah and J. Dittmann, Watermarking-based digital audio data authentiation, Eurasip J. Appl. Signal Proess., pp ,. [4] A. Boho, G. Van Wallendael, A. Dooms, J. De Cok, et al., End-To- End Seurity for Video Distribution, IEEE Signal Proessing Magazine, vol., no., pp.7-17, 1. [5] S. Majumder, K.J. Devi, S.K. Sarkar, Singular value deomposition and wavelet-based iris biometri watermarking, IET Biometris, vol., no.1, pp.1-7, 1. [] M. Arnold, X. Chen, P. Baum, U. Gries, and G. Dorr, A phase-based audio watermarking system robust to aousti path propagation, IEEE Trans. on IFS, vol., no., pp , 14. [7] M. Unoki, R. Miyauhi, Robust, blindly-detetable, and semi-reversible tehnique of audio watermarking based on ohlear delay, IEICE Trans. on Inf. Syst. vol.e8-d, no.1, pp.8-48, 15. [8] R. Nishimura, Audio watermarking using spatial masking and ambisonis, IEEE Trans. on ASLP, vol., no., pp.41-4, 1. [] G. Hua, J. Goh, and V. L. L. Thing, Time-spread eho-based audio watermarking with optimized impereptibility and robustness, IEEE Trans. ASLP, vol., no., pp.7-, 15. [1] G. Hua, J. Goh, and V. L. L. Thing, Cepstral analysis for the appliation of eho-based audio watermark detetion, IEEE Trans. on IFS, vol.1, no., pp , 15. [11] Y. Xiang, I. Natgunanathan, D. Peng, W. Zhou, S. Yu, A dual-hannel time-spread eho method for audio watermarking, IEEE Trans. IFS, vol.7, no., pp. 8-, 1. [1] Y. Xiang, I. Natgunanathan, S. Guo, W. Zhou, and S. Nahavandi, Pathwork-based audio watermarking method robust to desynhronization attaks, IEEE Trans. ASLP, vol., no., pp , 14. [1] C. M. Pun and X. C. Yuan, Robust segments detetor for desynhronization resilient audio watermarking, IEEE Trans. ASLP., vol.1, no.11, pp , 1. [14] B. Lei, I. Y. Soon, and E. L. Tan, Robust SVD-based audio watermarking sheme with differential evolution optimization, IEEE Trans. ASLP, vol.1, no.11, pp.8-77, 1. [15] D. Megas, J. Serra-Ruiz, M. Fallahpour, Effiient self-synhronised blind audio watermarking system based on time domain and FFT amplitude modifiation, Signal Proessing, vol., no.1, pp.78-, 1. [1] N. M. Ngo, M. Unoki, Robust and reliable audio watermarking based on phase oding, IEEE ICASSP, pp.45-4, 15. [17] R.D. Patterson, B.C.J. Moore, Auditory filters and exitation patterns as representations of frequeny resolution, Aademi Press Ltd., Frequeny Seletivity in Hearing, London, pp.1-177, 187. [18] M. Slaney, An Effiient Implementation of the Patterson-Holdsworth Auditory Filter Bank, Apple Computer Tehnial Report 5, 1. [1] N. Nikolaidis, I. Pitas, Benhmarking of Watermarking Algorithms, in Book: Intelligent Watermarking Tehniques, World Sientifi Press, pp , 4. [] S.M. Valiollahzadeh, M. Nazari, M. Babaie-Zadeh, C. Jutten, A new approah in deomposition over multiple-overomplete ditionaries with appliation to image inpainting, Mahine Learning for Signal Proessing, IEEE MLSP, pp.1-,. [1] Ch. H. Son, H. Choo, Watermark detetion from lustered halftone dots via learned ditionary, Signal Proessing, vol.1, pp.77-84, 14. [] A. Adler., V. Emiya, M.G. Jafari, M. Elad, R. Gribonval, M.D. Plumbley, Audio Inpainting, IEEE Trans. ASLP, vol., no., pp.-, 1. [] C. Fevotte, L. Daudet, S.J. Godsill, B. Torresani, Sparse Regression with Strutured Priors: Appliation to Audio Denoising, IEEE ICASSP, pp.57-,. [4] E. Smith, M. S. Lewiki, Effiient Coding of Time-Relative Struture Using Spikes, Neural Computation, vol.17, no.1 pp.1-45, 5. [5] R. Pihevar, H. Najaf-Zadeh, L. Thibault, H. Lahdili, Auditory-inspired sparse representation of audio signals, Speeh Communiation, vol.5, no.5, pp.4-57, 11. [] K. Khaldi, A.O. Boudraa, Audio Watermarking Via EMD, IEEE Trans. ASLP, vol.1, no., pp.75-8, 1. [7] S. Quakenbush, MPEG Unified Speeh and Audio Coding, IEEE MultiMedia, vol., no., pp. 7-78, 1. [8] Y. Yamamoto, T. Chinen and M. Nishiguhi, A new bandwidth extension tehnology for MPEG Unified Speeh and Audio Coding, 1 IEEE ICASSP, pp.5-57, 1. () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.

13 This artile has been aepted for publiation in a future issue of this journal, but has not been fully edited. Content may hange prior to final publiation. Citation information: DOI 1.11/TIFS.1.4, IEEE JOURNAL OF, VOL.?, NO.?, MONTH YYYY [] M. Neuendorf, P. Gournay, M. Multrus, J. Leomte, B. Bessette, R. Geige, S. Bayer, G. Fuhs, J. Hilpert, N. Rettelbah, R. salami, G. Shuller, R. Lefebvre, B. Grill, Unified speeh and audio oding sheme for high quality at low bit rates, IEEE ICASSP, pp.1-4,. [] [1] [4] E. Lehmann, A. Johansson, and S. Nordholm, Reverberation-time predition method for room impulse responses simulated with the imagesoure model, IEEE Workshop on Appliations of Signal Proessing to Audio and Aoustis (WASPAA 7), pp.15-1, New Paltz, USA, 7. [] Th. Blumensath, Mike E. Davies, Iterative hard thresholding for ompressed sensing, Applied and Computational Harmoni Analysis, vol.7, no. pp.5-74,. [] Joel A. Tropp, Anna C. Gilbert, Signal reovery from random measurements via orthogonal mathing pursuit, IEEE Trans. information theory vol.5, no.1, pp , 7. [4] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Ekstein, Distributed Optimization and Statistial Learning via the Alternating Diretion Method of Multipliers, Foundations and Trends in Mahine Learning, vol., no.1, pp.11, 11. [5] H. Najaf-Zadeh, R. Pihevar, H. Lahdili, and L. Thibault, Pereptual mathing pursuit for audio oding, Audio Engineering Soiety Convention 14, vol.5, 8. [] S. Mallat and Z. Zhang, Mathing pursuit with time-frequeny ditionaries, IEEE Trans. on SP, vol.41, pp.7-415, 1. [7] ITU-R BS.111-1, Methods for the Subjetive Assessment of Small Impairments in Audio Systems Inluding Multihannel Sound Systems, Geneva, Switzerland: Int. Teleomm. Union, 17. [8] R. Pihevar, H. Najaf-Zadeh, L.Thibault, New Trends in BiologiallyInspired Audio Coding, Reent Advanes in Signal Proessing, In-Teh Pub.,. [] P. Borwein, J. M. Mossinghoff, Barker sequenes and flat polynomials, In James MKee; Chris Smyth. Number Theory and Polynomials. LMS Leture Notes 5. Cambridge University Press., pp.71-88, 8. [4] S. Voloshynovskiy, S. Pereira, T. Pun, J. J. Eggers, J. K. Su, Attaks on digital watermarks: Classifiation, estimation-based attaks and benhmarks, IEEE Communiations Magazine, vol., no.8, pp.118-1, 1. [41] A. Klein, Stream Ciphers, Springer-Verlag, 1. [4] P. Kabal, An Examination and Interpretation of ITU-R BS.187: Pereptual Evaluation of Audio Quality, TSP Lab Tehnial Report, Dept. Eletrial and Computer Engineering, MGill University, May. [4] [44] [45] [4] D. J. Coumou and G. Sharma, Insertion, Deletion Codes With FeatureBased Embedding: A New Paradigm for Watermark Synhronizastion With Appliations to Speeh Watermarking, IEEE Trans. on IFS, vol., no., pp , 8. [47] Shaoquan Wu, Jiwu Huang, Daren Huang and Y. Q. Shi, Effiiently self-synhronized audio watermarking for assured audio data transmission, IEEE Trans. on Broadasting, vol. 51, no. 1, pp. -7, Marh 5. doi: 1.11/TBC [48] D. Kirovski and H. S. Malvar, Spread-spetrum watermarking of audio signals, IEEE Trans. on SP, vol.51, no.4, pp.1-1, Apr. [4] J. Allen and D. Berkley, Image method for effiiently simulating smallroom aoustis, Journal of the Aoustial Soiety of Ameria, vol.5, no.4, pp.4-5, April 17. [5] V. Bhat, I. Sengupta, A. Das, An adaptive audio watermarking based on the singular value deomposition in the wavelet domain, Elesevier Journal on DSP, vol., no., pp , 1. [51] I.K. Yeo, H.J. Kim, Modified pathwork algorithm: A novel audio watermarking sheme, IEEE Trans. on SAP, vol.11, no.4, pp.81-8,. [5] P. Zhang, SH.Z. Xu, H.Z. Yang, Robust Audio Watermarking Based on Extended Improved Spread Spetrum with Pereptual Masking, International Journal of Fuzzy Systems, vol.14, no., pp.8-5, 1. [5] ] G. Hua, L. Zhao, and G. Bi, A seure data hiding system based on overomplete ditionary partitioning, in Pro. IEEE International Conferene on Advaned Intelligent Mehatronis (AIM 1), in press, Banff, 1. [54] [55] 1 Yousof Erfani reeived his B.S. degree in eletrial engineering (eletronis) and his M.S. degree in eletrial engineering (ommuniation) both from Sharif university of tehnology, Tehran, Iran, in and 4, respetively and the Ph.D. degree from Universite de Sherbrooke, Sherbrooke, QC, Canada, in eletrial and omputer engineering (signal proessing) in 1. He has reently joined the auditory engineering lab of MMaster university, ON, Canada, as a postdotroal fellow, where he works on improving auditory models. His researh interests inlude mahine learning, neural signal proessing, bio-inspired sparse representation, deep learning, Bayesian inferene, watermarking and auditory models. Ramin Pihevar was born in 174, in Paris, Frane. He reeived his bahelor of siene degree in eletrial engineering (eletronis) in 1 and his master of siene in eletrial engineering (teleommuniation systems) in 1, both in Tehran, Iran. He reeived his Ph.D. in eletrial and omputer engineering from Universit de Sherbrooke, Qube, Canada in 4. In 1 and he performed two summer internships at Ohio State University (USA) and at the University of Grenoble (Frane), respetively. From 4 to, he was a postdotoral researh assoiate in the omputational neurosiene and signal proessing laboratory at the University of Sherbrooke under an NSERC (Natural Sienes and Engineering Counil of Canada) Idea to Innovation (II) grant. He was a researh sientist at the Communiations Researh Center (CRC), Ottawa, Canada from to 1. He is urrently with Apple In. (California). His domains of interest are signal and speeh proessing, neural networks with emphasis on bio-inspired neurons, speeh reognition, audio and speeh oding, and sparse tehniques. Jean Rouat (S8M88SM) reeived the M.S. degree in physis from Universit de Bretagne, Brest, Frane in 181, an E. & E. M.S.A. degree in speeh oding and speeh reognition from Universit de Sherbrooke, Q., Canada in 184 and an E. & E. Ph.D. in ognitive and statistial speeh reognition jointly with Universit de Sherbrooke and MGill University, Montral, Q., Canada in 188. His post-do has been in psyhoaoustis with the MRC, App. Psyh. Unit, Cambridge, UK and in eletrophysiology with the Institute of physiology, Lausanne, Switzerland. He is urrently with Universit de Sherbrooke, where he founded the Computational Neurosiene and Intelligent Signal Proessing Researh group (NECOTIS). He is also an Adjunt professor with the department of Biologial Sienes, Universit de Montral, and full member of the Centre for Interdisiplinary Researh in Musi Media and Tehnology (CIRMMT), housed at the Shulih Shool of Musi at MGill University, Montral, Q., Canada. His translational researh links neurosiene and engineering for the reation of new tehnologies and appliations with the integration of a better understanding and integration of multimodal representations (vision & audition). Information hidding in multimedia signals, development of hardware low power onsumption neural proessing units (NPU) for a sustainable development, interations with artists for multimedia and musial reations are examples of transfers that he leads based on the knowledge he gains from neurosiene and his knowledge of visual & auditory systems. He is leading mahing learning funded projets to develop sensory substitution and intelligent devies. He is also the oordinator of the interdisiplinary IGLU CHIST-ERA european onsortium (IGLU - Interative Grounded Language Understanding) for the development of an intelligent agent that learns through multimodal grounded interations. () 1 Crown Copyright. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by ing pubs-permissions@ieee.org.