BLACKBOARD SYSTEM AND TOP-DOWN PROCESSING FOR THE TRANSCRIPTION OF SIMPLE POLYPHONIC MUSIC. Juan Pablo Bello and Mark Sandler

Transcription

1 BLACKBOARD SYSTEM AND TOP-DOWN PROCESSING FOR THE TRANSCRIPTION OF SIMPLE POLYPHONIC MUSIC Juan Pablo Bello and Mak Sandle Depatment of Electonic Engineeing, King s College London, Stand, London WC2R 2LS, UK uan.bello_coea@kcl.ac.uk ABSTRACT A system is poposed to pefom the automatic music tansciption of simple polyphonic tacks using top-down pocessing. It is composed of a blackboad system of thee hieachical levels, eceiving its input fom a segmentation outine in the fom of an aveaged STFT matix. The blackboad contains a hypotheses database, a schedule and knowledge souces, one of which is a neual netwok chod ecognise with the ability to econfigue the opeation of the system, allowing it to output moe than one note hypothesis at a time. The basic implementation is explained, and some examples ae povided to illustate the pefomance of the system. The weaknesses of the cuent implementation ae shown and next steps fo futhe development of the system ae defined. 1. INTRODUCTION Musical tansciption of audio data is the pocess of taking a sequence of digital data coesponding to the sound wavefom and extacting fom it the symbolic infomation elated to the high-level musical stuctues that might be seen on a scoe [1]. The scoe and the ochesta ae the pats that can be defined in a musical tack [2] and in an academic music epesentation, ust the fome can be descibed. The pupose of the pesent wok is to automatically extact scoe featues fom monophonic and simple polyphonic music tacks (monotimbic music with chods), using a computational easoning model called blackboad system [3][4] and combining top-down (pedictiondiven) pocessing with the bottom-up (data-diven) techniques aleady implemented in [5] Blackboad system The blackboad system is a elatively complex poblem-solving model pescibing the oganisation of knowledge and data, and the poblem-solving behaviou within the oveall oganisation [4]. It eceives its name fom the metapho of a goup of expets tying to solve a poblem plotted on a blackboad, each expet ust act when he/his specific aea of expetise is equied in the developing of the solution. In contast to the usual paadigm of signal pocessing algoithms, whee algoithms ae descibed by data flowchats showing the pogess of infomation along chains of modules [6], the achitectue of the blackboad system is oppotunistic, choosing the specific module needed fo the development of the solution at each time step. Due to its open achitectue diffeent knowledge can be easily integated into the system, allowing the utilisation of vaious aeas of expetise. The basic stuctue of a blackboad system is depicted in figue 1 and consists of thee fundamental pats: the blackboad: global database whee the hypotheses ae poposed and developed, which inteact with all the modules pesent in the system; the schedule o oppotunistic contol system: detemines how the hypotheses ae developed and by who; and the knowledge souces o expets of the system: modules that execute the actions intended to develop the hypotheses pesent in the blackboad. Schedule Knowledge souces Blackboad o hypotheses database INPUT Figue 1: The basic stuctue of a blackboad system The system opeates in time steps, executing one action at a time. The schedule pioitises within the existing list of knowledge souces, detemining the ode in which these actions ae executed. Each knowledge souce consists of a sot of if/then (pecondition/action) pai. When the pecondition of a cetain knowledge souce is satisfied, the action descibed in its pogamming body is executed, placing its output in the blackboad. These knowledge souces can pefom diffeent kinds of activities, such as detecting and emoving unsuppoted hypothesis fom the blackboad o stimulating the seach fo hamonics of a given note hypothesis. Thee ae seveal implementations of blackboad systems in automatic music tansciption [3][7][8], howeve pat of the knowledge a human being use to tanscibe music is based on his/he expeience heaing music files and the inheent stuctues pesent in these, and in those systems this knowledge is ignoed. DAFX-1

2 As [6] specifies, the stuctue of the blackboad makes little distinction between explanatoy and pedictive opeations; hypotheses geneated fo modules of infeence can econfigue the opeation of the system and bias the seach within the solution space Top-Down and Bottom-up Pocessing In bottom-up pocessing, the infomation flows fom the lowlevel stage, that of the analysis of the aw signal, to the highest level epesentation in the system, in ou case that of the note hypotheses. In this technique, the system does not know anything about the obect of the analysis pevious to the opeation, and the esult depends on the evolution of the data in its unidiectional flow though the hieachy of the pocesso. This appoach is also called data-diven pocessing. In contast, the appoach when the diffeent levels of the system ae detemined by pedictive models of the analysed obect o by pevious knowledge of the natue of the data is known as top-down o pediction-diven pocessing [9]. Despite the fact that top-down pocessing is believed to take place in human peception, most of the systems implemented until now ae based on bottom-up pocessing, and ust in the last yeas the implementation of pedictive pocessing to eceate these peceptual tasks had become a common choice between eseaches of this field [1][6][9][10]. On tasks such as automatic music tansciption, the inflexibility of bottom-up systems made them unable to achieve esults in a geneal context, outlining the need fo the pediction-diven appoach to be used. In this wok, the top-down pocessing is achieved though the implementation of a connectionist system. This kind of system consists of many pimitive cells (units), which ae woking in paallel and ae connected via diected links. Though these links, activation pattens ae distibuted imitating the basic mechanism of the human bain, which is why these models ae also called neual netwoks [11]. Knowledge is usually distibuted thoughout the net and stoed in the stuctue of the topology and the weights of the links; the netwoks ae oganized by automatic taining methods, which help the development of specific applications. If adequately tained, these netwoks can acquie the expeience to make decisions in vey specific poblems. Hee, the poblem is to identify the pesence of a chod in a given segment of a music signal. As extensive documentation of neual netwoks is available, no futhe explanation of this topic will be developed hee, ust the basics of the implemented system ae explained in section Segmentation 2. IMPLEMENTATION Just a bief explanation of the system s font end is descibed hee. The onset detection aims to evaluate the time instant when a new note is played in a sound file. Analysing the unning spectum of a sound it is possible to notice that when a new event occus the high fequency content is inceased [12][13]. The measue of the high fequency content (HFC) is given by: 2 ( X ( k) k) ( N / 2) + 1 HFC = (Eq.1) k = 2 Whee X(k) is the FFT aay of the audio signal and N is its length, while k is used as a linea facto to emphasize the high fequencies in the fame. The HFC and the Enegy (E) calculated on each fame ae used to build the detection function: DF HFC = HFC 1 HFC E (Eq.2) Figue 2: The oiginal signal (a teno sax iff), the detection function and the estimated onsets and offsets (cicled) ove the signal. As can be seen in figue 2, this function shows shap peaks in the instant when the tansient occus. A citeia based on the slope of these peaks was used to detemine the onset s time. Afte this pocess, the segmentation is pefomed aveaging the signal s STFT between onsets. This is used as the input of the blackboad system Blackboad Implementation The Blackboad system s achitectue is based on that of Matin s implementation [3] and is shown in figue 3. At the lowe level, the system eceives the aveaged STFT of the signal and identifies the peaks of the spectum. Of this goup ust the peaks highe than an amplitude theshold ae consideed to build a Tacks matix, containing the magnitude and fequency of each. This infomation is fed to the database and exposed to the evaluation of the knowledge souces () to poduce new hypotheses. Thee ae thee diffeent levels of infomation pesent on the database: tacks, patials and notes. The tacks infomation is automatically povided at the beginning of the system opeation, howeve the notes and patials infomation ae the poduct of the knowledge souces inteaction with the database. It is the main task of the Schedule to detemine the need fo a specific kind of infomation and to activate the coesponding knowledge souce. In the pesent system a table of peconditions is evaluated at each DAFX-2

3 time step and a ating is given to each knowledge souce detemining the ode in which these will opeate. Output notes patials tacks Actions STFT Peak selection Segmentation Figue 3: The blackboad achitectue of this implementation. At the tacks level, all the emaining peaks of the STFT have an equal chance of becoming notes, but as the opeation of the system goes fowad and new hypotheses ae poduced and evaluated by the, atings ae given to naow the seach fo musical notes in the spectum. In the case of the patials, the ating is based on the magnitude of the neaest peak (within a specific ange) to the ideal fequency of the hypothesis. Fo notes, ating is based on the pesence and magnitude of peaks coesponding to the ideal patials this note should have [14]. All this infomation is stoed in a matix called Hypotheses Neual Netwok Implementation Schedule In the neual netwok implemented, the infomation flows in one way fom input to output. Thee is no feedback, which means that the output of any laye does not affect that same laye. This type of netwok is known as feed-fowad. The stuctue of this implementation consists of thee layes: an input, an output and a hidden laye. The activation function implemented fo all the neuons is the sigmoid tansfe function. The leaning is supevised. Taining a feed-fowad neual netwok with supevised leaning consists of the following pocedue [11]: 1. An input patten is pesented to the netwok. The input is then popagated fowad in the net until activation eaches the output laye. This is called the fowad popagation phase. 2. The output of the output laye is then compaed with the teaching input. The eo, i.e. the diffeence δ between the output o and the teaching input t of a taget output unit, is then used togethe with the output o i of the souce unit i to compute the necessay changes of the link w i. To compute the deltas of inne units fo which no input is available, (units of hidden layes) the deltas of the following laye, which ae aleady computed, ae used in a fomula given below. In this way the eos (deltas) ae popagated backwad, so this phase is called backwad popagation. Peconditions Signal 3. In this implementation offline leaning is used, which means that the weights changes ω i ae cumulated fo all pattens in the taining file and the sum of all changes is applied afte one full cycle (epoch) though the taining patten file. This is also known as batch leaning. Hee, the input patten consists of a 256 points spectogam of a piano signal s segment (eithe a note o a chod), pat of the batch of samples coveing five octaves of the instument. The taget output is ust epesented fo the absence 0 o pesence 1 of a chod in the sample. The weight changes wee calculated using the backpopagation weight update ule, also called genealized delta-ule, which eads as follows [11]: ω = ηδ ο i (Eq.3) whee, δ i = f ( net )( t ο ) (Eq.4) if unit is an output unit o δ = f ( net ) k δ ω k k (Eq.5) if unit is a hidden unit, whee: η leaning facto eta (a constant) δ eo (diffeence between the eal output and the teaching input) of unit t teaching input of unit o i output of the peceding unit i i index of a pedecesso to the cuent unit with link w i fom i to index of the cuent unit k index of a successo to the cuent unit with link w k fom to k 2.4. Neual Netwok Inteaction with the Blackboad The netwok is tained offline to obtain a set of paametes adequate to the task equied, in this case the ecognition of the pesence of a chod in a spectogam. When the oveall system is unning, the netwok eceives as an input the same STFT data the blackboad system analyses. In the oiginal blackboad s pocess, ust the note hypotheses with ating bigge than a cut-off theshold emained as valid hypotheses [5], in this vesion of the system, the output of the neual netwok changes the pefomance of the system allowing moe than one note hypothesis to suvive if necessay. As illustated in figue 4, this pocess eshapes the Hypotheses matix, changing its stuctue and adding a new level of infomation in the system: chods. Due to this, the knowledge souces that inteact with the mentioned matix ae stuctually modified and uged to link stong note hypotheses pesent in the blackboad to poduce hypothetic chods. As the schedule detects the need fo this new kind of infomation, the pioity list of opeation is econfigued to favou the modified knowledge souces. Howeve it is possible that even with the neual netwok poposing the pesence of a chod in the segment, ust a single note is output by the system due to the lack of multiple stong ated note hypotheses in the system. In this fist appoach, ust chods of two o thee notes can be identified by the system. DAFX-3

4 oiginal notes played in that segment. The othe twelve notes of the piece and the last chod wee coectly identified by the system. STFT Chods Neual netwok chod ecognise Notes Patials Tacks Figue 5: Example of automatic tansciption of a piano iff. Figue 4: The Inteaction between the blackboad and the neual netwok. Afte the selection of hypotheses is made, each of the fequencies obtained is ounded towads the neaest musical fequency (ideal fequency coesponding to a musical note) and intoduced into a scoe file. This is a text file witten in CSOUND language [15], which can be compiled and endeed with an ochesta file (a sine wave sound fo these expeiments), obtaining an audible epesentation of the oiginal signal. The MIDI numbe coesponding to each fequency is calculated as well [16][17], and the output is epesented in the fom of a piano oll, popotioning a gaphical epesentation of the poposed musical events detected by the system. 3. EXAMPLES In the fist example, illustated in figue 5, a piano iff is plotted, consisting on a succession of fou notes (C 5 D 5 E 5 F 5 ) followed by a C mao chod (C 5 E 5 G 5 ). The notes and the chods ae ecognised successfully by the system. This example is intended ust to show the main capabilities of the cuent system. Notice that the notes and silences ae well diffeentiated and the netwok identified the pesence of a chod elated with the last onset, causing the blackboad to output the thee highe ated hypotheses of the segment. The second example shown in figue 6 epesents a fou ba section of a piano song, including fou chods. Seveal mistakes ae made in the tansciption of the notes of thee chods (the fist thee of the figue), whee coect note hypotheses wee discaded by the system in favou of thei lowe octave equivalents. This octave eo was detected as well in the note befoe the last chod, whee the note C 6 was selected ove the coect C 5. Anothe eo in the tansciption is elated to the non-detection of an onset in the seventh second of the song causing a wong segmentation of the piece. The spectogam of this segment was identified as a chod by the neual netwok, pobably due to the pesence of two stong fundamental pitches in the time window aveaged. As can be seen in the figue 6 a nonexistent chod was plotted between the times of and seconds, containing both the Figue 6: Example of automatic tansciption of a simple polyphonic piano song of fou measues. 4. CONCLUSIONS AND NEXT STEPS Moe tests needs to be pefomed to fully evaluate the cuent system. Howeve, fom the peliminay tests some stong limitations can be detected in the system than need to be ovecome in futue implementations. The incoect selection of notes in diffeent octaves than played, is a ecuent mistake in diffeent tests pefomed. This is due to the fact that same notes in diffeent octaves have the same hamonic content and the knowledge souces of the blackboad base thei selection of notes on the patials infomation. This is fully dependent on the ating function implemented, specifically of the weight used fo the fundamental, even and odd patials. This woks as a timbe model and modifying it can make the system moe obust in identifying octaves fo specific instuments (piano in this case) but will affect the geneality of the system. Moe knowledge souces could be implemented that use musical knowledge to discen this type of stuctues. The onset detection is a limitation as well in the pefomance of the system. It does not wok unifomly in all the fequency ange, and with diffeent kind of instuments. It elies on the enegy content of the signal, which makes it especially weak on facing DAFX-4

5 expessive featues in music (legato, glissando, temolo, vibato, etc). A moe successful appoach would be to ely on pitch changes, but due to the computational load that the blackboad s opeation imply this is not feasible with the cuent spectal analysis. Moe efficient signal pocessing methods will help to impove the elation between computational time and onset eliability. Analysis with wavelets, Multiesolution Fouie Tansfom (MFT) [18]o the log-lag coelogam [6][7] ae cuently studied fo futue systems. The achitectue of the blackboad needs to be modified, incopoating dynamic stuctues to handle diffeent sized hypotheses, e.g. chods of moe than thee notes. Also, the taining space of the netwok has to be expanded to all the octaves of the piano. In geneal the system elies on heuistic paametes fo its opeation. This makes it less geneal, specifically in the handling of diffeent timbes and diffeent fequency anges. Tests ae being pefomed to detemine these constaints and to develop moe geneal ules to make the system less paamete dependant. The flexible achitectue of the blackboad is its stong asset. As a fist appoach, the esults depicted hee ae vey encouaging showing that futhe development of these ideas could be the way fo moe obust and geneal esults. Cuently we ae woking on these points mentioned to impove the system. [13] P. Masi and A. Bateman. Impoved Modelling of Attack Tansient in Music Analysis-Resynthesis. Univesity of Bistol [14] Randall Davis, Buce Buchanan, and Edwad Shotliffe. Poduction Rules as a epesentation fo a Knowledge- Based Consultation Pogam. Atificial Intelligence, 8:15-45, [15] Bay Vecoe. CSOUND A Manual fo the Audio Pocessing System and Suppoting Pogams with Tutoials. Media Lab, M.I.T, Massachusetts, USA [16] James H. McClellan, Ronald Schafe and Mak Yode. DSP Fist: A Multimedia Appoach. Pentice Hall, USA [17] MIDI Manufactues Association. The Complete MIDI 1.0 Detailed Specification, [18] E.R.S Peason. The Multiesolution Fouie Tansfom and its Application to the Analysis of Polyphonic Music. PhD Thesis, Wawick Univesity, REFERENCES [1] Eic Scheie. Extacting expessive pefomance infomation fom ecoded music. Maste s thesis, MIT, [2] Eic Scheie. Edito, MPEG-4 Stuctued Audio. FCD Subpat 5, May [3] Keith Matin. A Blackboad system fo Automatic Tansciption of Simple polyphonic Music. MIT Media Lab, Technical Repot # 385, [4] R.S Engelmoe and A.J. Mogan. Blackboad Systems. Addison-Wesley publishing, [5] J.P. Bello, G. Monti and M. Sandle. An Implementation of Automatic Tansciption of Monophonic Music with a Blackboad System. Poceedings of the ISSC, June [6] Daniel Ellis. Pediction-diven computational auditoy scene analysis. PhD Thesis, MIT, June [7] Keith Matin. Automatic Tansciption of Simple polyphonic Music: Robust Font End Pocessing. MIT Media Lab, Technical Repot # 399, Decembe [8] Daniel Ellis. Mid-level Repesentation fo computational auditoy scene analysis. In Poc. Of the Computational Auditoy Scene Analysis Wokshop; 1995 Intenational Joint Confeence on Atificial intelligence, Monteal, Canada, August [9] Anssi Klapui. Automatic Tansciption of Music. MSc Thesis, Tampee Univesity of Technology, [10] Malcolm Slaney. A citique of pue audition. In Poc. Of the Computational Auditoy Scene Analysis Wokshop, Monteal, Canada, August [11] Stuttgat Neual Netwok Simulato. Use Manual, vesion 4.1. Univesity of Stuttgat, Institute fo Paallel and Distibuted High Pefomance Systems. Repot No. 6/95. [12] Tistan Jehan. Music Signal Paamete Estimation. CNMAT Bekeley, USA DAFX-5