To appear in Inernaional Join Conference on Neural Neworks, Porland Oregon, 2003. Predicion of Pich and Yaw Head Movemens via Recurren Neural Neworks Mario Aguilar, Ph.D. Knowledge Sysems Laboraory Jacksonville Sae Universiy Jacksonville, AL 36265 marioa@ksl.jsu.edu Yair Barniv, Ph. D. NASA/Ames Research Cener Human Facors Research Division Moffe Field, CA 94035-1000 ybarniv@mail.arc.nasa.gov Aaron Garre MCIS Deparmen Jacksonville Sae Universiy Jacksonville, AL 36265 agarre@jsucc.jsu.edu Absrac In Virual-Environmen (VE) Applicaions, where virual objecs are presened in a head-mouned display, virual images mus be coninuously sabilized in space agains he user s head moion. Laencies in head-moion compensaion cause virual objecs o swim around insead of being sable in space. This resuls in an unnaural feel, disorienaion, and simulaion sickness in addiion o errors in fiing/maching of virual and real objecs. Visual updae delays are a criical echnical obsacle for implemenaion of head-mouned displays in a wide variey of applicaions. To address his problem, we propose o use machine learning echniques o define a forward model of head movemen based on angular velociy informaion. In paricular, we uilize recurren neural nework o capure he emporal paern of pich and yaw moion. Our resuls demonsrae an abiliy o predic head moion up o 40 ms. ahead hus eliminaing he main source of laencies. The accuracy of he sysem is esed for condiions akin o hose encounered in virual environmens. These resuls demonsrae successful generalizaion by he learning sysem. ha he virual objec is displayed in he same locaion in space as i was prior o he moion. Fig. 2 shows ha he poin on he HMD represening a saionary poin in space needs o be shifed on he nex display updae in order o compensae for he user s head moion. Tha can be accomplished by using he linear and angular locaion, velociy, and acceleraion measuremens ou of he head racker o compue and render he nex updaed image (henceforh, we will omi he word angular). Currenly hese operaions ake an overall minimum of abou 25 ms. I. INTRODUCTION In Virual-Environmen (VE) applicaions (a.k.a. Virual-Realiy), where virual objecs are presened in a head-mouned display, here is a need o compensae for head moion in order o presen space-sabilized virual objecs. Because of laencies in head-moion measuremens and image sabilizaion, virual objecs end o "swim" around insead of being sable in space. The purpose of his projec is o develop mehods for overcoming hese inheren laencies. Head moion is currenly measurable by a headmouned 6-degrees-of-freedom (6-DOF) inerial measuremen uni (IMU). However, even given his informaion, overall VE-sysem laencies canno be reduced under abou 25 milliseconds (ms) (see [1]). The deleerious effecs of laencies in image sabilizaion agains head moion have long been known o degrade manual conrol [2] and visual-moor adapaion o spaial disorions [3]. Laencies in VE disrup boh objecive measures of performance [4], [5], [6] as well as subjecive sense of presence [7], [8]. Fig. 1 shows an operaor rying o superimpose real and virual objecs observed hrough a semi-ransparen HMD display. Any head moion would ake he virual objec wih i unless he moion is sensed and compensaed for so Figure 1: A subjec aemping o superimpose virual and physical objecs. Saionary objec Figure 2: The poin, where he ray from a saionary objec o he observer s eye pierces he display panel, moves as a resul of head/display moion.
In our approach (see Fig. 3), we propose o uilize velociy daa colleced from rae gyros as inpus o a neural nework sysem ha has learned he forward model of he plan. In paricular, i learns he mapping from curren raegyro readings o fuure velociy readings (where fuure is defined as a few milliseconds ahead.) This informaion, a predicion of head movemen, is used o define he creaion of he new image for he virual environmen, hus eliminaing laency effecs. In previous work [9], we uilized a recurren neural nework approach o he problem of angular velociy predicion. There, we presened preliminary resuls of predicing yaw moion 20 ms. ino he fuure across wo subjecs. In his manuscrip, we repor on advances we have made o exend predicion o 40 ms. and esed he sysem across a broader subjec base. Furhermore, we have generalized he sysem o suppor predicion of boh pich and yaw moion. The srucure of he paper is as follows: we presen deails of he implemenaion in secion II. In secion III, resuls are presened along wih a discussion of he generalizaion capabiliies of he neural nework. We conclude wih a summary of resuls and a descripion of ongoing work aimed a embedding he predicion sysem wihin a VE sofware plaform. II. METHODS Our approach consiss of he use of a learning sysem o define a forward model of head moion. The learning sysem chosen is a neural nework which uilizes head velociy inpus o produce a predicion of velociy 40 ms. ino he fuure. The firs subsecion discusses he specifics of our daa collecion. Then, we describe how he daa is pre-processed o define he inpu feaures for he neural nework. The final subsecion presens he basic learning mechanism and a descripion of how i has been applied o predicing boh pich and yaw moion. A. Experimenal Seup An InerSense IS-600-series moion racker was used o collec angular velociy daa a 160 Hz from four differen subjecs (for convenience re-sampled a 200 Hz.) Each subjec was fied wih a head band conaining he inerial sensors and was insruced o perform movemens from a pre-defined se. The movemens were defined o provide represenaive raining and esing ses for he learning sysem: 1. abrup righ-lef moion, 2. smooh, coninuous righ-lef moion, 3. abrup op-boom moion, 4. smooh, coninuous op-boom moion, 5. smooh righ-lef moion wih a pause in he middle, 6. abrup righ-lef moion wih a pause in he middle, 7. smooh op-boom moion wih a pause in he middle, 8. abrup op-boom moion wih a pause in he middle, 9. smooh circular moion, and 10. smooh diagonal moion in boh 45 o and 135 o slans wih respec o he verical. The firs four movemen ypes were chosen o isolae he yaw and pich roaions of he head. Boh abrup and smooh movemens were recorded in order o sample wo exremes of velociy profiles. The movemens ha included a pause a he mid-poin of moion (i.e. subjec facing forward) were recorded o obain velociy daa ha could be segmened manually o provide compleemovemen daa o he neural nework. Specifically, when he subjec pauses a he mid-poin, heir head velociy goes o zero, hus allowing he experimeners o easily idenify end-of-movemen poins as he daa is prepared for he neural nework. Boh circular and diagonal movemens were recorded o provide a se of combined pich-and-yaw moions ha could be used o es our full predicion sysem. Subjecs were asked o move heir heads wihin a comforable range of moion. B. Feaure Descripion Figure 3. Proposed archiecure. A neural nework sysem is used o define a forward model ha produces a predicion of head moion o eliminae display laencies. One of he keys o successful learning and performance of a machine learning sysem is he qualiy of he inpus i is provided. The se of angular velociy-based feaures ha we used are summarized in Table 1. The moving average, average slope, average frequency, and average period are self-explanaory. The number of zero-crossings a ime, Z, is a measure of signal frequency. I is defined o be he number of insances where he angular velociy crosses 0 wihin a 400 ms window and is described by he following equaion:
Moving Average Average Slope Number of Zero-crossings Average Frequency Average Period Z = z( i), where i= W 1 z( ) = 0 if ω( ) ω( 1) 0 oherwise In each of hese equaions, ω() represens angular velociy a ime, and he W erm represens he window size. The curve complexiy is a feaure ha aemps o measure he shape of he curve. Eq. 2 provides he definiion of his feaure based on he waveform feaure in [10]: S ( ) = ω ( i) ω( i 1) (2) i= W The cumulaive, non-overlapping gradien was found o be he mos useful feaure for predicing he angular velociy daa. Eq. 3 defines his feaure: G cum Feaure Cumulaive, Non-overlapping Gradien Curve Complexiy W 2 (1) ( ) = ( ω ( i( W + 1) W ) ω( i( W + 1))) (3) i= 0 Table 1 Summary of Feaures Descripion Average angular velociy wihin a 400 ms window Average slope of angular velociy wihin a 400 ms window Number of insances where he angular velociy was 0 wihin a 400 ms window Insananeous frequency a each ime-sep Average amoun of ime beween zero-crossings wihin a 200 ms window Cumulaive measure of he gradien using non-overlapping 160 ms windows Sum across insananeous gradiens wihin a 400 ms window This feaure provides a measure of he raio of ongoing gradien rend and overall curve gradien. Thus, i provides a running measure of change per full movemen. This is because he measure reses iself whenever he angular velociy reaches he angular velociy a ime 0. (Generally, ω( 0 ) 0.) This is very useful when rying o predic on a movemen-by-movemen basis. We found ha he seven feaures presened above provided a good represenaion of he dynamics and characerisics of he raw angular velociy daa. As such, hese feaures were normalized and used as he inpus ino he learning sysem. Through principal componen analysis, we have found ha he number of feaures canno be reduced wihou losing performance. C. Learning Sysem A recurren neural nework known as an Elman neural nework [11] was used as he main building block of our learning sysem. The Elman neural nework is based on he back-propagaion algorihm which suppors off-line supervised learning. An Elman neural nework includes he ypical feed-forward conneciviy wih he addiion of a recurren connecion from he oupu of he hidden layer a ime o he inpu of he hidden layer a ime +1. This recurren connecion gives he nework an exponenial memory of pas evens [11]. This memory makes he Elman nework very effecive in learning emporal paerns, such as he ype we see in angular velociy daa. The appropriae parameers for he nework, which include he number of hidden neurons, he raining funcion, he ransfer funcions, ec., were deermined experimenally. A oal of 160 parameer combinaions were esed o derive he mos effecive archiecure. The final parameers chosen were as follows: Number of hidden neurons 15 Training funcion Levenberg-Marquard [12] (as compared o he use of he Quasi-Newon mehods in our original sysem [9]). Transfer funcions Hyperbolic angen sigmoid funcion for hidden neurons; linear funcion for oupu neurons Search funcion One-dimensional minimizaion using he mehod of Charalambous [13] Weighs iniializaion funcion Nguyen-Widrow layer iniializaion funcion The nework was rained by selecing relevan sample movemens from some of he experimens and esing on he remaining movemens and subjecs. For bes resuls, wo independen neural neworks were rained. One in which he eaching signal was defined as he desired predicion of yaw velociy (acual velociy manually shifed by 40 ms.). And a second one were raining focused on pich moion. Hence, he raining inpus were exraced from daa files for lef-righ and op-boom movemens respecively. As a resul of his independen raining, we were able o fully evaluae he performance of he sysem o he specific movemen regimes. Furhermore, by running he wo neworks in andem, we were able o combine heir resuls and obain moion predicion for boh degrees-of-freedom.
III. RESULTS A. Training he Yaw Predicion Neural Nework For yaw moion, we rained he nework on a series of movemens recorded from wo differen subjecs. The eaching signal for he neural nework was a ime-shifed copy of he recorded angular velociy. In his case, he emporal shif was se o 40 ms. o provide he nework wih a arge oupu signal ha would in effec lead o predicions of ha ime frame. The nework rained for approximaely 500 epochs and is MSE reached 0.02 for is [-1,1] normalized oupu. The nework was hen esed agains he full se of movemens colleced from our subjecs. Represenaive resuls of his es are shown in Fig. 4. Here, a subse of eleven differen movemens from various subjecs has been segmened and concaenaed ogeher o creae he resuling curve. In he figure, he red solid curve corresponds o he acual fuure angular velociy 40ms. (ime-shifed). The blue curve corresponds o he angular velociy as prediced by he neural nework. Hence, successful predicion resuls are indicaed by he large amoun of overlap beween he wo curves. Fig. 5 presens an enlarged image of he secion of he curves conained wihin he solid black recangle in Fig. 4. In his figure, he qualiaive degree of mach beween he wo curves can be seen in greaer deail. Clearly, he neural nework is capable of predicing well even in hose areas of abrup changes in angular velociy. B. Exending he Sysem o Pich Predicion In order o address he predicion of moion in he orienaions mos common in VE applicaions, our sysem was exended o predic head roaions in boh he yaw and pich direcions. In paricular, in virual environmens, users scan he horizonal plane wih greaer frequency and ampliude han he oher wo orienaions. Furhermore, as a secondary head moion, pich plays an imporan role for achieving spaial awareness. To his end, a new neural nework was rained uilizing he same baseline parameers and configuraion as described in he previous secion (e.g. number of hidden layers, ec.). Our iniial es resuls indicaed good ampliude mach bu an increased emporal error. Upon closer examinaion, i was found ha given he significan difference in moion dynamics, beer performance was obained when he raining funcion was replaced wih a conjugae gradien mehod. The sysem was rained over 500 epochs, and he emporal predicions obained were in line wih hose for yaw predicion. The raining se included hree ypes of moion: smooh op-boom, diagonal (45 o ), abrup op-boom. Fig. 6a presens he resuls on he raining se. Fig. 6b presens resuls obained wih he es se. Here, he movemens used included: diagonal (135 o ), circular, smooh opboom and abrup op-boom. The las wo movemens were recorded from a second se of subjecs. The spaial and emporal mach beween he curves provides very accurae predicions for he 40 ms. arge. Sample Number Figure 4. Normalized velociy predicion resuls for yaw movemens. Full curve consiss of 11 differen concaenaed movemens. Resuls indicae a very good mach beween he acual velociy manually shifed by 40ms (red) and he predicions of he neural nework (blue). X-axis indicaes he sample number where each sample is 5 ms. apar. Y-axis represens normalized velociy.
Figure 5. Deail of yaw predicion resuls. See ex for deails. X- and y-axes as in previous figure. C. Combined Pich and Yaw Predicion Tess The final design for our predicion sysem consiss of combining boh neural neworks o obain wo simulaneous oupus ha conrol he updaing of he VE display and accouns for moions around boh axes of moion. The resuling sysem was implemened in Malab and compues he required inpu feaures for boh yaw and pich rae-gyro signals. I hen feeds he corresponding inpus separaely o each of he neural neworks and records he resuling predicions. In order o perform a more horough es of he sysem, we se up a novel experimen o obain unconsrained moion daa. In his case, a new subjec was asked o perform head movemens analogous o hose performed during scanning of a virual environmen. Specifically, he subjec was asked o scan he immediae surrounding so as o invenory all of he objecs presen in he environmen. This resuled in a se of daa for which boh yaw and pich signals did no conain he ypically rhyhmic paerns ha had been used during raining of he neural neworks. The resuling moion and corresponding predicions are presened in Fig. 7. Here, he mach beween he acual and prediced curves once again indicaes effecive predicions across he enire experimen. Of significance is he fac ha accuracy is preserved for all abrup posiion changes performed by he subjec. 6 4 2 0-2 -4 (a) Our original sysem design had called for he inroducion of a secondary learning sysem ha would learn, in real-ime, any subjec-specific differences ha could lead o degradaion in predicion. However, hese resuls indicae ha i migh be possible o deploy he offline learning sysem and effecively apply i across muliple subjecs and ask condiions. -6 0 1000 2000 3000 4000 5000 6000 7000 8000 (b) Figure 6. Predicion resuls on he raining (a) and esing (b) ses. Y-axis is he scaled angular velociy.
200 Prediced Yaw Fuure Acual Yaw Prediced Pich 150 Fuure Acual Pich Yaw 100 50 Pich 0-50 0 2000 4000 6000 8000 10000 12000 Figure 7. Tes resuls for predicion of boh yaw and pich unconsrained moion. Y-axis: angular posiion (angular vel. inegraed o provide esimaed posiion). Noice he significan difference in ampliude beween yaw and pich signals as i is ypical for subjecs performing VE scanning paerns. IV. CONCLUSIONS We have exended our original work o produce a sysem beer ailored o reducing laency in virual environmen applicaions. In paricular, we have exended predicion imes from 20 o 40 ms. wihou a decrease in accuracy. Furhermore, our resuls of combining pich and yaw predicions indicae very good generalizaion across VE-relevan movemen dynamics. Of significance o his sudy is he finding ha he Elman recurren neural nework does a very good job of learning he seemingly complex emporal paern inheren in free-form head moion. A presen, we are working o por our feed-forward sysem o a real-ime plaform. Our goal is o inegrae he resuling forward model wihin he conrol loop of a funcioning virual environmen es plaform. Through his effor, we are hoping o fully assess he effeciveness of predicions in eliminaing laency effecs. ACKNOWLEDGEMENTS This research was conduced under NASA projec 711-51-12, P.I. Yair Barniv. Suppor for he firs and hird auhors was provided by NASA/Ames Research Cener gran No. NCC 2-1330. The auhors would like o hank S. Ellis and B. Adelsein of NASA/Ames for heir valuable insighs and suggesions. REFERENCES [1] Jacoby, R.H., B.D. Adelsein, and S.R. Ellis, Improved emporal response in virual environmens hrough sysem hardware and sofware reorganizaion", Proceedings, SPIE Conference on Sereoscopic Displays and Applicaions VII, Vol. 2653, Bellingham WA, pp. 271:284, 1996. [2] Sheridan, T.B., & Ferrell, W.R. (1963). Remoe manipulaive conrol wih ransmission delay. IEEE Trans. Hum. Fac, Elec., HFE-4(1), 25-29. [3] Held, R., Efsahiou, A., &Greene, M. (1966). Adapaion o displaced and delayed visual feedback from he hand. J. Exp. Psychol., 72(6), 887-891. [4] Liu, A., Tharp. G., French, L., Lai, S., & Sark, L. (1993) Some of wha one needs o know abou using head mouned displays o improve eleoperaor performance. IEEE Trans. Rob. Auom., 9(5), 638-648. [5] Ellis, S.R., Brean, F., Menges, B., Jacoby, R.H., & Adelsein, B.D. (1997) Operaor ineracion wih virual objecs: effecs of sysem laency. Proceedings, HC1 97 Inernaional, San Francisco, pp. 973-976. [6] Ellis, S.R., Adelsein, B.D., Baumeler, S., Jense, G.J., & Jacoby, R.H. (1999) Sensor Spaial Disorion, Visual Laency, and Upddae Rae Effecs on 3D Tracking in Virual Environmens. Proceedings, IEEE VR 99, Houson TX. [7] Welch, R.B., Blackman, T.T., Liu, A., Mellers, B.A., & Sark, L.W. (1996). The effecs of picorial realism, delay of visual feedback, and observer ineraciviy on he subjecive sense of presence. Presence, S(3), 263-273. [8] Ellis, S.R., Adelsein, B.D., Baumeler, S., Jense, G.J., & Jacoby, R.H. (1999). Sensor Spaial Disorion, Visual Laency, and Upddae Rae Effecs on 3D Tracking in Virual Environmens. Proceedings, IEEE VR 99, Houson TX. [9] A. Garre, M. Aguilar, and Y. Barniv (2002). A recurren neural nework approach o virual environmen laency reducion. Proceedings, IJCNN, Honolulu, HI. [10] H. Huang and C. Chen, Developmen of a Myoelecric Discriminaion Sysem for a Muli-degree Prosheic Head, Proceedings of 1999 IEEE Inernaional Conference on Roboics and Auomaion, pp. 2392-2397, 1999. [11] J. L. Elman, Finding Srucure in Time, Cogniive Science, Vol. 14, pp. 179-211, 1990. [12] M.T. Hagan and M. Menhaj, Training feedforward neworks wi he Marquard algorihm, IEEE Transacions on Neural Neworks, Vol. 5(6), pp. 989-993, 1994. [13] C. Charalambous, Conjugae-gradien algorihm for efficien raining of arificial neural neworks, IEEE Proceedings, Vol. 39 (3), pp. 301-310, 1992.