Effect of Dynamic Local Lag Control with Dynamic Control of Prediction Time in Joint Haptic Drum Performance Mya Sithu, Yutaka Ishibashi, and Norishige Fukushima Graduate School of Engineering Nagoya Institute of Technology Nagoya, Japan Feb. 17, 2014 Yangon, Myanmar ICCA 2014
Outline Background Previous Work Purpose Networked Haptic Drum System Dynamic Local Lag Control with Dynamic Control of Prediction Time Assessment System and Methods Assessment Results Conclusions and Future Work
Background A number of researchers have been paying attention to musical performance in networked haptic environments. Haptic interface device Network Haptic interface device Network delay Synchronization quality of sound and interactivity may seriously be deteriorated.
Previous Work (1/5) To achieve high synchronization quality of sound, Irie et al., used the local lag control for a networked ensemble. *1 The local lag control *2,3 buffers the local information for a constant time (called the local lag). The interactivity is degraded. They set the local lag to the same value as the network delay from the local terminal to the other terminal. The interactivity may seriously be degraded when the network delay is large. *1 Y. Irie et al., (in Japanese), IPSJ SIG Technical Report, vol. 2009-DPS-141, no. 23, Nov. 2009. *2 M. Mauve et al., IEEE Trans. on Multimedia, Feb. 2004. *3 D. Stuckel and C. Gutwin, Proc. ACM CSCW, pp. 447-456, Nov. 2008.
Previous Work (2/5) They assumed that the network delay from the local terminal to the other terminal is equal to that in the opposite direction (called the symmetric delay case). Usually, in a network, the network delay from the local terminal to the other terminal is different from that in the opposite direction (called the asymmetric delay case). High synchronization quality of sound may not be achieved.
Previous Work (3/5) *4 M. Sithu et al., Proc. IEEE GCCE, pp. 461-465, Oct. 2013. We subjectively investigated the effect of the local lag control on the synchronization quality of sound, interactivity, and comprehensive quality in the joint performance of a networked haptic drum system. *4 There exists the optimum value of local lag for joint musical performance. The optimum value of local lag is the same as the network delay when the network delay is small, but the value is smaller than the network delay when the network delay is large. This is because the interactivity is severe in the joint performance. The optimum value of local lag is dependent on the network delay from the other terminal to the local terminal.
Previous Work (4/5) We proposed the dynamic local lag control, which dynamically changes the local lag according to the network delay from the other terminal to the local terminal for both of the symmetric and asymmetric delay cases. *5 We set the local lag to the value smaller than or equal to the network delay. The dynamic local lag control is effective, but the interactivity slightly deteriorates. We need to further improve interactivity. *5 M. Sithu et al., Proc. ACM NetGames 2013, Dec. 2013.
Previous Work (5/5) Prediction *6,7,8,9 is one of the methods which can improve the interactivity. We proposed a group synchronization control scheme with prediction to keep the interactivity high for haptic work. *7 There is the optimum value of prediction time according to the network delay and the type of work. We investigated the effect of dynamic local lag control with prediction in the joint performance of the networked haptic drum system. *9 There exists the optimum value of prediction time according to the network delay. *6 L. A. Zadeh et al., Journal of Applied Physics, vol. 21, pp. 645-655, 1950. *7 P. Huang et.al., IJCNS, vol. 5, no. 6, pp. 321-331, June 2012. *8 Y. Hara et.al., Proc. ACM NetGames 2012, Noc. 2012. *9 M. Sithu et.al., IEICE Technical Report, CQ2013-67, Jan. 2014.
Purpose It is worth changing the prediction time according to the network delay. Such a study has not been done so far. This Work We propose dynamic local lag control with dynamic control of prediction time which dynamically changes the prediction time according to the network delay in the joint performance of the networked haptic drum system. We make a comparison between the dynamic local lag control with dynamic control of prediction time and that with fixed value of prediction time on subjective quality.
Networked Haptic Drum System User 1 s drumstick User 2 s drumstick High-hat cymbals Snare drum Floor tom Bass drum Terminal 1 Terminal 2 Headset PC 1 Display Display PC 1 Headset Haptic interface device (PHANToM Omni) PC 2 Switching hub Network Switching hub PC 2 Haptic interface device (PHANToM Omni) User 1 User 2
Dynamic Local Lag Control with Dynamic Control of Prediction Time (1/3) Dynamic Local Lag Control The dynamic local lag control dynamically changes the local lag (denoted by ) according to the network delay from the other terminal to the local terminal. The value of is calculated from the following equation *5 : D : = 0.637 D + 6.578 The time interval from the moment an media unit (MU) is generated at the other terminal until the instant the MU is output at the local terminal. An MU is the information unit for media synchronization and includes the identification number (ID) of the user, the positional information of the PHANToM cursor, and the sequence number of the servo loop. *5 M. Sithu et al., IEICE Technical Report, MVE2013-2, May 2013.
ynamic Local Lag Control with Dynamic ontrol of Prediction Time (2/3) Dynamic Local Lag Control with Prediction The prediction control outputs the position information by predicting the future position later than the output time of an MU by the prediction time T predict ( 0 ms) to keep the interactivity high. The first-order prediction is used for simplicity. The control also advances the output time of each MU at the local terminal by T predict ms. However, if there does not exist an MU which should be output after T predict ms, the MU is output by prediction.
Dynamic Local Lag Control with Dynamic Control of Prediction Time (3/3) Dynamic Local Lag Control with Dynamic Control of Prediction Time T predict is changed dynamically according to the network delay. We investigated the relation between T predict and the network delay by regression analysis, and we obtained the following equation. T predict = max (0.007D 2 0.327D + 1.818, 0) T predict : The optimum value of prediction time D : The time interval from the moment an MU is generated at the other terminal until the instant the MU is output at the local terminal.
Assessment System Local information ms (Local Lag 1) Headset Terminal 1 Received information Network emulator (NIST Net) Terminal 2 Received information PC 1 Display PC 1 Display Local information ms (Local Lag 2) Headset Haptic interface device PC 2 PC 2 Constant delay Haptic interface device User 1 User 2 The network emulator generates an additional constant delay for each packet transmitted between the terminals.
Assessment Methods (1/4) Each subject hits the same drum component repeatedly. The subject does not need to move his/her right drumstick to the other drum components. High-hat cymbals Snare drum Floor Tom Rhythm 1 Rhythm 2 He/she needs to move his/her right drum stick between the snare drum and the floor tom. High-hat cymbals Snare drum Floor Tom The distances of the drumstick movements are different between the two rhythms. Subjects played the rhythms at 60 bpm (beats per minute) and 100 bpm. Slow tempo Fast tempo
Assessment Methods (2/4) The constant delay and two types of control were selected in random order for each pair of subjects. In the control with fixed value of prediction time, T predict was selected in random order for the pair. The constant delay from terminal 2 to terminal 1 is set to the same value as that in the opposite direction (i.e., the symmetric constant delay). Two combinations of rhythms and tempos are employed as follows: *9 Rhythm 1 at slow tempo Rhythm 2 at fast tempo We found that the results of the other combinations of rhythm and tempo are almost the same as those of rhythm 1 at the slow tempo. *9 *9 M. Sithu et al., IEICE Technical Report, MVE2012-93, Jan. 2013.
Assessment Methods (3/4) Each subject was asked to base his/her judgment on the following qualities: Output quality of own drumstick Output quality of other drumstick Synchronization quality of sound Interactivity Comprehensive quality (weighted sum of the above four qualities) Comprehensive quality is the most important quality because it is the synthesis of the other four qualities.
Assessment Methods (4/4) Five-grade impairment scale Score Description 5 Imperceptible 4 Perceptible, but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying Subjects Number of subjects: 16 Age: Between 20 and 28 Gender: Male and female We obtain Mean Opinion Score (MOS) by averaging scores of all the subjects. Each stimulus: 30 seconds Total assessment time: One and half hours
MOS Assessment Results (1/6) Output quality of own drumstick Rhythm 1 at slow tempo 5 4 3 2 1 I 95% confidence interval 0 50 100 150 Constant delay (ms) Prediction time 0 ms 10 ms 20 ms 30 ms 40 ms 50 ms 60 ms 70 ms 80 ms 90 ms 100 ms 110 ms dynamic
MOS Assessment Results (2/6) Output quality of other drumstick Rhythm 1 at slow tempo I 95% confidence interval 5 4 3 2 1 0 50 100 150 Constant delay (ms) Prediction time 0 ms 10 ms 20 ms 30 ms 40 ms 50 ms 60 ms 70 ms 80 ms 90 ms 100 ms 110 ms dynamic
MOS Assessment Results (3/6) Synchronization quality of sound Rhythm 1 at slow tempo 5 Prediction time 0 ms 10 ms 4 20 ms 30 ms 3 40 ms 50 ms 60 ms 2 70 ms I 95% confidence interval 80 ms 90 ms 1 100 ms 0 50 100 150 110 ms Constant delay (ms) dynamic
MOS Assessment Results (4/6) Interactivity Rhythm 1 at slow tempo 5 Prediction time 0 ms 10 ms 4 20 ms 30 ms 40 ms 3 50 ms 60 ms 2 70 ms 80 ms I 95% confidence interval 1 90 ms 100 ms 0 50 100 150 110 ms Constant delay (ms) dynamic
MOS Assessment Results (5/6) Comprehensive quality Rhythm 1 at slow tempo 5 4 3 2 1 I 95% confidence interval 0 50 100 150 Constant delay (ms) Prediction time 0 ms 10 ms 20 ms 30 ms 40 ms 50 ms 60 ms 70 ms 80 ms 90 ms 100 ms 110 ms dynamic
MOS Assessment Results (6/6) Comprehensive quality Rhythm 2 at fast tempo I 95% confidence interval Prediction time 5 0 ms 10 ms 4 20 ms 30 ms 3 40 ms 50 ms 60 ms 2 70 ms 80 ms 90 ms 1 100 ms 0 50 100 150 110 ms Constant delay (ms) dynamic
Conclusions We proposed the dynamic local lag control with dynamic control of prediction time in order to keep the interactivity and synchronization quality of sound high. We investigated the effect of the proposed control by subjective assessment in the joint performance of a networked haptic drum system. The dynamic local lag control with dynamic control of prediction time is effective.
Future Work Enhance the control so that three or more users can perform joint musical performance Handle the joint musical performance with different kinds of musical instruments