Gaussian Naive Bayes for Online Training Assessment in Virtual Reality-Based Simulators

Transcription

1 Mathware & Soft Computing 16 (2009), Gaussian Naive Bayes for Online Training Assessment in Virtual Reality-Based Simulators Ronei Marcos de Moraes, 1, Liliane dos Santos Machado 2 1 Department of Statistics, Federal University of Paraiba, Brazil 2 Department of Informatics, Federal University of Paraiba, Brazil ronei@de.ufpb.br, liliane@di.ufpb.br Abstract Training systems based on virtual reality are used in several areas, as in the medical sciences. In these systems the user is immersed into a virtual world to have realistic training through realistic interactions. In such training is important to know the quality of user's training and by didactic reasons the user must receive his/her assessment immediately after of end of training. For this reason, an online assessment system allows the user to improve his/her learning because it can identify, where he committed mistakes or presented low efficiency. Several approaches to perform assessment in training simulators based on virtual reality have been proposed. In this paper, we present a new approach to online training assessment based on Gaussian Naive Bayes for modeling and classification of simulation in M pre-defined classes. Gaussian Naive Bayes is a generalization of Naive Bayes Networks, which are a special case of probabilistic networks that allows treating continuous variables. Keywords: Gaussian Naive Bayes; online assessment; virtual reality; training systems. 1 Introduction Virtual Reality (VR) systems [3] are realistic environments that can be created and used for training. They can provide significant benefits over other methods of training, mainly in critical tasks, as medical procedures [5, 7, 8, 12]. However, it is important assess user's training to know the quality of his/her skills. Due to its complexity, several kinds of training cannot be simply classified as bad or good. Thus, the existence of an assessment tool incorporated into a simulation system based on VR is important to allow learning improvement and users assessment [5]. According to Aalsvoort et al. [1], assessment is a tool to help the construction of the 123

2 124 Ronei M. Moraes & Liliane S. Machado knowledge and the cognitive training, as well as to improve user s performance in execution of future activities [2]. Just a few years ago were proposed the first methodologies for training assessment performed in VR systems. Specific assessment methodologies for training through virtual reality simulators are still more recent [4]. Due to the fact that VR simulators are real-time systems, an assessment tool must continuously monitor all user interactions and compare his performance with pre-defined expert's classes of performance. By didactic reasons, it is more interesting the use of online assessment tools: the user can easily remember his mistakes and learn how to correct them. The main problems related to online training assessment methodologies applied to VR systems are the computational complexity and the accuracy. An online assessment tool must have low complexity to do not compromise VR simulations performance, but it also must provide high accuracy to do not compromise the user assessment. In general, an online assessment system should be capable to monitor user interactions while user operates the simulation system. In order to achieve that, it is necessary to collect the information about positions in the space, forces, torque, resistance, speeds, accelerations, temperatures, visualization and/or visualization angle, sounds, smells and etc. This information will be used to feed the assessment system. In the Figure 1 [16], we can observe that the virtual reality simulator and the assessment system are independent systems, however they act simultaneously. Figure 1: Diagram of a Virtual Reality Simulator with an Assessment System. Recently, some models for offline or online assessment of training have been proposed. Some of them use Discrete Hidden Markov Models [20] or Continuous Hidden Markov Models [21] to modelling forces and torque during a simulated training in a porcine model. Using an optoelectronic motion analysis and video

3 Gaussian Naive Bayes for Online Training Assessment in VR-Based Simulators 125 records, McBeth et al. [9] acquired and compared postural and movement data of experts and residents in different contexts by use of distributions statistics. Moraes and Machado proposed several methods for online assessment [13, 14, 15, 16]. After that, Morris et al. [17] suggested the use of statistical linear regression to evaluate user s progress in a bone surgery. Some of those models previously mentioned are based on machine learning and use discretization of continuous variables [15]. They use an assessment system based on Naive Bayes (NB) method. However, this approach can result in loss of the information. In this paper, we propose a new system for assessment based on a Gaussian Naive Bayes (GNB) classifier. Previous searches in literature did not identify other assessment methods based on this approach. However, it is important to observe the present assessment method is more robust than a simple application of a pattern recognition method. In fact, it is the kernel of a larger system, whose architecture is shown in Figure 1. This system can perform an online training assessment for VR simulators. The system uses a vector of information with data collected from user interactions in the VR simulator. These data are compared, by the assessment system, with M predefined classes of performance. To test the method proposed, we are using a bone marrow harvest simulator [5]. The simulator uses a robotic arm, which allows six degrees of freedom movements and provides force feedback [19], to give to the user the tactile sensations felt during the puncture in the virtual patient s body [5]. In the system the robotic arm simulates the needle used in the real procedure (Figure 2), and the virtual body visually represented has the tactile properties of the real tissues. The assessment tool proposed supervised the user movements during the puncture and evaluated the training according to M possible classes of performance. Figure 2: Robotic arm to simulate a needle in the virtual environment.

4 126 Ronei M. Moraes & Liliane S. Machado 2 Assessment tool based on Gaussian Naive Bayes This section presents the method for training assessment based on GNB. For reader's better understanding, we first present a short review about NB method [10, 15] and after the GNB method. NB method The NB Method, so called Discrete or Multinomial NB, is a robust method for classification data. Formally, let be the classes of performance in space of decision Ω={1,...,M} where M is the total number of classes of performance. Let be w i, i Ω the class of performance for an user. A NB classifier computes conditional class probabilities and then predict the most probable class of a vector of training data X, according to sample data D, where X is a vector with n features obtained when a training is performed, i.e. X={X 1, X 2,, X n }. Using the Bayes Theorem: P(w i X) = [P(X w i ) P(w i )] / P(X) = [P(X 1, X 2,, X n \ w i ) P(w i )] / P(X) (1) The NB classifier receives this name because its naive assumption of each feature X k is conditionally independent of every other feature X l, for all k l n. Unless a scale factor S, which depends on X 1, X 2,, X n, the equation (1) can be expressed by: P(w i X 1, X 2,, X n ) = (1/S) P(w i ) Π n k=1 P(X k \ w i ) (2) Then, the classification rule for NB is done by: X w i if P(w i X 1, X 2,, X n ) > P(w j X 1, X 2,, X n ) for all i j and i, j Ω (3) and P(w * X 1, X 2,, X n ) with * = {i, j i, j Ω}, is done by (2). GNB method As mentioned above, the NB Method must be applied over discrete or multinomial variables. Some approaches were developed to use NB Method with continuous variables, as several discretization methods [6, 23] were used in the first stage to allow the use of the Naive Bayes method after. However, this approach can affect classification bias and variance of the NB method. Other approach is use Gaussian distribution for X and to compute its parameters from D, i.e., mean vector and

5 Gaussian Naive Bayes for Online Training Assessment in VR-Based Simulators 127 covariance matrix [11]. From equation (2) and using some mathematical simplification, it is possible to reduce computational complexity of that equation: log [P(w i X 1, X 2,, X n )] = log [(1/S) P(w i ) Π n k=1 P(X k \ w i )] = log (1/S) + log P(w i ) + n k=1 log[p(x k \ w i )] (4) As S is a scale factor, it is not necessary be computed in classification rule for GNB. Then: X w i if {log P(w i ) + n k=1 log[p(x k \ w i )]} > {log P(w j ) + n k=1 log[p(x k \ w j )]} for all i j and i, j Ω (5) Based on the same space of decision with M classes, a GNB method computes conditional class probabilities and then predicts the most probable class of a vector of training data X, according to sample data D. The parameters of GNB method are learning from data and the conditional probabilities are estimated using equation (4) and the final decision about vector of training data X is done by equation (5). The assessment tool The assessment tool proposed should supervise the user s movements and other parameters associated to them. The VR simulator and the assessment tool are independent systems, however they act simultaneously. The user's interactions with the simulator are monitored and the information is sent to the assessment tool that analyzes the data and emits a report on the user's performance at the end of the training. The VR system used for the tests is a bone marrow harvest simulator [5]. For reasons of general performance of the VR simulator, were chosen to be monitored the following variables: spatial position, velocities, forces and time on each layer. Previously, an expert, according M classes of performance defined by him, calibrated the system. The calibration process consists in to execute several times the procedure and to classify each one according to classes of performance. The number of classes of performance was defined as M=3: 1) correct procedures, 2) acceptable procedures, 3) badly executed procedures. So, the classes of performance for a trainee could be: "you are well qualified", "you need some training yet" and "you need more training". The information of variability about these procedures is acquired using GNB method. Then, user can interact with the simulator using a haptic device and feel the reactions. The haptic loop rate is around 1000Hz. However, in order to not compromise the real time of the simulation, the interaction samples were acquired 5 times per second, without compromise the assessment process.

6 128 Ronei M. Moraes & Liliane S. Machado In our case, we assume that the font of information for w i classes is the vector of the sample data D. The user makes his/her training in VR simulator and the Assessment Tool based on GNB (ATBGNB) collects the data from his/her manipulation. All probabilities of data for each class of performance are calculated by (4) and at the end the user is assigned to a w i class of performance by (5). So, when a trainee uses the system, his performance is compared with each expert's class of performance and the ATBGNB assigns the better class, according to the trainee's performance. At the end of the training, the assessment system reports the classification to the trainee. The calibration of the ATBGNB was performed off-line, before any assessment of training. For that, an expert executed the procedure twenty times for each class of performance. After, for a controlled and impartial analysis, several users used the system and 150 training procedures were monitored. The data collected from these trainings were manually labeled according to experts specifications. Such task demands several hours of work from experts, which is not always possible. Thus, this is a restriction of the assessment problem that influences the development of the presented methodology. These same cases were labeled using the ATBGNB and it generated the classification matrix showed in Table 1. The diagonal of that matrix shows the correct classification. In the other cells, we can observe the mistakes of classifications. It was used the Kappa Coefficient to perform the comparison of the classification agreement [18], as recommended by literature of pattern recognition [22]. From the classification matrix obtained, the Kappa coefficient for all samples was K=80.0% with variance In only 20 cases, the assessment tool made mistakes. It is important to note that for the class acceptable procedures, only one classification was incorrect. That performance is very acceptable and it shows the good adaptation of ATBGNB in the solution of this assessment problem. Table 1. Classification matrix for the Assessment Tool based on Gaussian Naive Bayes. Class of performance according to experts Class of performance according to Assessment Tool based on Gaussian Naive Bayes Another important result is the computational performance of the assessment tool: with a Pentium IV PC compatible, 2GB of RAM, the average time of CPU consumed by the assessment was seconds of CPU. Then, we can affirm that the ATBGNB has low computational complexity. It allows the inclusion of other

7 Gaussian Naive Bayes for Online Training Assessment in VR-Based Simulators 129 variables in the assessment tool without degradation of the performance of the VR simulation. 3 Comparison with an assessment tool based on classical Bayes rule A comparison was performed between the ATBGNB and the Assessment Tool based on NB [15] (ATBNB). The ATBNB was configured and calibrated by the expert for the same three classes used before. The same sixty samples of training (twenty of each class of performance) were used for calibration of the two assessment systems. The same way, the data of the same 150 procedures from users training were used for a controlled and impartial comparison between the two assessment systems. Due to structural aspects of ATBGNB this comparison occurred with the ATBNB: both have the same statistical correlation structure but differ in the distribution of presumed probability for the random variables. The comparison between the present assessment method and others with correlation structural difference in their statistical characteristics did not allow a direct comparison of methods, but only their final results. The classification matrix obtained for the ATBNB is presented in the Table 2. The Kappa coefficient was K=66.0% with variance %. In 34 cases, the assessment tool made mistakes. However, that performance is acceptable and shows that an ATBNB is a competitive approach in the solution of assessment problems. It is possible to see by Tables 1 and 2 and by Kappa coefficients that the performance of the ATBNB is lower than the one based on GNB. In statistical terms, the difference of performance between those assessment methods is significant. About computational performance of the Assessment Tool, the one based on GNB was faster than the one based on NB. The average of CPU time consumed for assessment of training based on NB was seconds of CPU using the same Pentium IV PC compatible. Table 2. Classification matrix for Assessment Tool based on Naive Bayes. Class of performance according to experts Class of performance according to Assessment Tool based on Naive Bayes

8 130 Ronei M. Moraes & Liliane S. Machado 4 Conclusions and further works In this paper we presented a new approach to online training assessment in VR simulators. This approach uses an Assessment Tool based on Gaussian Naive Bayes and solves the main problems in assessment procedures: low complexity and high accuracy. Systems based on this approach can be applied in VR simulators for several areas and can be used to classify a trainee into classes of learning to give him a status about his performance. A simulator based on VR was used as base for the performance tests. The performance obtained by an Assessment Tool based on Gaussian Naive Bayes was compared with an Assessment Tool based on Naive Bayes. From the obtained data, it is possible to conclude that the Assessment Tool based on Gaussian Naive Bayes presents significant better results when compared with an Assessment Tool based on Naive Bayes for the same case. Besides, the Assessment Tool based on Naive Bayes presented better computational performance in terms of CPU time. The performance aspect of ATBGNB shows the aspect of spatial separability of the problems. The GNB method was already applied in other problems with excellent results. However, the statistical separability among performance categories is difficult to be obtained due to subjectivity aspects of the experts to classify performance (own and of trainees). It can explain a possible low performance of assessments, but also estimates its adaptability to the presented problem. The presented assessment method can also be used for other complex scenario, as driving and flight training and several skills training based on virtual reality. Equally, longer training sessions could be simulated since the hardware used could provide enough performance to sustain the real time required for a virtual reality simulation. Acknowledgments This work is partially supported by Brazilian Council for Scientific and Technological Development, CNPq (Process / and Process CT-INFO- CNPq /2004-6) and Brazilian Research and Projects Financing, FINEP (Grant ). References [1] Aalsvoort, G.M., Resing, W.C.M. and Ruijssenaars, A.J.J.M. Learning potential assessment and cognitive training: Actual research and perspectives in theory building and methodology. JAI/Elsevier Science, [2] Arter, J.A. and Mctighe, J. Scoring Rubrics In The Classroom: Using Performance Criteria For Assessing And Improving Student Performance. Corwin Press, 2000.

9 Gaussian Naive Bayes for Online Training Assessment in VR-Based Simulators 131 [3] Burdea, G. and Coiffet, P. Virtual Reality Technology, 2 nd ed., Wiley Interscience, [4] Burdea, G., Patounakis, H., Popescu, V. and Weiss, R.E. Virtual Reality Training for the Diagnosis of Prostate Cancer. Proceedings of IEEE Virtual Reality Annual Int. Symposium, pp , [5] Färber, M. et al. Heinz Handels Training and evaluation of lumbar punctures in a VR-environment using a 6DOF haptic device. Studies in Health Technology and Informatics, 132: , [6] Kononenko, Inductive and Bayesian learning in medical diagnosis. Applied Artificial Intelligence 7(4), , [7] Kumagai, K., Yamashita, J., Morikawa, O. and Yokoyama, K. A New Force-Based Objective Assessment of Technical Skills in Endoscopic Sinus Surgery. Studies in Health Technology and Informatics. 125: , [8] Machado, L. et al., A Virtual Reality Simulator for Bone Marrow Harvest for Pediatric Transplant. Studies in Health Technologies and Informatics. 81, , IOSPress, [9] McBeth, P. et al., Quantitative Methodology of Evaluating Surgeon Performance in Laparoscopic Surgery. Studies in Health Technologies and Informatics. 85, , IOSPress [10] Mitchell, T. Machine Learning, McGraw-Hill, [11] Mitra. P.S. Automated Knowledge Discovery from Functional Magnetic Resonance Images using Spatial Coherence. PhD Thesis. School of Med.University of Pittsburgh, 2006, 145 p. [12] Moody, L. et al., Objective Surgical Performance Evaluation Based on Haptic Feedback. Studies in Health Technology and Informatics. 85: , [13] Moraes, R. and Machado, L. Hidden Markov Models for Learning Evaluation In Virtual Reality Simulators. International Journal of Computers & Applications, 25(3): , [14] Moraes, R. and Machado, L. Using Fuzzy Hidden Markov Models for Online Training Evaluation and Classification in Virtual Reality Simulators. International Journal of General Systems. 33(2-3), , [15] Moraes, R. and Machado, L. Assessment Based on Naive Bayes for Training Based on Virtual Reality. Proceedings of International conference on Engineering and Computer Education (ICECE2007). Brazil, , [16] R. Moraes and L. Machado, Fuzzy Bayes Rule for On-Line Training Assessment in Virtual Reality Simulators. Jounal of Multiple-Valued Logic and Soft Computing. 14, , [17] Morris, D. et al., Visuohaptic simulation of bone surgery for training and evaluation. IEEE Computer Graphics & Aplications 26(6), pp , IEEE, [18] Richards, J. Remote Sensing Digital Image Analysis: An Introduction, 2 nd ed. Springer-Verlag: Berlin, [19] Robles-De-La-Torre, G. The Importance of the Sense of Touch in Virtual and Real Environments. IEEE MultiMedia, 13(3), [20] Rosen, J. et al., Hidden Markov Models of Minimally Invasive Surgery. Studies in Health Technologies and Informatics. 70, , IOSPress, [21] Rosen, J. et al. Objective Laparoscopic Skills Assessments of Surgical Residents Using Hidden Markov Models Based on Haptic Information and Tool/Tissue Interactions. Studies in Health Technologies and Informatics. 81, , IOSPress, 2001.

10 132 Ronei M. Moraes & Liliane S. Machado [22] Webb, A. Statistical Pattern Recognition, 2 nd ed. John Wiley & Sons: Chichester, England [23] Yang, Y. and Webb, G. A Comparative Study of Discretization Methods for Naive- Bayes Classifiers. Proceedings of 2002 Pacific Rim Knowledge Acquisition Workshop (PKAW'02), , 2002.