Review: Recent Development of Signal Processing Algorithms for SSVEP-based Brain Computer Interfaces

Transcription

1 Journal of Medical and Biological Engineering, 34(4): Review: Recent Development of Signal Processing Algorithms for SSVEP-based Brain Computer Interfaces Quan Liu 1 Kun Chen 1,2,* Qingsong Ai 1 Sheng Quan Xie 1,2 1 School of Information Engineering, Wuhan University of Technology, Wuhan , China 2 Department of Mechanical Engineering, The University of Auckland, Auckland 1010, New Zealand Received 25 Apr 2013; Accepted 12 Aug 2013; doi: /jmbe.1522 Abstract Steady-state visual evoked potential (SSVEP)-based brain computer interfaces (BCIs) have gained considerable research interest because of their higher signal-to-noise ratio and greater information transfer rate than those of other BCI techniques. The signal processing algorithm is of key importance to the performance of BCI systems, and therefore plays a significant role in practical applications. However, there is no comprehensive review of the signal processing algorithms used for SSVEP-based BCIs. This paper reviews relevant papers and analyzes recent developments in use of these algorithms. The aim is to find their limitations to provide a guideline for researchers in this field of SSVEP-based BCIs. Techniques employed for signal preprocessing, feature extraction, and feature classification are discussed. Algorithms that can be applied to nonlinear and non-stationary signal processing are increasingly employed rather than traditional Fourier-based transforms because they are more suitable for the characteristics of SSVEPs. Spatial filtering techniques for channel selection are better at eliminating nuisance signals than those that use a single channel signal for processing. In addition, other factors that affect the performance of the system are discussed. Keywords: Steady-state visual evoked potential (SSVEP), Brain computer interface (BCI), Signal processing, Spatial filtering 1. Introduction A brain computer interface (BCI) is a communication system that does not depend on the brain s normal output pathways of peripheral nerves and muscles [1]. The ultimate goal of BCI is to create a specialized interface that will allow an individual with severe motor disabilities to have effective control of devices such as computers, speech synthesizers, assistive appliances, and prostheses [2]. Electroencephalography (EEG) is most commonly used for BCIs due to its portability and ease of use. There are four typical EEG BCI paradigms: steady-state visual evoked potentials (SSVEPs), slow cortical potentials (SCPs), the P300 component of evoked potentials, and sensorimotor rhythms (SMRs) [3]. An SSVEP is a periodic response to a visual stimulus modulated at a frequency of higher than 6 Hz [4] (or high than 4 Hz [5]). It can be recorded from the scalp as a nearly sinusoidal oscillatory waveform with the same fundamental frequency as that of the stimulus, and often includes some higher harmonics. The amplitude and phase characteristics of SSVEPs depend upon the stimulus intensity and frequency. * Corresponding author: Kun Chen Tel: ; Fax: chenkun_200411@sina.com SSVEP-based BCIs are becoming a research hotspot due to their many advantages over other BCI systems, including a higher signal-to-noise ratio (SNR) and a faster information transfer rate (ITR). They also do not require intensive training [6]. According to the modulation method and feature variable used for classification, there are two typical types of SSVEPbased BCI, namely frequency- and phase-coded. Frequencycoded SSVEPs use visual stimuli with distinct frequencies, and detect targets by checking the spectral peaks in the recorded spectrum [7]. The number of stimuli is always equal to the number of targets. Phase-coded SSVEPs use visual stimuli with the same frequency but different phases. These systems identify gazed targets by comparing the phase lags between the measured SSVEPs and a reference one [8]. In the last few years, many SSVEP-based BCI systems have been developed. In 1999, cursor control based on SSVEPs was implemented [9]. Four rectangles that flashed at different frequencies on the screen around the cursor were used to indicate four directions. Another study [10] employed SSVEPs to control a humanoid robot system asynchronously. A spelling system [11] was successfully commanded by 11 healthy subjects to spell BRAINCOMPUTERINTERFACE at an average ITR of 27 bits/min. A study [12] designed an asynchronous SSVEP-based hospital bed nursing system that can be used without training of the user. Fifteen subjects took part in the test, with the system achieving an average accuracy

2 300 J. Med. Biol. Eng., Vol. 34 No of 92.5% and an average command transfer interval of 5.22 seconds per command. Various signal processing methods have been applied to detect and process SSVEPs. The algorithms have a great impact on the performance of the whole system, particularly the detection accuracy and the ITR. In this review, algorithms commonly used in SSVEP-based BCIs are compared. Techniques used in the three stages of signal processing, namely signal preprocessing, feature extraction, and feature classification, are compared and analyzed. 2. Objectives A standard BCI system usually includes multiple stages, including data acquisition, data preprocessing, feature extraction, feature classification, and command translation. Figure 1 shows a BCI system divided into three layers: input (EEG data acquired from the subjects), system function (data processing algorithm), and output (control of external devices). Sometimes, the subjects can get feedback from the actions of the devices and then make adjustments to have better control of the devices. Figure 1. Flow chart of a BCI system. The data processing algorithm usually includes data preprocessing, feature extraction, and feature classification. The aim of data preprocessing is to eliminate nuisance signals. Feature extraction and feature classification use the characteristics of SSVEP signals to identify a subject s intent to control an external device. The algorithm acts like a bridge that links the subjects and the external devices. Its influence on the performance of a BCI system should thus be fully understood. This paper conducts a comprehensive review of algorithms used for SSVEP-based BCIs. It focuses on data preprocessing, feature extraction, and feature classification. The objectives of this review are to determine which algorithms are mostly used and what are the key issues for SSVEP signal processing algorithms, and to identify current development. This information can be used as a guideline and reference for future work. The purpose of signal preprocessing is to remove nuisance signals to get pure EEG signals and improve the recognition of target signals. Most filtering methods are based on frequency filtering (i.e., they modify the characteristics of signals in the frequency domain), such as the band-pass filter. In a lot of EEG research, a band-pass filter and a notch filter are embedded in the design of the data acquisition hardware. This review focuses on filters implemented in software. Spatial filtering uses a linear combination of signals collected from multiple electrodes in order to improve detection accuracy [13]. Spatial filtering is usually utilized for signal preprocessing (see Table 1). In some cases, it can also be used to extract features, as done in the minimum energy combination (MEC) method. Two typical techniques are used for feature classification, namely comparing the feature values corresponding to different stimulations and using a classifier to confirm the target with these feature variables. 3.1 Signal preprocessing SSVEPs are easily contaminated by other bio-signals or environmental noise. Therefore, the first step is usually to remove these nuisance signals as much as possible. As shown in Table 1, band-pass and notch filters are commonly used. The former is used to limit the signals to be processed to within a certain frequency range, which corresponds to the stimulation frequencies used and their harmonics. Since SSVEPs are located in a narrow-band frequency range, the frequency width of the band-pass filter is often relatively narrow. The stimulation frequency can be classified into three bands [14]: low (1-12 Hz), medium (12-30 Hz), and high (30-60 Hz). Most systems used low frequencies because they evoke higheramplitude SSVEPs. However, low-frequency flickering may lead to visual fatigue. In addition, low and medium flickering can even induce epileptic seizures [15]. For safety and comfort, higher stimulation frequencies are preferable [16]. It can be seen in Table 1 the higher pass frequency for papers [17,18] are 60 Hz and 100 Hz because they used higher stimulation frequencies. The notch filter is utilized to remove power line interference. In most countries, this frequency is 50 Hz. In addition, an adaptive filter has been used for signal preprocessing [19], which is not very common in SSVEP-based BCIs. Almost all papers reviewed used band-pass and notch filters, implemented either in hardware or software. Many EEG signal acquisition devices contain these filters (hardware filtering). 3. Discussion The methods used for signal preprocessing, feature extraction, and feature classification are shown in Tables 1, 2, and 3 respectively. The spatial filtering methods can be used for feature extraction as well as signal preprocessing. These methods are compared in more detail in the section on feature extraction.

3 Development of Algorithms for SSVEP-based BCIs 301 Signal preprocessing method Frequency filtering Band-pass filter Description Table 1. Signal preprocessing methods used for SSVEP-based BCIs. Filters are designed according to frequency characteristics of related signals. A band-pass filter is used. The frequency range is designed according to the stimulation frequencies or their harmonics. A band-bass filter is easy to implement, but might be too stringent to explain time-varying signals. References [7] Hz [8] Hz [17] 3-60 Hz [18] Hz [21,22] Hz [27] 3-40 Hz [41] 1-30 Hz [45] 3-45 Hz [48] 5-50 Hz [52,53] 3-60 Hz [54] 5-35 Hz [55] Hz [58] 8-24 Hz [64] Hz [65] 5-40 Hz [18,58] 50 Hz Notch filter A notch filter is used, usually to remove power line interference. Spatial filtering Spatial filtering combines signals from different channels to magnify the SSVEP responses or reduce the interference of noise. Signals from multiple channels are less affected by noise than signals from a unipolar or bipolar system. The spatial filtering technique can be also used to extract features. MEC Used to cancel nuisance signals as much as possible. Similar [11,36-38,56,66] with MCC, it is based on a model between SSVEPs and standard sine-cosine waves and noise. MCC The MCC method aims to maximize the ratio between [16,33,34] SSVEPs and the background signals. The principle is similar to MEC, but the object function used for computing filter weight coefficients is different. The computation is a little more complex than that for MEC. PCA PCA is used to decompose signals into components of [40] SSVEP responses and brain activities. It aims to reduce the dimension of original data. It needs other methods for feature extraction. ACSP A common spatial pattern method based on the [42] analytic representation of signals. It reflects both amplitude and phase information of SSVEPs based on the CSP method. MCC + CCA [39] CAR For the CAR method, the average value of all electrodes [50-53] is subtracted from the channel of interest to make the EEG recording nearly reference-free. CCA Computes the relation between two multi-variable data sets [6,46,67] after linear combinations of original data. KCCA Developed based on the traditional CCA method to project [45] data into a high-dimension space. Multiway CCA Uses the optimal reference signals after adjustment. However, [48] it increases computation time. p-cca The phase information is used in the reference signals. [49] Feature extraction methods Fourier-based transform FFT Description Table 2. Feature extraction methods used for SSVEP-based BCIs. Commonly used for power spectrum analysis or computing SSVEP phases. It is easy to implement and consumes small computation time. However, it was originally designed for linear and stationary signals. FFT is a fast computation algorithm for DFT. It usually needs long data segments for power estimation, which could influence practical applications. In real applications, the stimulation frequencies available might be limited because the frequency resolution is limited for a given data segment length. Reference [20] [21,22] average of normalized power of three channels [23] FFT for power and phase computation; combination of two features [24] computing SSVEP phase [58,63,65] [68] sum of power at fundamental and

4 302 J. Med. Biol. Eng., Vol. 34 No DFT Wavelet transform Wavelet packet decomposition Discrete Fourier transform. Its computation time is higher than that of FFT. Can be seen as an adjustable window Fourier analysis. Compared with Fourier transform, it has good time-frequency resolution and is suitable for processing non-stationary signals. However, it is still not better for nonlinear signal processing. Wavelet packet decomposition is an extension of wavelet transform. It decomposes the signal into a complete wavelet packet tree. second harmonics [69] projecting the Fourier coefficients onto reference phase directions; preliminary parameters adjustment [70] sum of normalized power at fundamental, second and third harmonics [71,72] [25] wavelet coefficients used for power computation [29] [27] CWT CWT means continuous wavelet transform. In real applications, the choice of mother wavelet is worth researching. EEDM + MFD EEMD was developed to overcome the mode-mixing problem of EMD. [30] It is suitable for non-stationary signal processing at the cost of higher computation time. MFD is used to estimate the power by computing the inner product between SSVEPs and the normalized complex-sinusoid signals. EMD + rgzc After decomposition by EMD, the refined generalized zero-crossing method is used to compute the instantaneous frequency. EMD can be used to reduce noise not related to SSVEP. EMD + Quadrature The combination of EMD and the quadrature detection method. The [32] detection method principle of the latter is quite similar to that of the MFD method. HT Hilbert transform is used to compute SSVEP phases after spatial [16,34] phase difference filtering with MEC. It needs a shorter data length than that for the Fourier method. HHT HHT is the combination of EMD and HT. Like EMD, it is suitable for [64] non-stationary and nonlinear signal processing with shorter data segments at the cost of higher computation time. MCC + HT The power at stimulation frequency is calculated after original data [33] being filtered by the MCC method. Then, HT is used to compute signal phases. MCC + CCA [39] MEC The SSVEPs are filtered by the MEC method, and then the estimated [11,36-38] power at each stimulation frequency is computed as feature variables. It can make better use of signals from different channels. MEC + Auto-regressive estimation MEC is used to compute signal power at stimulation frequencies. The auto-regressive mode is employed to estimate noise power. ACSP The analytic common spatial pattern method is applied to phase-coded [42] SSVEPs. It is an improvement of CSP. CCA-based method An effective method for computing the relation between two multi-variable data sets. Only the first maximum coefficient is used. The use of other canonical coefficients should be worth researching. CCA After linear combinations of original data, the two new signals are used [6,46] to compute the correlation coefficient. It has better classification accuracy than that of PSDA method. KCCA The kernel is used to project the data into a high-dimension space to [45] improve traditional CCA of which the two data sets are supposed to be linearly related. How to choose an appropriate kernel is worth discussing. Multiway CCA It uses the optimal reference signals after adjustment, but this increases [48] computation time. p-cca The phase information is used in the reference signals. [49] SFT Spectral F-test is implemented for feature extraction. Power spectral analysis is needed to build the test model. One of the proposed two tests does not need a baseline signal, making it better for direct use. [17,18,50-53] CCA + FFT PCA + High order statistics + Auto-regressive estimation PCC0 FFT is used to confirm the signs of weight vectors computed by CCA for phase-coded SSVEP. The high-order statistics method is used to confirm SSVEP-related and unrelated components decomposed by PCA. Then, the auto-regressive mode is used for power spectral density analysis. C0 complexity is used to detect idle state and work state (SSVEP state) when combined with the PCA method. SDFSs Six stepping delay flickering sequences are used to form a phase-coded [7] [31] instantaneous frequency and frequency contribution [56] combination of SNR of all channels at harmonic frequencies [66] [67] [40] [41]

5 Development of Algorithms for SSVEP-based BCIs 303 SSVEP application. The computation is based on the Fourier transform. Time-locked Calculates the amplitude peak between SSVEPs and reference signals epoch-average process for a phase-coded module. MFD MFD is used to estimate the power by computing the inner product between SSVEPs and the normalized complex-sinusoid signals. It needs less computation time compared with other methods. Its computation model is similar to that of CCA but suitable for single-channel signals. LASSO Assumes that SSVEPs are a standard linear regression model of stimulation signals. The hypothesis is quite similar to that of MEC or MCC. [8] [55] [54] Feature classification methods Value comparison Maximum value Minimum value Description Table 3. Feature classification methods used for SSVEP-based BCIs. The target is confirmed by comparing the values of feature variables. It is quite simple. It usually needs a threshold or a dwell time to reduce false positives, which increases computation time. Maximum value is used for frequency-coded SSVEPs with power values as feature variables. Minimum value is commonly used for phase-coded SSVEPs with phase lags between SSVEPs and the stimulation signals as feature variables. Reference Ensemble feature model The canonical coefficients at different harmonic frequencies are linearly combined to confirm the work state if the combined value is over the threshold. Decision tree A rule-based decision tree is used to classify different targets after feature extraction by [17,18,50-53] spectral F-test method. Classifier This method uses different classifiers to train offline data. Then, the classifiers are used to classify testing data. The training procedure increases preparation time. LDA classifier This classifier is simple but effective. [42] Linear Fisher The principle of a Fisher classifier is to find the best direction of a straight line, and the classifier feature variables can be separated by projection onto this line. PNN A radial basis network that uses the Parzen window to estimate the probability density function of each class with labeled training data. SVM The support vector machine determines the classification boundary adaptively and is suitable for two classes. Linear classifier + Alpha The alpha wave detection is used when the stimulus frequency is between 8-12 Hz to wave detection reduce false positives. Phase direction The signs of weight coefficients are used to recognize different targets of phase-coded SSVEP. [6,23,31,32,37], [40,41,46,48,49,54], [56,58,63,68,69,71] [7,11,30,36,38], threshold [55,65,66,72] [20-22,70] dwell time [8] fifteen consecutiv e results made a decision [24] [45] threshold [25] three consecutive results made a decision [29,64] [16,33] [27,34] [39] [67] Spatial filtering preprocesses multiple channel signals in the spatial domain. The principle of spatial filtering is to enhance stimulation-related signals or to reduce other nuisance signals through linear combinations of signals from different channels. MEC and maximum contrast combination (MCC) are the most frequently used methods for spatial filtering (see Table 1). Their objective functions used to compute the weight coefficients for spatial filters are different. MEC tries to eliminate activity unrelated to SSVEP whereas MCC tries to maximize the SNR. They are used in 43% of papers reviewed that employ spatial filtering. Other methods, such as canonical correlation analysis (CCA), common average reference (CAR), and principal component analysis (PCA), have also been utilized. Of note, even for multi-channel SSVEPs, frequency filtering is always used together with spatial filtering. Another advantage of spatial filtering is that it combines signal preprocessing and feature extraction. Spatial filtering techniques used for feature extraction are discussed in the following section. 3.2 Feature extraction Table 4. Number of times typical methods were used for feature extraction. Methods Number of reviewed studies FT WT HT EMD MEC MCC CCA SFT As mentioned, the signal processing algorithm is of vital importance to the whole BCI system, and feature extraction is a key issue. A variety of methods have been used (see Table 2). Table 4 shows how often the typical methods were used for feature extraction based on Table 2. Here, the fast Fourier transform (FFT) and the discrete Fourier transform (DFT) are

6 304 J. Med. Biol. Eng., Vol. 34 No combined into Fourier-based transform. The Hilbert-Huang transform (HHT) includes empirical mode decomposition (EMD) and Hilbert transform (HT), so it was counted once for each of these two methods. These methods are discussed in detail below. In order to clearly compare the differences between these approaches, the analysis methodology of using partial conclusion is employed. The feature extraction methods for single- and multi-channel signal processing are summarized in the sub-sections Partial conclusion of single-channel signal processing methods and Partial conclusion of multi-channel signal processing methods Fourier-based transform Fourier-based transform methods are mostly used for power spectrum density analysis (PSDA). Their advantages include simplicity and small computation time. As shown in Table 4, 13 papers employed Fourier-based transforms. Most of them used the transforms to compute the power at the stimulation frequencies and their harmonics for frequencycoded SSVEPs. In one study, the average power centered on the stimulation frequency was calculated [20], and then the frequency with the maximum power value was confirmed as the target one. In some research, the power was equalized with a baseline EEG obtained when subjects did not focus on any stimulus [21,22]. However, this method is time-consuming. For multi-channel signals, the optimal electrode for different subjects can be chosen using preliminary experiments with the FFT method. In some papers [23,24], the FFT was employed to compute the SSVEP phases for phase-coded systems. In the experiments, the detection accuracy increased with increasing length of the SSVEP epoch. Frequency resolution is a key issue for the FFT method. It is defined as the sampling frequency divided by the data length. Thus, the time window length of SSVEP signals needs to be long enough to enhance the frequency resolution when the sampling frequency is fixed. This might limit practical applications because it causes a lower ITR [25]. Additionally, a larger window length could cause classification errors during a change of stimuli. This is due to the memory effect of using EEG signals corresponding to a previous class in the current classification procedure. Zero padding is often used for FFT computing. However, it only enhances the visual frequency resolution rather than the physical frequency resolution. Fourier transforms were originally designed for linear and stationary signals [26]. SSVEP signals are non-stationary and their frequency components vary as a function of time [27,28]. It is therefore usually presumed that the individual segments of the EEG signal can be approximately considered as being linear and stationary signals. This presumption may affect the processing results Wavelet transform Wavelet transforms are based on Fourier transforms. They are an adjustable-window Fourier analysis [26], and provide good time-frequency resolution, so are suitable for processing non-stationary SSVEP signals. Many studies [25,27,29] have used wavelet analysis. The original signals are decomposed into several components corresponding to different frequency ranges. Then, the components containing the stimulation frequencies are extracted for further processing. The wavelet coefficients are used to estimate power at relevant frequency points. A key problem with applying wavelet analysis is how to choose an appropriate mother wavelet to attain good performance. One study [27] experimented with different kinds of wavelet, and found that the complex Morlet wavelet was best. This is an empirical method for choosing the wavelet. How to find wavelets suitable for SSVEP in less time while retaining high performance is worth researching. Although wavelet analysis is better for non-stationary signal processing compared with Fourier transform, it is based on the Fourier transform. Therefore, methods suitable for nonlinear and non-stationary signal processing, such as the HTT, are needed Hilbert-Huang transform The HHT was designed for nonlinear and non-stationary signals [26]. It includes EMD and HT. EMD is used to decompose a signal into a number of intrinsic mode functions (IMFs). An IMF is an oscillatory function with time-varying frequencies that can represent the local characteristics of non-stationary signals [30]. Studies [31,32] have used EMD to decompose original SSVEPs into several IMFs. SSVEP-related IMFs were selected by computing the instantaneous frequency, and then the frequency with the maximum presence probability and closest to the stimulation frequency was identified as the target [31]. In another study [32], all IMFs were used to find which one had the maximum correlation index for various stimulation frequencies. Ensemble empirical mode decomposition (EEMD) was developed to overcome the modemixing problem of EMD caused by signal intermittences [26,30]. EEMD decomposes the original signal added by white noise into several IMFs. In the HHT, the HT is used after EMD. The HHT is more stable than the FFT, which means that its recognition accuracy does not change greatly with data length. Although the HHT can be well used for nonlinear and nonstationary SSVEPs, its computation time is higher than that of the Fourier transform. The HT has also been employed to compute SSVEP phases [16,33,34] Partial conclusion of single-channel signal processing methods Fourier-based transforms are quite popular for calculating the power values corresponding to stimulation frequencies. One advantage of this method is its simplicity and small computation time. The number of harmonics used for calculating this power value varies. The fundamental, second, and third harmonic frequency power values can be summed, or a weight vector for all harmonics can be used. One study [6] conducted experiments on the number of harmonics used for classification, and showed that it had no significant impact on performance. However, another study [35] stated that the use of the fundamental, second, and third harmonic frequencies yielded higher classification accuracy. Most researchers utilized the fundamental and second harmonic frequencies when processing SSVEPs. The FFT can also be used for calculating

7 Development of Algorithms for SSVEP-based BCIs 305 SSVEP phases. The Fourier transform was originally designed for linear and stationary signals, while SSVEPs are nonlinear and nonstationary. Therefore, some researchers have used the wavelet transform or the HHT for SSVEP signal processing, which are suitable for linear, non-stationary and nonlinear, non-stationary signals Minimum energy combination and maximum contrast combination All the methods discussed above are commonly employed to process single-channel SSVEPs. However, there is the problem of inter-subject variation, which can be overcome by determining the optimal electrode combination. Signals from multi-channel EEG are less affected by noise than those from a unipolar or bipolar system. The combination of signals collected from different channels (electrodes) is also referred to as spatial filtering. As shown in Table 4, 8 of the reviewed papers used this technique. MEC and MCC are the most commonly utilized methods. Through linear combinations of multi-channel signals, the nuisance signals can be greatly reduced and the ratio between SSVEP-related signals to unrelated signals can be enhanced. As mentioned, spatial filtering methods can be used for signal preprocessing. After being combined, obtained signals can be processed with other algorithms as single signals. MEC or MCC can be used to compute the power [11,36-38] or SNR [39] at the stimulation frequencies and their harmonics. That is to say, signal preprocessing and feature extraction are conducted at the same time, reducing computation time. Besides MEC and MCC, PCA [40], principal component C0 complexity (PCC0) [41], and analytic common spatial pattern (ACSP) [42] have also been used by researchers. Several methods for multi-channel signal feature extraction were investigated [43], and the results showed that the MEC method achieved the best accuracy Canonical correlation analysis MEC and MCC are types of spatial filtering used for feature extraction. The basic idea is to compute the power or SNR after the multi-channel signals are combined into one signal. CCA can also be employed to extract features. The CCA method is used to determine the relationship between two sets of data, which can be considered as the development of traditional correlation between two variables. In real life, it is necessary to study this kind of multivariate correlation. The potential and feasibility of CCA for SSVEP-based BCIs are currently being investigated. CCA is a multivariable statistical method used when there are two sets of data which may have some underlying correlation [44]. A pair of linear transformations are found, and then the corresponding correlation of the two sets of data is maximized [45]. The authors in paper [46] stated that they are the first researchers to use CCA to recognize SSVEP. For practical applications, the correlation between SSVEPs and reference signals, which included sine and cosine components, was computed [6,46]. A comparison between the CCA and PSDA methods was done by Hakvoort et al. [47], who found that CCA had better performance. However, standard CCA supposes that the two sets of data are linearly related. A kernel CCA (KCCA) method has thus been proposed [45]. It projects data into a high-dimension space, but the inner product of the new data can be still calculated using the original low-dimension data. Another paper [48] proposed a multiway CCA method to find optimal reference signals rather than sine and cosine waves in order to consider the inter-subject variability and trial-to-trial variability. An approach called phase-constrained canonical correlation analysis (p-cca) [49] improves the reference signals of standard CCA by taking the SSVEP phase into account. Only the first maximum canonical coefficient was used in almost all reviewed papers. The use of other coefficients or canonical variables is worth researching Partial conclusion of multi-channel signal processing methods Spatial filtering methods such as MEC and MCC were utilized to extract features for multi-channel SSVEPs, and the experimental results show improvement over single-channel SSVEP signal processing methods. The CCA method calculates the correlation between SSVEPs and the stimulation signals to discriminate between different evoked potentials. One study [6] showed that CCA outperforms traditional PSDA methods. Several studies [45,48,49] proposed improved CCA methods that achieved good results. However, these studies used only the maximum canonical correlation coefficient. The use of other canonical coefficients or canonical variables should be investigated Spectral F-test Muller et al. used the spectral F-test (SFT) for feature extraction. Two implementations were used, one for timelocked changes and the other for phase-locked changes. Some studies [17,50,51] have used the first SFT, which was based on the relation mode between EEG signals before stimulation (rest period) and during stimulation. The ratio between states without and with stimulation was calculated. Other studies [18,52] have used the second SFT, which computed the ratio between the power at the stimulation frequency point and the average power in its neighbouring frequencies. The critical value of the F-test was used in both types of SFT to confirm the presence of evoked responses. Then, the decision tree was utilized to identify different stimulation targets. However, the second SFT does not require a rest EEG signal, which is an advantage for practical applications as it reduces computational cost. One study [53] implemented both SFTs and found that the second is more suitable for smaller data segments. For SSVEP-base BCI applications that didn t use the SFT method, a calibration or training procedure is always needed to adjust some parameters, such as the threshold used to distinguish the idle state from the work state. This can increase the preparation time required before the actual online experiment can be started and is inconvenient for BCI users. The second SFT implementation overcomes this problem.

8 306 J. Med. Biol. Eng., Vol. 34 No Other methods Some other techniques have also been proposed. Sparse signal processing is especially useful when few samples are available [54]. Least absolute shrinkage and selection operator (LASSO) assumes that SSVEPs are a standard linear regression model of stimulation signals, and its hypothesis is quite similar to those of spatial filtering methods such as MEC. One study [54] proposed a LASSO method for computing the contribution degree of different stimulus frequencies and harmonics to EEG signals. The target frequency was recognized as that with the maximum contribution degree. The results showed that the LASSO method had better performance and a shorter time window length was needed compared to those of the CCA method. The matched filter detector (MFD) method was proposed [55] for computing the inner product between SSVEPs and normalized complex-sinusoid signals. The frequency with the maximum inner product is confirmed as the target stimulation. The method is based on the assumption that SSVEPs are a combination of stimulation signals and noise. One study [30] applied the MFD method after the EEMD method. The idea of MFD is quite similar to that of the CCA method. Both of them try to find the maximum correlation between SSVEPs and the stimulation signals. The difference lies in that they are suitable for dealing with multi-channel and single-channel signals, respectively. Other methods such as auto-regressive (AR) estimation have also been used by researchers [40,56]. They are not so commonly used compared with the aforementioned methods, and are thus not discussed in detail. 3.3 Feature classification From Table 3, the two frequently used methods for feature classification are value comparison and the classifier method. Almost 67% of the reviewed papers employed the value comparison method. With value comparison, the feature variables extracted correspond to different stimulation signals. The target is confirmed by comparing these values and finding the maximum or minimum one. The maximum value of the feature variables is often used for frequency-coded SSVEPs, when the feature is the power at the stimulation frequencies and their harmonics. However, when the feature variable represents the phase difference between SSVEPs and the stimulation signals, the minimum value is used to identify the target. The value comparison method is quite easy to understand and implement. Of note, this method is always used with a threshold or a dwell time, especially in asynchronous BCIs, to reduce false positives. If the feature value is over a specific value or several consecutive choices have been identified, the corresponding command is confirmed. The threshold or dwell time is usually obtained from preliminary experiments. This procedure might increase the preparation time required for online experiments. Typical classifiers, such as the linear Fisher classifier, were employed in some research. Features corresponding to different stimulations are used as a feature vector to train the classifier with offline data. Finally, the online experiment is conducted with the classifier to identify different targets. Compared with the value comparison method, the classifier method needs offline data training, which can increase preparation time. However, a threshold is not necessary for this method. Other techniques have also been used. One study [45] used an ensemble feature model for detecting idle and work states. The canonical coefficients at different harmonic frequencies were linearly combined. If the value was over a threshold, then the work state was confirmed. Both weights and the threshold were trained with offline data. A decision tree was used when the feature was extracted using the F-test method [17,18,50-53]. The classifier method is more complex than the value comparison method. It often involves training with testing data to calibrate some parameters of the system. One study [23] utilized linear discriminant analysis (LDA) to obtain a weight vector for combining frequency and phase features. 3.4 Other key factors Besides the algorithms themselves, other factors, such as the frequency range, stimulation type, calibration procedure, parameter adjustment, and number of harmonics, affect the performance of the system. One study [57] did an experiment to illustrate this impact, and another study [58] showed that accuracy increases with the time window length of the SSVEP epoch. One study [59] tested the effect of stimulus specificity on the classification performance of an SSVEP-based BCI. Similarly, another study [60] conducted experiments to show the effects of stimulus parameters such as colour and frequency of flicker. Low- and medium-frequency flickering may lead to visual fatigue or even induce epileptic seizures [15]. For safety and comfort, higher stimulation frequencies are preferable [16]. A review paper [61] pointed out that BCIs using LEDs for stimulation had higher ITR than those using graphic stimuli. The setup of the experiment can also affect performance. A dictionary-driven SSVEP speller resulted in a significant increase in spelling performance compared with other SSVEPbased spelling systems, especially in terms of ITR [62]. One study [63] found that the SSVEP amplitude differs among individuals, which means adjustments have to be made. 4. Conclusion This literature review compared the signal processing algorithms used in SSVEP-based BCIs to outline their development. The results can serve as a guideline and reference for future work. The procedure of SSVEP signal processing usually includes signal preprocessing, feature extraction, and feature classification. Among the reviewed papers, band-pass and notch filters are the most commonly employed to eliminate noise. Spatial filtering is utilized especially for multi-channel signals, which can reduce the influence of inter-subject variation. Regarding feature extraction, the power at the stimulation frequencies and harmonics and the SSVEP phases are typical

9 Development of Algorithms for SSVEP-based BCIs 307 feature variables for frequency-coded and phase-coded SSVEPs. CCA, LASSO, MFD, and spatial filtering methods are also typically used. The relationship between SSVEPs and the reference signals, which usually include standard sine and cosine waves, is computed. FFT is quite popular for calculating the power corresponding to different stimulation frequencies because of its simplicity and small computation time. FFT can be used for calculating SSVEP phases. However, the Fourier transform was originally designed for linear and stationary signals. Therefore, some researchers have used the wavelet transform or the HHT for SSVEP signal processing. These are suitable for linear, nonstationary and nonlinear, non-stationary signals, respectively. Single-channel signals are easily polluted by noise. In this condition, multi-channel signals have advantages. Spatial filtering methods, such as MEC and MCC, have been utilized to extract features for SSVEPs, with experimental results showing improvement. For multi-channel signals, computing the correlation between SSVEPs and the stimulation signals can be adopted. CCA and LASSO are typically used. To avoid calibration or an adjustment procedure, SFT can be used. As for classification methods, the value comparison method and the classifier method are commonly used. The classifier method is more complex than the value comparison method, as it usually needs a training procedure to calibrate some parameters before online experiments. Each proposed algorithm is suitable for use in specific applications. Besides research on algorithms used for signal processing, it is important to use these methods efficiently in practical BCI systems, which should be easy to command by users. The goals should be lower preparation time, a userfriendly interface, and subject comfort. The systems have to benefit disabled people, but most methods in the reviewed papers were tested with healthy people. Of course, they can also be used by ordinary people in other fields such as entertainment. Acknowledgments This research was sponsored by the China Scholarship Council (CSC), the International Collaborative Project of Nature Science Foundation of Hubei Province, China (grant 2009BFA006) and the Fundamental Research Funds for the Central Universities (grant 2012-IV-088). The authors would like to thank Dr. Ryan McCardle and Dr. Xing Song for their proofreading. Appendix Index term SSVEP BCI SNR ITR EEG SCP SMR MEC List of abbreviations Description Steady-state visual evoked potential Brain computer interface Signal-to-noise ratio Information transfer rate Electroencephalography Slow cortical potential Sensorimotor rhythm Minimum energy combination MCC CCA CAR PCA FFT DFT CWT HHT EMD rgzc HT PSDA IMF EEMD PCC0 SDFS ACSP KCCA p-cca SFT LASSO MFD LDA PNN SVM References Maximum contrast combination Canonical correlation analysis Common average reference Principal component analysis Fast Fourier transform Discrete Fourier transform Continuous wavelet transform Hilbert-Huang transform Empirical mode decomposition Refined generalized zero crossing Hilbert transform Power spectral density analysis Intrinsic mode function Ensemble empirical mode decomposition Principal component C0 complexity Stepping delay flickering sequence Analytic common spatial pattern Kernel canonical correlation analysis Phase constrained canonical correlation analysis Spectral F-test Least absolute shrinkage and selection operator Matched filter detector Linear discriminant analysis Probabilistic neural network Support vector machine [1] J. R. Wolpaw, N. Birbaumer, W. J. Heetderks, D. J. McFarland, P. H. Peckham, G. Schalk, E. Donchin, L. A. Quatrano, C. J. Robinson and T. M. Vaughan, Brain-computer interface technology: a review of the first international meeting, IEEE Trans. Rehabil. Eng., 8: , [2] S. G. Mason and G. E. Birch, A general framework for brain-computer interface design, IEEE Trans. Neural Syst. Rehabil. Eng., 11: 70-85, [3] E. Pasqualotto, S. Federici and M. O. Belardinelli, Toward functioning and usable brain-computer interfaces (BCIs): a literature review, Disabil. Rehabil. Assist. Technol., 7: , [4] Y. J. Wang, R. P. Wang, X. R. Gao, B. Hong and S. K. Gao, A practical VEP-based brain-computer interface, IEEE Trans. Neural Syst. Rehabil. Eng., 14: , [5] D. Regan, Human brain electrophysiology: evoked potentials and evoked magnetic fields in science and medicine, J. Clin. Neurophysiol., 7: , [6] G. Y. Bin, X. R. Gao, Z. Yan, B. Hong and S. K. Gao, An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method, J. Neural. Eng., 6: , [7] H. C. Chang, P. L. Lee, M. T. Lo, I. H. Lee, T. K. Yeh and C. Y. Chang, Independence of amplitude-frequency and phase calibrations in an SSVEP-based BCI using stepping delay flickering sequences, IEEE Trans. Neural Syst. Rehabil. Eng., 20: , [8] P. L. Lee, J. J. Sie, Y. J. Liu, C. H. Wu, M. H. Lee, C. H. Shu, P. H. Li, C. W. Sun and K. K. Shyu, An SSVEP-actuated brain computer interface using phase-tagged flickering sequences: a cursor system, Ann. Biomed. Eng., 38: , [9] M. Cheng and S. K. Gao, An EEG-based cursor control system, Proc. 1 st Joint BMES/EMBS IEEE Engineering in Medicine and Biology 21 st Annual Conf. and Annual Fall Meeting of the Biomed. Eng. Society Conf., 1: 669, [10] Z. D. Deng, X. Q. Li, K. H. Zheng and W. T. Yao, A humanoid robot control system with SSVEP-based asynchronous brain-computer interface, Jiqiren/Robot, 33: , [11] O. Friman, T. Luth, I. Volosyak and A. Graser, Spelling with steady-state visual evoked potentials, Proc. 3 rd Int. IEEE/EMBS Conf. on Neural Eng., , [12] K. Shyu, Y. J. Chiu, P. L. Lee, M. H. Lee, J. J. Sie, C. H. Wu, Y. T. Wu and P. C. Tung, Total design of an FPGA-based brain computer interface control hospital bed nursing system, IEEE Trans. Ind. Electron., 60: , 2013.

10 308 J. Med. Biol. Eng., Vol. 34 No [13] G. Garcia-Molina and D. H. Zhu, Optimal spatial filtering for the steady state visual evoked potential: BCI application, Proc. 5 th Int. IEEE/EMBS Conf. on Neural Engineering, , [14] T. Picton, Human brain electrophysiology: evoked potentials and evoked magnetic fields in science and medicine, J. Clin. Neurophysiol., 7: , [15] R. S. Fisher, G. Harding, G. Erba, G. L. Barkley and A. Wilkins, Photic- and pattern- induced seizures: a review for the epilepsy foundation of America working group, Epilepsia, 46: , [16] G. Garcia-Molina, D. H. Zhu and S. Abtahi, Phase detection in a visual-evoked-potential based brain computer interface, Proc. 18 th European Signal Processing Conf., , [17] S. M. T. Muller, T. F. Bastos and M. Sarcinelli, Using a SSVEP-BCI to command a robotic wheelchair, Proc. IEEE Int. Symp. on Industrial Electronics, , [18] S. M. Müller, P. F. Diez, T. F. Bastos-Filho, M. Sarcinelli-Filho, V. Mut and E. Laciar, SSVEP-BCI implementation for Hz frequency range, Proc. Annual Int. Conf. of the IEEE Eng. in Medicine and Biol. Society, , [19] P. F. Diez, A. G. Correa and E. L. Leber, SSVEP detection using adaptive filters, Proc. 5 th Latin American Congress on Biomedical Engineering, 33: , [20] G. R. Muller-Putz and G. Pfurtscheller, Control of an electrical prosthesis with an SSVEP-based BCI, IEEE Trans. Biomed. Eng., 55: , [21] P. F. Diez, V. A. Mut, E. M. Avila Perona and E. Laciar Leber, Asynchronous BCI control using high-frequency SSVEP, J. Neuroeng. Rehabil., 8: 39, [22] P. F. Diez, S. M. Torres Müller, V. A. Mut, E. Laciar, E. Avila, T. F. Bastos-Filho and M. Sarcinelli-Filho, Commanding a robotic wheelchair with a high-frequency steady-state visual evoked potential based brain-computer interface, Med. Eng. Phys., 35: , [23] C. M. Wong, B. Y. Wang, F. Wan, P. U. Mak, P. I. Mak and M. I. Vai, A solution to harmonic frequency problem: frequency and phase coding-based brain-computer interface, Proc. Int. Joint Conf. on Neural Networks, , [24] C. M. Wong, B. Y. Wang, F. Wan, P. U. Mak, P. I. Mak and M. I. Vai, An improved phase-tagged stimuli generation method in steady-state visual evoked potential based brain-computer interface, Proc. 3 rd Int. Conf. on Biomedical Engineering and Informatics, 2: , [25] Y. Bian, H. W. Li, L. Zhao, G. H. Yang and L. Q. Geng, Research on steady state visual evoked potentials based on wavelet packet technology for brain-computer interface, Proc. Eng., 15: , [26] N. E. Huang, Z. Shen, S. R. Long, M. C. Wu, H. H. Shih, Q. Zheng, N. C. Yen, C. C. Tung and H. H. Liu, The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis, Proc. Roy. Soc. Lond., 454: , [27] Z. Zhang, X. Li and Z. Deng, A CWT-based SSVEP classification method for brain-computer interface system, Proc. Int. Conf. on Intelligent Control and Information Processing, 43-48, [28] V. J. Samar, A. Bopardikar, R. Rao and K. Swartz, Wavelet analysis of neuroelectric waveforms: a conceptual tutorial, Brain Lang, 66: 7-60, [29] B. Yan, Z. Li, H. Li, G. Yang and H. Shen, Research on braincomputer interface technology based on steady state visual evoked potentials, Proc. 4 th Int. Conf. on Bioinformatics and Biomedical Engineering, 1-4, [30] P. L. Lee, H. C. Chang, T. Y. Hsieh, H. T. Deng and C. W. Sun, A brain-wave-actuated small robot car using ensemble empirical mode decomposition-based approach, IEEE Trans. Syst. Man Cybern. Syst. Hum., 42: , [31] C. H. Wu, H. C. Chang, P. L. Lee, K. S. Li, J. J. Sie, C. W. Sun, C. Y. Yang, P. H. Li, H. T. Deng and K. K. Shyu, Frequency recognition in an SSVEP-based brain computer interface using empirical mode decomposition and refined generalized zerocrossing, J. Neurosci. Methods, 196: , [32] C. H. Wu, H. C. Chang and P. L. Lee, Instantaneous gaze-target detection by empirical mode decomposition: application to brain computer interface, Proc. World Congress on Medical Physics and Biomedical Engineering, 25: , [33] D. Zhu, G. Garcia-Molina, V. Mihajlović and R. M. Aarts, Online BCI implementation of high-frequency phase modulated visual stimuli, Universal Access in Human- Computer Interaction. Users Diversity, 6766: , [34] D. H. Zhu, G. Garcia-Molina, V. Mihajlović and R. M. Aartsl, Phase synchrony analysis for SSVEP-based BCIs, Proc. 2 nd Int. Conf. on Computer Engineering and Technology, 2: V2329- V2333, [35] G. R. Muller-Putz, R. Scherer, C. Brauneis and G. Pfurtscheller, Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components, J. Neural. Eng., 2: , [36] I. Volosyak, SSVEP-based Bremen-BCI interface-boosting information transfer rates, J. Neural. Eng., 8: , [37] I. Volosyak, T. Malechka, D. Valbuena and A. Graser, A novel calibration method for SSVEP based brain-computer interfaces, Proc. 18 th European Signal Proc. Conf., , [38] H. Cecotti, A self-paced and calibration-less SSVEP-based brain-computer interface speller, IEEE Trans. Neural Syst. Rehabil. Eng., 18: , [39] N. Y. Wang, T. Y. Qian, Q. Zhuo and X. R. Gao, Discrimination between idle and work states in BCI based on SSVEP, Proc. 2 nd IEEE Int. Conf. on Advanced Computer Control, 4: , [40] S. Pouryazdian and A. Erfanian, Detection of steady-state visual evoked potentials for brain-computer interfaces using PCA and high-order statistics, Proc. World Congress on Medical Physics and Biomedical Engineering, 25: , [41] R. Ren, G. Y. Bin and X. R. Gao, Idle state detection in SSVEP-based brain-computer interfaces, Proc. 2 nd Int. Conf. on Bioinformatics and Biomedical Engineering, , [42] O. Falzon, K. Camilleri and J. Muscat, Complex-valued spatial filters for SSVEP-based BCIs with phase coding, IEEE Trans. Biomed. Eng., 59: , [43] O. Friman, I. Volosyak and A. Graser, Multiple channel detection of steady-state visual evoked potentials for braincomputer interfaces, IEEE Trans. Biomed. Eng., 54: , [44] H. Hotelling, Relations between two sets of variates, Biometrika, 28: , [45] Z. M. Zhang and Z. D. Deng, A kernel canonical correlation analysis based idle-state detection method for SSVEP-based brain-computer interfaces, Proc. 2 nd Int. Conf. on Material and Manufacturing Technology, : , [46] Z. L. Lin, C. S. Zhang, W. Wu and X. R. Gao, Frequency recognition based on canonical correlation analysis for SSVEPbased BCIs, IEEE Trans. Biomed. Eng., 53: , [47] G. Hakvoort, B. Reuderink and M. Obbink, Comparison of PSDA and CCA detection methods in a SSVEP-based BCI-system, Available: /01/Comparison_of_PSDA_and_CCA_detection_methods _in_a_ssvep-based_bci-system.pdf [48] Y. Zhang, G. X. Zhou, Q. B. Zhao, A. Onishi, J. Jin, X. Y. Wang and A. Cichocki, Multiway canonical correlation analysis for frequency components recognition in SSVEP-based BCIs, Proc. 18 th Int. Conf. on Neural Information Processing, , [49] J. Pan, X. Gao, F. Duan, Z. Yan and S. Gao, Enhancing the classification accuracy of steady-state visual evoked potentialbased brain-computer interfaces using phase constrained canonical correlation analysis, J. Neural. Eng., 8: , [50] S. M. T. Müller, W. C. Celeste, T. F. Bastos-Filho and M. Sarcinelli-Filho, Brain-computer interface based on visual evoked potentials to command autonomous robotic wheelchair, J. Med. Biol. Eng., 30: , [51] S. M. T. Muller, T. F. Bastos and M. Sarcinelli, Incremental SSVEP analysis for BCI implementation, Proc. Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, , 2010.