SIXTH FRAMEWORK PROGRAMME PRIORITY 2 Information Society Technologies. Contract No Sensing seat for human authentication module

SIXTH FRAMEWORK PROGRAMME PRIORITY 2 Information Society Technologies Contract No. 026990 Sensing seat for human authentication module Deliverable D3.5 (Final M18) Workpackage No. WP3 Workpackage Title Behavioural and other biometrics profile creation Task No. T3.5 Task Title Sensing seat for human authentication Authors Dr. Giovanni Pioggia (UNIPI) Dr. Marcello Ferro (UNIPI) Status: F: final D: draft RD: revised draft F File name HUMABIO_D3_5_final_R1.pdf Project start date and duration 01 January 2006, 30 months June, 2006 1 UNIPI

Contents 1 Introduction 11 1.1 Chapters Outline.............................. 11 1.2 A Sensing Seat for Human Authentication................. 12 2 State of the art analysis 14 2.1 Tekscan................................... 14 2.2 Novel.................................... 15 2.3 Euralab by Wronz.............................. 15 2.4 ComfortLab by Johnson Controls...................... 16 3 UNIPI Strain Sensors 17 3.1 Description and Realisation of the Sensors and the Connections...... 17 3.2 Test on a Single Sensor........................... 18 3.3 Sensor Net.................................. 20 3.3.1 Series Network........................... 21 3.3.2 Parallel Network.......................... 22 3.3.3 Quadripolar Network....................... 22 3.4 Sensor Patch Design and Realization.................... 23 3.5 Data Acquisition Board Front-End..................... 23 4 Software Framework 26 4.1 Software Analysis Using a Single Sensor Array Patch........... 26 4.2 Results Obtained with a Single Sensor Array................ 27 4.3 Single Sensor Array Patch Conclusions.................. 29 5 Sensing Seat Architecture 30 5.1 Sensing Seat Cover Design......................... 30 5.2 Recording Protocol Definition....................... 30 5.2.1 Predefined positions........................ 32 5.2.2 Initialization............................ 32 5.2.3 Enrolment.............................. 33 5.2.4 Authentication........................... 33 5.3 Human Profile Signature.......................... 33 5.3.1 Client Application Signature Structure............... 34 5.4 Database Architecture............................ 35 5.5 Final Sensing Seat Prototype........................ 36 5.6 Software Analysis.............................. 36 5.6.1 Classification algorithms...................... 38 5.6.2 Global classification module.................... 38 2

5.6.3 Personal classifiers......................... 39 5.6.4 Enrolment State........................... 39 5.6.5 Authentication State........................ 40 5.6.6 Monitoring State.......................... 41 5.7 Classification Software Overview...................... 42 5.7.1 Enrolment State........................... 43 5.7.2 Authentication State........................ 44 5.7.3 Monitoring State.......................... 44 5.7.4 Communication Architecture.................... 44 5.8 Hardware specifications........................... 44 6 Measurment Campaign and Experimental Results 46 6.1 Data Analysis................................ 47 6.2 Limits of the existing architecture..................... 48 7 Future work 50 June, 2006 3 UNIPI

List of Figures 2.1 Standard Tekscan sensor.......................... 14 2.2 Resistance vs. force for Softswitch pressure sensors............ 15 2.3 ComfortConforms system.......................... 16 3.1 Transduction principle of the strain sensor................. 18 3.2 a) A pressure impulse; b) Sensor response a pressure impulse and selected features................................... 19 3.3 a) Acquired signal during pressure deformations (thin line) and extracted exponential law (bold dotted line); b) Quadratic error........... 19 3.4 Proposed equivalent electric model of each sensor............. 20 3.5 Values of the parameters of the equivalent electric model extracted from ten cycles of a reference experimental signal................ 21 3.6 Series network............................... 21 3.7 Parallel network............................... 22 3.8 Quadripolar network............................ 22 3.9 A single sensor array patch......................... 23 3.10 a) The mask used for the sensing cover; b) The equivalent electric scheme of sensing cover............................... 24 3.11 Constant current generator......................... 24 4.1 The mechanical platform.......................... 27 4.2 Learning phase: the presence of a high amplitude rhythm......... 28 4.3 Learning phase: the spiking rhythm becomes uncorrelated........ 28 5.1 The sensing seat: a) truck seat provided by VOLVO; b) seat cover (the sensors will be placed on the back side of the cover)............ 31 5.2 Final prototype of the sensing seat: the sensor arrays are embedded into a Lycra tissue, placed in the back side of the seat cover. The sensing cover is placed over the truck seat......................... 31 5.3 Two predefined positions currently taken into account; a) the subject is fully seated; b) The subject is not in contact with the back side of the sensing cover................................ 33 5.4 The final sensing seat prototype without the external cover........ 37 5.5 The classification software. The GUI is visible at the top-left, showing the real-time acquired signals from the sensing system, the classification result for all the classification sub-modules and the final authentication result. The state of the artificial neural network modules are shown in the bottom-right window. The signals acquired in real-time are also shown in the top-right window............................. 37 5.6 Block diagram of the sensing seat authentication system......... 38 4

5.7 Flow chart of the enrolment task...................... 40 5.8 Flow chart of the authentication task.................... 40 5.9 Flow chart of the monitoring task...................... 41 6.1 Experimental setup............................. 46 6.2 Global classification module: each colour corresponds to an enrolled subject; a) KSOM; b) MLP........................... 47 6.3 Personal classification module; a) KSOM; b) MLP............ 47 6.4 Details of a personal classification module belonging to a particular subject; a) KSOM; b) MLP........................... 49 June, 2006 5 UNIPI

List of Tables 3.1 Considered analogous features....................... 20 4.1 Confusion matrix reporting ANN classification results........... 29 6.1 Acceptance and rejection rates using the personal classification modules. Values are averaged over the classification modules belonging to all the enrolled subjects. Each classification module stores internally the threshold level the corresponds to the equal error rate.............. 48 June, 2006 6 UNIPI

List of abbreviations CE ANN SOM KSOM MLP TNGS STDP PCA LDA DAQ GUI LIFO API FAR FRR EER FPS BCM Conductive Elastomer Artificial Neural Network Self Organizing Map Kohonen Self Organizing Map Multi Layer Perceptron Neuronal Group Selection Spike-Timing-Dependant Plasticity Principal Component Analysis Linear Discriminant Analysis Data AcQuisition board Graphical User Interface Last In, First Out Application Programming Interface False Acceptance Rate False Rejection Rate Equal Error Rate Frames Per Second Biometric Confusion Matrix June, 2006 7 UNIPI

Executive Summary During the first reporting period (January 1 st, 2006 to January 1 st, 2007) of the HUM- ABIO project, UNIPI focused on the realisation of the preliminary prototype of the sensing seat for human authentication as well as on the development of the algorithms for the enrollment, authentication and monitoring task. As the final prototype of the sensing seat was completed, a new measurement campaign was conducted in order to assess the classification capabilities of the system (June 30 th, 2007). The analysis of the state of the art was the starting point. Several companies are currently working in order to realize comfortable interactive seats. These systems, some of them already on the market, use different technologies and materials, but they share the use of sensors that measure the pressure exerted on the seat by the passenger. Actually, no existing systems based on sensing seats are currently able to perform the human authentication task and no results on this topic, even if in a preliminary stage, were found. Indeed the sensor technology used in such systems in not adequate to perform the authentication task. In order to detect and correctly classify the human profile signature, many sensors should be used and the hardware architecture will result in a complex and delicate system. As a result, no database of collected data of human profiles was found and for this reason the database structure of the UNIPI sensing seat data was built from scratch. The realisation of a sensing cover equipped with redundant strain sensors is the idea of the UNIPI sensing seat for human authentication. The sensors we develop allow piezoresistive sensing fabrics to be realized. The non-commercial sensors used for this application are currently being developed at UNIPI labs. Conductive Elastomers (CEs) composites show piezoresistive properties when a deformation is applied. In order to develop strain sensors, they can be integrated into fabric or other flexible substrates. The CE we used is based on a WACKER Ltd (Elastosil LR 3162 A/B) product. It consists in a mixture of graphite and silicon rubber. WACKER Ltd guarantees the non-toxicity of the product. The sensing cover consists in several arrays of strain sensors directly printed into a Lycra tissue. The resulting design results in a high-impedance circuitry where a reference current is injected. The high-impedance characteristic allows both sensors and wires to be realised by means of the same technology and to gain unobtrusivity. In fact, the use of common electrical wires is avoided within the sensing cover. Moreover the power consumption is quite zero resulting in a completely safe system. A connector plug is placed in one side of the sensing cover in order to connect the system to the front-end module. The fabric with new generation of distributed and redundant smart unobtrusive sensors allows us to guarantee: Plasticity Coat whole the seat Low dimension June, 2006 8 UNIPI

Lightness Directly printed on the fabric Adherence to the seat Low cost After the study of the signal response on a single strain sensor, several types of topology were taken into account (series, parallel and quadripolar networks) and finally the best compromise between the technical complexity and the signal classification results was found using the series network. A front-end device for the signal conditioning and a data acquisition board were subsequently designed and developed in order to guarantee the flow on data between the sensing system and the PC. Preliminary tests were conducted making use of a sensing patch: a series array of strain sensors directly printed on a small (20cm 20cm) Lycra tissue. The system was placed in a mechanical platform equipped with servo-motors in order to develop an appropriate classification algorithm. The goal was to correctly classify the deformations of the Lycra starting from the signal acquired from the sensors. Since the number of the sensors printed on the small Lycra tissue was very low, the standard classification algorithms did not work properly (PCA, KSOM, MLP). On the other side, only an advanced real-time neural network [5, 7] showed good results. As it will be shown later, the realisation of the final sensing seat prototype equipped with a large number of distributed strain sensors allowed us to avoid the use of the previous mentioned complex classification architecture. A software architecture was developed following a modular and layered strategy. The low level interfaces were developed in order to act as buffers between the hardware acquisition system and the high level processing units. The data communication is managed in an asynchronous way in order to let the processing units able to run in real-time and with a time-resolution that is independent from the sampling rate of the data acquisition board. The low level interfaces of the final prototype are designed in order to be easily replaced by the BioSec API layer. The sensing seat prototype was realised making use of a sensing cover placed between the seat (actually a truck seat, provided by VOLVO) and an external cover that protects the sensors from the external world. The sensing cover is equipped with several strain sensors (Conductive Elastomers, CEs) grouped in several patches. The series sensor network is connected to one side to a connector plug that is interfaced to the front-end hardware device. The nature of the CEs allowed the realisation of a completely safe and versatile system that is equipped with no electronic component or classical wires. Indeed, as it is mentioned above, both the strain sensors and the connecting wires are realised using the same CE technology. As a result, the UNIPI sensing seat system does not interfere with the mechanical structure of the seat and it is designed as an extension of the seat. As the sensing seat system, the front-end hardware device, the data acquisition board and the software framework were completed, the next step consisted in the realisation of the authentication algorithms. Thanks to the redundant sensor network, we decided to use many cooperating classifiers and to replicate recursively in this task the main idea of the whole HUMABIO project. Actually three classifiers (a distance-based classifier, a KSOM and a MLP) perform the classification task sharing the input data and supplying June, 2006 9 UNIPI

three different classification results. A final classifier performs the fusion of the results and supplies the final response. The recording protocol was defined as an interactive procedure where the subjects collaborates with the system in order to perform the main tasks (enrollment, authentication), while the monitoring task is performed in real time by the system (the authentication has to be repeated if the system detects the absence of the subject for a period of 1 second). During the enrollment and the authentication steps, the user is asked in turn to seat in some predefined positions. In the case of the enrolment procedure, the new subject is asked to repeat the measurement for 10 times for each predefined position. Then the acquired data are stored into the database and the classifiers are trained again in order to take into account the new user. In the case of the authentication procedure, the user is asked to perform a measurement for each predefined position and the acquired data are compared with the data previously stored into the database. On the basis of the response of the classifiers, the user will be authorised or rejected by the system. The human profile signature was subsequently specified taking into account the information about the user (an identifier) and the steady state value for each strain sensor belonging to each repeated measurement of each predefined position. A structure data type was defined for both the client application and for the database. Methods for the conversion between the client and the database structures are supplied. Finally the tests were conducted on 40 human subjects (10 repeated measures with 2 predefined positions) using 20 couples (10 men and 10 women ranging from 20 to 40 years old) of physically similar subjects. The results are discussed below in this document and they shows the robustness of the system. The correct recognition percentage assesses on 93% +/- 2% (an enrolled subject claims to be himself, FRR = 7% +/- 2%), while the correct rejection percentage assesses on 96% +/- 2% (an unenrolled subject claims to be an enrolled subject, FAR = 4% +/- 2%). Also the continuous monitoring was tested. During this stage the processing system runs continuously in parallel with the acquisition system. Since the nobody user is an enrolled user as well as the other real users, the system is able to detect his presence (i.e. nobody is seated on the sensing seat) in about 1 second. Starting from this information the system is able to detect if the user must repeat the authentication procedure. The system is currently able to recognise the absence of the human subject in less than 1 second with a 100% success (more than 100 measurements were carried out). More measurement will be performed in the future in order to assess the stability of the classification capabilities over time. If needed, the classification algorithms will be tuned and improved. The classification algorithms are now stable and the BioSEC API integration will be completed before than the end of October 2007. June, 2006 10 UNIPI

1 Introduction The HUMABIO project aims to develop a multi-modal authentication system for human subjects in several scenarios where the security is a critical aspect. Many modules will cooperate in order to perform the authentication task over the same human subject. Such modules (EEG analysis, ECG analysis, speech recognition, face recognition, gait recognition and human profile analysis) will be available within the HUMABIO framework and, according to application scenario (truck pilots, airplane pilots, plant personnel, people in airports and, in general, people in environments where the security is mandatory), a subset of the authentication modules will be used. Within the consortium, the UNIPI goal is to develop a sensing seat for human authentication module, to be used in all the critical scenarios where a seat is available to the human subject (truck seat, office seat and airplane seat among the others). As the analysis of the state of the art in sensing seat systems has been conducted, the authors decided to adopt a novel technology in order to face the authentication problem. This is mainly due to two aspects: the unavailability of an existing sensing seat system for human authentication and the inadequacy of the existing sensor technology in order to address the human subject recognition as well as the integration of the sensors in the seat. 1.1 Chapters Outline This section describes the structure of the present document. In chapter 2 the state of the art analysis is reported. As it is shown, all the existing systems based on sensing seats are focused on the weight measurement task and on the study of the (dis)comfort of the human subject. That is, the human authentication task was never taken into account, resulting in the absence of a reference database for benchmarking comparison. Moreover all these systems use classical electronic sensors that are not easily integrable in different kind of seats. For this reason, UNIPI decided to adopt an its own sensing seat strategy and technology, both for materials and algorithms. The absence of reference databases makes it impossible to obtain a comparison between the authentication result of UNIPI sensing seat system and the other ones. In chapter 3 the UNIPI strain sensor technology is outlined. The sensor technology is described as well as the innovative aspects. The fabric with novel distributed and unobtrusive sensors allows us to address plasticity, the covering of the whole seat (shape independent), low dimension, lightness, adherence to the seat and low cost. The single sensor analysis is reported. In chapter 3.4 the development and the characterization of a sensor patch consisting in an array of strain sensors are described. The sensor patch is the building block of the sensing cover, which is realised by means of several sensing patches. The signal analysis and the data processing task are outlined. At this point, the front-end device for signal June, 2006 11 UNIPI

conditioning and the data acquisition board were designed and realised as part of the final prototype. In chapter 4 the software architecture for the system management is discussed. The software architecture is realised in order to be easily adaptable to the HUMABIO framework. The architecture is able to manage the data acquired from the sensing system in an asynchronous way in respect to the running authentication system. This strategy let us to develop a real-time analysis module with a time-resolution that is independent from the sampling rate of the data acquisition board. In chapter 5 the final prototype of the sensing seat is discussed. The sensing cover, the recording protocol, the human profile signature data structure and the database architecture are described. The GUI interface provides the main functionalities (enrollment, authentication and monitoring tasks). In chapter 6 the experimental setup and the measurement campaign are described and the test results are provided. A measurement campaign was conducted taking into account 40 human subjects, including 20 men and 20 women ranging from 20 to 40 years old. 10 repeated measurements were carried out for each subject and for each predefined position. Global and personal classifiers were trained and tested on the basis of the acquired dataset. The data-set was partitioned in order to obtain a training-set and a test-set. Many combinations were taken into account so that some subjects were randomly removed from the training set in order to simulate impostor subjects. While good results were obtained both for global and personal classifiers regarding the True Acceptance Rate (TAR) and the False Rejection Rate (FRR), only the personal classifers showed a high True Rejection Rate (TRR) and a low False Acceptance Rate (FAR). In chapter 7 the open issues are discussed and the planning for other test campaign is reported. 1.2 A Sensing Seat for Human Authentication Sensors embedded in seats can measure both the local pressure exerted by a user and reconstruct the contact profile on the seat and at its back. These two features can help in discriminating among different persons which may seat on a workstation. This task will focus on the following topics: Determination of the information necessary to discriminate different anthropometric profiles by using a sensing seat. Design and implementation of a sensing network embedded in the fabric which covers different seats (office and car seats) capable to address the previous point. Implementation of the sensor network analysis software package devoted to the pressure acquisition and the profile reconstruction. Behavioural and other biometrics profiles for authentication, validation and monitoring applications will be created. This will also involve a preliminary research on inter-modal data fusion, exploration of the different statistical models and artificial intelligence methods for fusion and analyse their effect on the compound performance vs. single-modal June, 2006 12 UNIPI

biometric performance. Furthermore, integration of available commercial modules multimodal operation will be studied. Development of algorithms and tests will be done based on existing and newly collected data. Due to the nature of the used biometric features, these profiles will be applicable for every person. June, 2006 13 UNIPI

2 State of the art analysis The state of the art on sensing seat systems has been investigated during the first months of the HUMABIO project. These systems, some of them already on the market, use different technologies and materials, but they share the use of sensors that measure the pressure exerted on the seat by the passenger. Among the most important, we have mentioned: Tekscan, Novel, Euralab by Wronz, ComfortLab by Johnson Controls. 2.1 Tekscan This Pressure Measurement System is extensively used because of its simplicity and low cost. Tekscan supplies a matrix of sensitive elements. The standard sensor (Fig. 2.1) consists of two thin, flexible polyester sheets which have electrically conductive electrodes deposited in various patterns. The inner surface of one sheet forms a row pattern while the inner surface of the other forms a column pattern. Therefore the sensible locations are identified by the intersection of rows and columns. A patented, thin semi-conductive coating (ink) is applied as an intermediate layer between the electrical contacts (rows and columns). This ink provides the electrical resistance change at each of the intersecting points. By measuring the changes in current flow at each intersection point, the applied force distribution pattern can be measured and displayed on the computer screen. The various locations are selected electronically and it is possible to read the resistance variations and the force application time. The sensors are extremely thin (they can be as thin as 0.1 mm) and flexible, so that they can adapt theirself to a curve surface and they are capable of measuring pressures ranging from 0-15 kpa to 0-175 MPa. This system can sample thousands of sensors per second: this high sampling rate is possible due to the extreme thinness and the relatively high spring constant of the sensor materials. The manufacture supplies a connector interfaced to the matrix of sensors; it can be shaped according to the type of the measurement, an acquisition data card and a software Figure 2.1: Standard Tekscan sensor June, 2006 14 UNIPI

Figure 2.2: Resistance vs. force for Softswitch pressure sensors for the data management. This software presents a simple interface and it can visualize data in real time and in 2D or in 3D. The storing as data files allows the analysis and the visualization of the data through work sheets such as Microsoft Excel or Lotus. The Virtual System Architecture (VSA) is also provided: it can visualize wide areas thanks to multiple sensors that are positioned one near the other. So the data, managed by the software, are presented as if they would be originated from one sensor [1]. 2.2 Novel The Novel Pliance is a system made up by a flexible and elastic matrix of sensors, a multichannel analyzer, a circuit for the calibration and an enclosed software. The matrix is available in different shapes and sizes (the system PLIANCE 2000 can change the technical features of the matrix for different applications). The sensors are located inside the matrix: they are capacitive sensors and they depend on each other mechanically. The matrix of measurement adapts itself very well to the irregular surface. The analyzer allow a dynamic check of the amplification and the elimination of the cross-talk errors, so that the measurement is more accurate. The link with the computer does not require a special card, since it is performed through the standard serial communication EIA RS232. The data are acquired and managed by the software, that works under Windows environment [2]. 2.3 Euralab by Wronz A kind of technology, developed in the textile domain, is Softswitch: it is based on pressure sensors that can be integrated in fabrics. Softswitch combines conductive textile materials and a quantum tunnelling composite (QTC) with unique pressure controllable switching properties. In their normal state Softswitch fabrics are insulators until pressure is applied at which point the resistance decreases until the fabric achieves metal-like conductivity (Fig. 2.2) [3]. Different car components supplier companies have designed and put on the market car seats that utilize these pressure sensors. Among these we notice: June, 2006 15 UNIPI

Figure 2.3: ComfortConforms system 2.4 ComfortLab by Johnson Controls The ComfortConforms system (Fig. 2.3) [4], created by Johnson Controls, reads and responds to each individuals unique weight distribution to provide support where its most needed. This is made with a set of subcomponents: The occupants weight distribution informs the seat system of support needs Support is tailored using air cells positioned on top of the foam A closed loop comfort algorithm is used to refresh support needs periodically The comfort algorithm is based on data gathered from broad consumer testing The system is activated when the ignition is turned on, but a switch will also allow the user to control on/off The user may also increase/decrease the level of lumbar support. The research of seating comfort in the transportation industry is still an open problem; for solving this question, the Department of Industrial Manufacturing Systems Engineering of the University of Windsor (Canada) has effected a study that intends to determine if the advantages and the disadvantages of a new automotive seating concept could be reliably evaluated using both a physiological assessment technique (i.e. electromyography) and a subjective questionnaire. The results indicate that psychophysical measure of discomfort and the activity of the EMG are statistically related. More specifically, subjective perceptions of comfort were found to improve with decreasing levels of muscle activity. This implies that seat comfort can be evaluated on the basis of physiological as well as subjective responses to prolonged driving. June, 2006 16 UNIPI

3 UNIPI Strain Sensors The sensors we develop allow piezoresistive sensing fabrics to be realized. Mixtures of polymers deepened with coal directly printed onto the fabric allow strain to be transducted. An electronic interface manages, besides the pre-filtering, the digitizing of the signal and the communication with the PC. The sensors are positioned in such a way as to monitor all the contact surfaces between the seat and the occupant. Because of the nature of the adopted sensors and the possibility to cover the surfaces with a redundant net of them, we could choose the maximum number of the sensors for covering uniformly the surfaces of contact. The final prototype of the sensing cover placed over the seat is equipped with 36 strain sensors [8] realized by means of Conductive Elastomers (CE) composites. CE composites show piezoresistive properties when a deformation is applied and can be easily integrated into fabric or other flexible substrate to be employed as strain sensors (Fig. 3.1). The CE we used is based on a WACKER Ltd (Elastosil LR 3162 A/B) product. It consists in a mixture of graphite and silicon rubber. WACKER Ltd guarantees the non-toxicity of the product. The sensing cover consists in several arrays of strain sensors directly printed into a Lycra tissue. The resulting design results in a high-impedance circuitry where a reference current is injected. The high-impedance characteristic allows both sensors and wires to be realised by means of the same technology and to gain unobtrusivity. Indeed, the use of common electrical wires is avoided within the sensing cover. Moreover the power consumption is quite zero resulting in a completely safe system. A connector plug is placed in one side of the sensing cover in order to connect the system to the front-end module. 3.1 Description and Realisation of the Sensors and the Connections The sensing seat system is composed by a seat coated by a removable Lycra sensing cover. The sensing cover is able to respond to simultaneous deformations in different directions by means of a piezoresistive network which consists of CEs composite rubbers screen printed onto a cotton Lycra fabric. They are elastic and do not modify the mechanical behaviour of the fabric. In the production process of sensing fabrics, a solution of CE and trichloroethylene is smeared on a Lycra substrate previously covered by an adhesive mask. The mask is designed according to the desired topology of the sensor network and cut by a laser milling machine. After the deposition, the cross-linking process of the mixture is obtained at a temperature of 130 C. Furthermore, by using this technology, both sensors and interconnection wires can be smeared by using the same material in a single printing and manufacturing process. June, 2006 17 UNIPI

Figure 3.1: Transduction principle of the strain sensor 3.2 Test on a Single Sensor From the technical viewpoint, a piezoresistive sensing fabric is a system whose local resistivity is a function of the local strain. In a discrete way, it can be thought of as a two dimensional resistive network where single resistors have a non-linear characteristic that depends on the local strain. The integral impedance pattern is a function of the overall shape of the sensing fabric and allows mapping between the electrical space and the shape space. For the characterisation of the sensors in terms of their quasi-static and dynamic electromechanical transduction properties, sensors were serially connected. In this case, a current is superimposed in the circuit and high impedance differential voltages are acquired from each sensor. Two multiplexers allow a sensor to be selected and the relative signal is acquired by a differential amplifier. A microprocessor drives the whole system, performs the analogous/digital conversion and exchanges data via USB interface. The device is provided with an automatic calibration subsystem which allows gain and offset to be tailored to each sensor. Strain impulses result in a typical differential voltage behaviour showed in Fig. 3.2. Sensor response shows a peak in correspondence to every mechanical transition. A pushing punch driven by a stepper motor is used to apply alternate mechanical deformations (by indentation) to each fabric based sensor. Several tests were carried out, by using rectangular-wave mechanical stimulations (series of pressure impulses). In order to model the electromechanical response of each sensor, an equivalent circuit based on the equivalence between the electrical response (current variation) of the circuit and the response (resistance variation) of the sensor was proposed. Each sensor was tested by applying a series of pressure impulses (Fig. 3.2a) and by acquiring the voltage drop across the sensor as its response (Fig. 3.2b). The sensor response shows a peak in correspondence to every mechanical transition. Data acquired were filtered, peaks were detected and relative maximum and minimum, and time constants were selected as features. Sensor responses during constant pressure time intervals were approximated by decreasing exponentials, assuming the local minimum as the steady-state value. This approximation results as true as long is the pressure time interval. In order to remove the contribution of high order exponentials, first order time constants were calculated discarding the first 5% of each curve. This choice allowed quantization errors introduced by the acquisition device in response to rapid transitions to be avoided and sensor steady state deformation, related to slower frequency components, to be maintained. During a series of pressure impulses, the time constants of the deformation phases pre- June, 2006 18 UNIPI

Figure 3.2: a) A pressure impulse; b) Sensor response a pressure impulse and selected features Figure 3.3: a) Acquired signal during pressure deformations (thin line) and extracted exponential law (bold dotted line); b) Quadratic error sented an average value of 9.32 seconds, while the time constants during the deformation recovery showed an average value of 4.72 seconds. Fig. 3.3a shows the acquired signal during pressure deformations (continuous line) and the extracted exponential discharging law (dashed line); Fig. 3.3b shows the quadratic error. Close to the mechanical transitions the differences between the signal and the exponential law is high; during the constant pressure phases the differences are very low (< 3 10 5 V ). In order to model the first-order components of the sensor response (resistance variation) to a rectangular stimulation (applied deformation), the equivalent circuit represented in Fig. 3.4 was considered. The power supply V is the electrical analogous of the imposed deformation. The switch T1 (initially open) is closed and open in correspondence of, respectively, the beginning and the end of the imposed deformation. The switch T2 (initially open) is closed when T1 is open again. Following a simple analysis of this circuit, it is easy to recognise that the variation of the charging and discharging currents of the circuit in consecutive phases of stimulation are analogous to the variation of the resistance of the sensor during, respectively, its deformation and the following release. The circuit parameters R1, R2, R3 and C can be derived by using the features, extracted from reference experimental signals, June, 2006 19 UNIPI

Figure 3.4: Proposed equivalent electric model of each sensor Feature of the variation of the Feature of the variation of the Symbol sensor resistance charging/discharging currents of the circuit Initial peak [kω] Initial peak [A] I 1 (0) Steady-state value for the Steady-state value for the charging I 1 ( ) deformation phase [kω] phase [A] Time constant of the first-order Time constant for the charging τ 1 exponential components for the phase [s] deformation phase [s] Time constant of the first-order Time constant for the discharging τ 2 exponential components for the phase [s] Table 3.1: Considered analogous features listed in Table 3.1. A circuit voltage of 1V was assumed as the analogous of a deformation of 1mm, while a circuit current of 1A was assumed to correspond to a variation of the sensor resistance of 1kΩ. Values of the features listed above were extracted from ten cycles of a reference experimental signal and were used to derive the circuit parameters by means of the following system of equations: τ 1 = C(R 1 R 2 ) I 1 (0) = R V 1 I 1 ( ) = R V 1 +R 2 τ 2 = C(R 2 R 3 ) The solution of these equations provided, for the considered ten cycles of stimulation, the results reported in Fig. 3.5. 3.3 Sensor Net The integration of pressure/deformation sensors directly into textile fabric allows the covering of non plane and deformable surfaces with a high number of sensors. This technology allows a large set of sensors distributed over a fabric to be realized. The advantage offered by this technology is quite clear: a lot of information can be obtained, leading to a great versatility in device implementation. June, 2006 20 UNIPI

Figure 3.5: Values of the parameters of the equivalent electric model extracted from ten cycles of a reference experimental signal Figure 3.6: Series network The data acquisition from a large number of sensors, one per channel (single sensor reading), increases the complexity of the electronic acquisition system. It is possible, however, by using several topologies of interconnection of sensors to reduce this complexity. By connecting sensors arrays in electrical networks and measuring the voltage values at their borders only, it is possible, in principle, to reconstruct a resistance variation of the value of an inside-located sensor. Nevertheless, these strategies of interconnection lead to a loss in terms of system sensitivity and accuracy in signal reconstruction, hence it is necessary to come to a fair compromise. Here we present three strategies that may be employed to perform the data acquisition from a large number of sensors: Series network Parallel network Quadripolar network 3.3.1 Series Network This strategy consists in connecting sensors in series as showed in Fig. 3.6. Pairs of sensors (whose distance is negligible with respect to the geometric dimension of the net) on the fabrics are connected rows by columns. The inevitable crossing of wires makes necessary to electrically insulate them. The small distance between two sensors ensures that their deformation is quite the same and differs from the other couples. A deformation of a pair of sensors implies a resistance variation of the entire row and column identifying the couple. So, by reading a variation of a column and of a row, we June, 2006 21 UNIPI

Figure 3.7: Parallel network Figure 3.8: Quadripolar network can identify a precise point into the net. For this strategy, considering a square array of sensors, a n-channels acquisition system can read a network of (n/2)2 pairs of sensors. So with a 32 channels electronic unit it is possible to read 256 sensing elements (pairs of adjacent sensors). 3.3.2 Parallel Network This strategy consists in connecting sensors in parallel as showed in Fig. 3.7. All the considerations made for series network are also valid for this type of interconnection. 3.3.3 Quadripolar Network In the quadripolar strategy (Fig. 3.8) connections are realized inside each sensor and no insulated cable is needed. In this configuration, signals are read by multiplexing both the channels and the ground of each column and row. By using common techniques of resistance network analysis, it can be proved that a sensor is localized by matching the row and the column which present the maximum resistance variation. For this strategy, considering a square array of sensors, a n-channels acquisition system can read a network of (n/2)2 of sensors. So with a 32 channels electronic unit it is possible to read 256 sensing elements. The series and parallel networks are limited by the strong constraint of reading only a local deformation, i.e. if only a single sensing element is deformed then it is possible to successful identify the spatial point where the deformation occurs. This limitation is overcome by the quadripolar network strategy, but the reconstruction of the status of the June, 2006 22 UNIPI

a) b) Figure 3.9: A single sensor array patch net could be too expensive from a computational point of view. In view of the above considerations, for this project it could be convenient to use the single sensor reading (one per channel), because 32 sensors are sufficient to gather desired information without increasing the complexity of the electronic unit 3.4 Sensor Patch Design and Realization The second step consisted in the design and realization of a preliminary two dimensional resistive sensor network in order to redundantly cover the deformable surface of the truck seat. UNIPI received from Volvo a real truck seat in order to tailor the sensing cover on it. Since a single sensor reading strategy may increase the complexity of the electronic acquisition system endangering the real-time performance, it is possible to adopt several topologies of interconnection of sensors to reduce this complexity. By connecting sensors arrays in electrical networks and reading their borders only, it is possible, in principle, to reconstruct a resistance variation of the value of an inside-located sensor. Nevertheless, these strategies of interconnection lead to a loss in terms of system sensitivity and accuracy in signal reconstruction, hence it is necessary to come to a fair compromise. Finally, 5 sensor arrays (36 strain sensors) were distributed to evenly cover the truck seat. In each sensor array, sensors are serially connected. A current is superimposed in the circuit and high impedance differential voltages are acquired from each sensor. A drawing and a picture of a single sensor array are shown respectively in Fig. 3.9a and 3.9b. 3.5 Data Acquisition Board Front-End The bold black track in Fig. 3.10a represents the set of sensors connected in series (S i ) and it covers the knee joints. The thin tracks (R i ) represent the connection between sensors and the electronic acquisition system. As the thin tracks are realized by the same conductive elastomer material, they undergo a significant change in electrical resistance when they are stretched. The electronic front-end unit is designed to compensate this variation. The piezo-resistive characteristic of the sensors let us able to perform a measurement of resistance variation associated with their strain deformation. The sensor series is supplied by a constant current I. The acquisition system consists of a high input impedance stage realized by instrumentation amplifiers, that are represented in Fig. 3.10b by the set of June, 2006 23 UNIPI

Figure 3.10: a) The mask used for the sensing cover; b) The equivalent electric scheme of sensing cover Figure 3.11: Constant current generator voltmeters. In this way, the voltages measured by instrumentation amplifiers (V mi ) are equal to the voltages which fall on the S i (V ni ), that is related to resistances of the sensors. This strategy allows to avoid the use of metallic wires for the sensor connections. The Analog Devices AD623 instrumentation amplifier is used to acquire the voltage V mi. This amplifier has an input impedance in the range of a GO. The AD623 bias current is very low; it flows through Ri and it makes the voltage difference between ni and mi negligible. The constant current generator is realized using the circuit showed in Fig. 3.11. In this way, the current that flows in the sensing cover is independent from the load. The current in the sensing cover must be high enough to increase the signal to noise ratio but it is bounded by the operational amplifier saturation limit. The electronic front-end acquires the signals coming from the prototype, then these signals are filtered by a lowpass filter with a cut-off frequency of about 20 Hz. Then the signals are digitized using a June, 2006 24 UNIPI

National Instruments AD conversion board (Series cdaq 9172) and they are sent to PC. June, 2006 25 UNIPI

4 Software Framework A software framework for the management and synchronization of data and processes was developed. The framework core and the application processes are interfaced to the sensor array through a framework I/O interface. The framework I/O interface was designed in order to act as a buffer for the flow of information from the sensors to the application process [9]. Signals that will be coming from different sensor arrays are gathered in parallel and are encoded following a standard protocol. The encoded information is received by a dedicated filter for each sensor, which then sorts them to framework I/O interface. Communication channels are established as connections between application processes. The domain of data flowing through connections and the flow chart of the application processes can be properly designed according to a specific application. Processes and connections are managed at run time and they can be manipulated under request. The presence of dynamic structures implies a configurable resource management, so the framework offers an optimised interface for enumeration and direct access requests. A spatial definition of the entities involved in the framework can moreover be supplied, making this information available to the control system for subsequent processing. To guarantee the execution of real-time applications an inner synchronization signal is provided from the framework core to the processes and to the framework I/O interface, enabling to gain time-space correlation. Such dynamic geometric representation can be visualised by a high efficiency 3D graphic interface, giving a support during experimental setup debug. 4.1 Software Analysis Using a Single Sensor Array Patch The integral impedance pattern of the two dimensional resistive sensor network is a function of the overall shape of the sensing fabric and allows mapping between the electrical space and the shape space. In this period we tried to code in real-time the shape space by a two-dimensional artificial neural network (ANN). The neuron model proposed by E. Izhikevich [5] was adopted because of the high computational efficiency and biological accuracy. The Theory of Neuronal Group Selection (TNGS) proposed by G. Edelman [6] was adopted as the learning strategy of the ANN. The TNGS suggests a novel way for understanding and simulating neural networks. To take into account this theory we have to use the time variable in the learning task, so that neural groups may raise from a selection process. In the artificial neural network model, the synaptic connections are modified according to the computationally efficient Spike-Timing-Dependant Plasticity (STDP) rule [7]. June, 2006 26 UNIPI

Figure 4.1: The mechanical platform 4.2 Results Obtained with a Single Sensor Array A single sensor array (i.e. 12 strain sensors) has been positioned on a soft fabric of a mechanical platform. Six servo motors have been radially positioned at the boundaries of a circular portion of the fabric containing the sensor array in order to singularly apply strain with different amplitudes along different directions (Fig. 4.1). A single neural network of 1000 artificial neurons consisting in 200 inhibitory neurons and 800 excitatory neurons has been realised as the main application process inside the framework core. A running process was delegated to actuate the servo motors in order to generate random actuation patterns during the training and test phases. The framework I/O interface dispatches data packets from the sensor array to a subset of the excitatory neurons. The network design has been inspired to the anatomical structure found in the mammalian cortex. In respect to the total number (N) of neurons, a percentage equal to 80% consists of excitatory neurons, while the remaining 20% are inhibitory neurons. Cortical pyramidal neurons showing a Regular Spiking (RS) behaviour have been adopted for the excitatory subsection, which correspond to appropriate values for the Izhikevich neuron model. Inhibitory neurons have been simulated adopting the model of the cortical interneurons which exhibits Fast Spiking (FS) property. Each neuron is connected to M different neurons in order to obtain a connection probability (M/N) equal to 0.1, but inhibitory neurons are connected only to excitatory neurons. Moreover, the synaptic weights of the connections arising from the inhibitory neurons remain unchanged during the learning process, while those regarding the connections from the excitatory neurons change according to the STDP rule. Axonal delays are fixed in the range between 1 millisecond and 20 milliseconds. The time resolution has been set to 1 millisecond. The training phase has been carried out for more than 8 hours. As the application starts, all the connections have the same synaptic weight. The network needs many seconds to get stabilised through depression and strengthening of the synaptic weights. During this first phase, the network shows the presence of a high amplitude rhythm, with frequency in the range between 2 Hz and 4 Hz (delta waves), as it is shown in Fig. 4.2. After a few hours of network activity the spiking rhythm becomes uncorrelated and June, 2006 27 UNIPI

Figure 4.2: Learning phase: the presence of a high amplitude rhythm Figure 4.3: Learning phase: the spiking rhythm becomes uncorrelated frequency in the range between 30 Hz and 70 Hz appear (gamma waves) as it is shown in Fig. 4.3. The arise of such rhythms is called PING (Pyramidal-Interneuron Network Gamma) and it seems to be related to the spikes of the pyramidal cells which excite the inhibitory interneurons. Such interaction allows a mutual inhibition which temporarily switch-off the network activity. As the network becomes stable, the oscillation rhythm is assessed in the frequency range between 2 Hz and 7 Hz and the training phase is ended. We noticed the presence of a large number of neural groups, each of them able to perform a reproducible spike sequence with a precision of one millisecond. The test phase consists in recording neural groups activity in response to random strain patterns. A labeling procedure allows to associate a specific strain pattern to a neural group. Each stimulus that is used as input pattern is able to select one group inside the network, showing that the network is able to perform classification tasks. Such classification is realised by a memory capability which is far greater than the number of entities involved into the network. Such a structure is able to perform an its own representation of a specified context. The test phase was conducted generating in a random order 100 low-amplitude deformations and 100 high-amplitude deformations for each servo motor. Recognition percentages result is summarized in the confusion matrix reported in Table 4.1. The confusion matrix possesses a number of both rows and columns equal to the number of strain patterns to be recognized. The generic element r(i,j) represents the degree of recognition of the strain pattern i as strain pattern j. A more diagonal confusion matrix corresponds to a higher degree of classification. Since each strain pattern may be confused with more than one strain patterns, the sum on each row and column may differs from the value of June, 2006 28 UNIPI

I low I high II low II high III low III high IV low IV high V low V high V I low V I high I low 78,8 5 6,8 3,4 3,3 10,1 5,9 5,8 7,9 6 8,2 6,8 I high 3,6 71,7 9,4 8,5 7 6,2 10,8 3,3 4,8 10,7 1,9 3,6 II low 5,3 9,9 79,3 0,7 8,7 2,1 4,5 4,3 6,2 10,5 7,3 10,9 II high 4,3 4,5 5,4 70,2 6,1 5,5 3 7,1 6,2 11,1 9 7,2 III low 6,1 3,1 9,3 0,8 74,8 2,1 8,3 1,8 7,9 0,7 7,3 5,4 III high 11,8 8,5 7,5 2,1 9 72,4 6 10,2 7,9 4,8 8,7 5,3 IV low 3,7 2,1 4 7,2 9,8 9,5 76,4 4,6 9,7 4,6 8,7 3,4 IV high 7,9 2,9 11 7 6,4 2,5 4,5 76,7 9,4 4,6 6,2 10,6 V low 4 3,8 2,9 7,5 1,6 2,8 4,3 11,8 73,6 8,4 9,6 10,7 V high 4,4 6,7 10,2 7,3 4 3 7,3 2,5 5,1 69,9 7,4 8,3 V I low 7 7,4 5,7 12 8,1 6 6,8 8,6 1,8 4,1 80,2 10,3 V I high 5,1 8,8 6 6,5 9,6 6,6 3 10 5,5 5,4 4,2 76,2 Table 4.1: Confusion matrix reporting ANN classification results 100%. It is important to underline the capability of the system in discriminating between different strain patterns as well as different amplitudes of the same strain pattern. 4.3 Single Sensor Array Patch Conclusions The real-time signals coming from a sensor array consisting of a single patch of the sensing cover have been analysed by a dedicated ANN. As a result different patterns are real-time coded by different neuronal groups, showing that the network is able to code the shape space of the sensing patch. This could allow to deconstruct a complex shape space into a combination of strain pattern features. According to the biological sensory systems, where environmental stimuli are deconstructed and then reconstructed in the brain to create perceptions, the presented architecture represents an initial step in the reconstruction of a shape space from its deconstructed features for identification purposes. June, 2006 29 UNIPI

5 Sensing Seat Architecture The classification algorithms using a single sensor array (12 deformable sensors) and the design were improved as the final prototype of the sensing cover of the truck seat (using many sensor arrays) was completed. The classification modules are based on an artificial neural network architecture able to run in real-time with a time resolution of 1 millisecond, according to the neuronal model proposed by E. Izhikevich and the real-time learning strategy proposed by G. Edelman. The software architecture has been improved making the system able to run in parallel with the acquisition system that, from general point of view, is not able to gain the same time resolution of the classification algorithm. Indeed the acquisition system of a single sensor array is able to gain a time resolution of 200 milliseconds using the USB acquisition hardware, while a time resolution of 10 milliseconds was obtained using a PCMCIA National Instruments data acquisition board (NI-cDAQCard-9172). The classification module runs asynchronously in respect to the acquisition module and the management of the acquisition driver is based on notification messages. That is, the classification module, running at 1000 frames per second, receives a notification message from the acquisition system only when new data are available from the sensing network and a sample-and-hold strategy has been adopted. Since the realisation process of the a single strain sensor impedes to obtain thin sensors, it was impossible to realise a scaled sensing architecture to simulate the classification over scaled human profiles. Indeed the thin sensors with 0.5 millimeter width are fragile and their life period is actually too short. Moreover the sensing support is made by a Lycra tissue and it is difficult to realise tissues with a scaled elastic constant. For these reasons the tests on profiles of real human subjects should be performed only when the whole sensing cover will be available. 5.1 Sensing Seat Cover Design Fig. 5.1 shows the truck seat provided by VOLVO and the seat cover developed by UNIPI and equipped with 5 sensor arrays. The sensor arrays are embedded into a Lycra tissue, with the same dimension of the seat cover. The resulting sensing cover can be easily placed over the truck seat (Fig. 5.2). 5.2 Recording Protocol Denition Both the enrollment and authentication tasks will be performed in controlled conditions and with the collaboration of the human subject. The user will interact with the sensing seat system interface following the protocol described below: June, 2006 30 UNIPI

a) b) Figure 5.1: The sensing seat: a) truck seat provided by VOLVO; b) seat cover (the sensors will be placed on the back side of the cover) Figure 5.2: Final prototype of the sensing seat: the sensor arrays are embedded into a Lycra tissue, placed in the back side of the seat cover. The sensing cover is placed over the truck seat. June, 2006 31 UNIPI

1. baseline measurement (seat without human subject) 2. calibration 3. recording: the subject is seated in the first predefined position 4. restart from point 1) until this recording is equal to the previous recording 5. recording: the subject is seated in the next predefined position 6. repeat point 5) for each predefined position 7. recorded data are compared with DB data The steps 1), 2) and 3) are repeated a few times in order to get the sensory system ready to work properly (e.g. the strain sensors have not been deformed for a long time). The protocol needs the definition of a set of predefined positions that may be assumed by the human subject while he is seated. Currently we defined two predefined positions 1. the subject is seated normally (i.e. both the bottom side and the back side of the subject are in contact with the sensing cover) 2. the subject is seated only to the bottom side of the cover (i.e. the back side of the subject is not in contact with the sensing cover) 5.2.1 Predened positions The final prototype of the sensing seat system is designed to take into account a set of predefined positions that the subject may assume when seated. As it will be showed below, this strategy gives the opportunity to improve the classification and misclassification rates. Both the software architecture and the database structure are able to manage a dynamic change of the number of the predefined positions. Two positions are currently taken into account (Fig. 5.3): 1. the subject is fully seated onto the seat and all the sensors of the sensing cover are consequently stretched 2. the subject is not in contact with the back side of the seat and only the sensors belonging to the bottom side of the sensing cover are stretched 5.2.2 Initialization During the initialization step the system will ask the user to leave the seat in order to perform the calibration procedure. In an ideal situation this procedure will take 5 seconds. However the system is able to detect external forces that cause a deformation of the sensing seat surface and the procedure will not be completed until a steady state value is gained. June, 2006 32 UNIPI

Figure 5.3: Two predefined positions currently taken into account; a) the subject is fully seated; b) The subject is not in contact with the back side of the sensing cover 5.2.3 Enrolment The enrolment step requires the subject to perform multiple measurements for each predefined position. The subject will be asked to seat in turn in each predefined position. Actually 6 repeated measurements are performed. Since each measurement will take 5 seconds at least, the total amount of time needed by the enrolment procedure is 5 2 6 = 60 seconds at least. During each measurement, as for the calibration step, the system will not stop the acquisition until a steady state value is reached. This, together with the delay related to the interaction between the user and the control system, should increase the enrolment time. 5.2.4 Authentication The authentication step requires the subject to perform a single measurement for each predefined position. The total amount of time needed by this step is 5 2 = 10 seconds at least. For the same reasons discussed above for the enrolment step, the authentication time may grow depending on the interaction of the subject with the sensing seat hardware and with the control interface. 5.3 Human Prole Signature For a single human subject, the profile signature consists in the steady state values of the strain sensors of the sensing cover, for each of the predefined position. From a general point of view, let us assume the sensing cover is equipped with #N strain sensors (N=14 in the preliminary prototype; N=36 in the final prototype) while #P predefined positions are defined (actually, P=2 in the final prototype). A steady state value of a single strain sensor represents a voltage information. The system stores this information as a floating-point value (i.e. a double data type, that is 8 bytes). For each measure on a single subject and for each predefined position, the amount of memory needed to store the measurement is N sizeo f (double). Moreover, a time-stamp information (i.e. a DATE data type, defined as double, that is 8 bytes) is needed in order to let the system able to detect the most recent measures. This is necessary because the human profile may change over time (e.g. weight changes, aging, etc...). For this reason, during the authentication June, 2006 33 UNIPI

step, the system will take into account small changes in the profile and it will automatically update the database information. This strategy will ensure the system to be able to follow the small profile variations over time if the human subject will frequently log into the authentication system (e.g. once a month, for example). Unfortunately if the subject profile is changed more than a predefined threshold, the user should be rejected and a new enrolment step should be performed in order to ensure the security of the system. Taking into account all the #P predefined position, the amount of data becomes P [N sizeo f (double) + sizeo f (DAT E)]. Moreover, in order to let the artificial neural networks able to train adequately on the human subject, #M repeated measurements are performed for each of the predefined positions for each subject (M=3 in the preliminary prototype; M=10 in the final prototype). Finally, the information about the subject s identifier and about the predefined position s identifier will be stored (unsigned int data type, 4 bytes) into the signature for each measurement. The resulting amount of data is M P [N sizeo f (double) + sizeo f (DAT E) + 2 sizeo f (unsigned int)]. Although the maximum number of strain sensors is now fixed to N=36, we will use N=64 because the hardware device of the final prototype is capable to manage up to 64 channels. In the final prototype the signature of each human subject is composed by 528 bytes for each predefined position. Actually (P=2 and M=10) the signature structure is 10.560 bytes long, however, as it is shown below, both the C++ structure of the client application and the database structure has been designed in order to let a dynamic change in the number of the predefined positions. 5.3.1 Client Application Signature Structure A C++ structure was defined to take into account the information of the signature of a human subject profile. The HumabioSensingSeatMeasurement structure contains the information of a single measurement. The structure takes 528 bytes in an Intel-compatible architecture equipped with a Win32 platform. This structure saves the information of the normalised voltage data of each strain sensor as well as the identifier of the human subject, the identifier of the predefined position and the information about the time-stamp when the measurement was conducted. #define HUMABIO_SENSING_SEAT_N 64 struct HumabioSensingSeatMeasurement { }; // the identifier of the corresponding predefined position unsigned int idposition; // the identifier of the human subject unsigned int idsubject; // the time-stamp of the measurement DATE datetime; double measurement[humabio_sensing_seat_n]; June, 2006 34 UNIPI

The HumabioSensingSeatSignature class encapsulates several methods useful for the client application. However it may be considered as a dynamic array of measurements. This allow us to have a flexibility in choosing the number of the repeated measures and of the predefined positions without changing the data structures. class HumabioSensingSeatSignature { }; public:... public: // the signature is a dynamic array of measures HumabioSensingSeatMeasurement * signature;... In order to guarantee the easy storage form the above structure to the data field of the BIR structure of the BioSec API (and vice-versa) two methods are supplied: void HumabioSensingSeatSignature2Buffer(const HumabioSensingSeatSignature * psignature, unsigned char * pbuffer); void Buffer2HumabioSensingSeatSignature(const unsigned char * pbuffer, HumabioSensingSeatSignature * psignature); 5.4 Database Architecture The database structure contains in the following tables all the information needed by the sensing seat system: 1) tab_measurements: this table contains the information on a single measurement. Since many measurements may be saved for the same subject (id_subject) and for the same predefined position (id_position), an additional auto-increment field is supplied as key-field (id_measurement). The time-stamp (measurement_datetime) is stored together with the steady state voltage value of each strain sensor (sensor_001... sensor_064). id_measurement*: integer id_subject: integer id_position: integer measurement_datetime: datetime sensor_001: double June, 2006 35 UNIPI

... sensor_064: double 2) tab_subjects: this table is just a look-up table to get some other information about the user. The auto-increment key-field is the subject s identifier (id_subject), while the name of the subject (name) should be useful for the client GUI operations. id_subject*: integer name: text 3) tab_positions: this table is just a look-up table to obtain some information about each predefined position. The auto-increment key-field is the predefined position s identifier (id_subject), while the description of the position (description) should be useful for the client GUI operations. id_position*: integer description: text 5.5 Final Sensing Seat Prototype The realisation of the final prototype includes the realisation the final sensing seat (Fig. 5.4), as well as the implementation of the running classification software for both enrolment and authentication tasks (Fig. 5.5). The final prototype includes the sensing cover parts (i.e. fabrics screen printed with strain sensors), an electronic interface (signal conditioning, A/D conversion and transmission) and the software classification module for enrolment, authentication and monitoring tasks. According to the use of multiple predefined positions, the control application is designed to handle a dynamic change of the number of the predefined positions. As the application starts, the system loads the subject signature from database and the classification modules are trained. As an enrolment or authentication session is successful completed, the training step is repeated in order to take into account the new signatures. In particular, since for each subject only the ten most recent measurements will loaded from the database, the described procedure guarantees the adaptation of the system on the short-time small changes in the physical structure of each subject. 5.6 Software Analysis The block diagram of the sensing seat authentication module is presented for each task (enrolment, authentication). The different scenarios (truck driver, car driver, airplane pilot, human operators high-security environments such as power plants or industrial refineries) have the same architecture. Next paragraphs analyze in details the block diagrams of user enrolment as well as the running states (authentication, real-time detection of the nobody is seated feature) for the sensing seat module. It should be noted that the June, 2006 36 UNIPI

a) b) Figure 5.4: The final sensing seat prototype without the external cover Figure 5.5: The classification software. The GUI is visible at the top-left, showing the real-time acquired signals from the sensing system, the classification result for all the classification sub-modules and the final authentication result. The state of the artificial neural network modules are shown in the bottom-right window. The signals acquired in real-time are also shown in the top-right window. June, 2006 37 UNIPI

Figure 5.6: Block diagram of the sensing seat authentication system sensing seat module is a part of the overall HUMABIO system. This system is actually based on the UNIPI API and the porting to BioSec API will be performed during the next months. However, as for the BioSec API, several components of the sensing seat module use an interface to acquire, extract and store/request the human profile data. The general block schema is showed in Fig. 5.6. The signals coming from the strain sensors of the sensing seat cover are acquired by a data acquisition board (DAQ). An interface (DRIVER) performs the data protocol translation to the server task. The data will be subsequently dispatched to the client task that will perform the specific tasks: enrolment, authentication and monitoring. The server and client tasks run asynchronously and a sample and hold strategy is applied in the client-side application. As the application starts, the system loads the subject signature from database and the classification modules are trained. As an enrolment or authentication session is successful completed, the training step is repeated in order to take into account the new signatures. In particular, since for each subject only the ten most recent measurements will loaded from the database, the described procedure guarantees the adaptation of the system on the short-time small changes in the physical structure of each subject. 5.6.1 Classication algorithms The classification module is based on three kind of classifiers (sub-modules): 1. a distance classifier (mainly used to detect if nobody is seated on the sensing seat) 2. a Kohonen Self organizing Map (KSOM) 3. a Multi Layer Perceptron (MLP) 5.6.2 Global classication module As for the preliminary prototype, a global classifier for each predefined position was trained and tested making use of the data belonging to the new measurement campaign. Advantages and drawback may be summarized as it follows: advantages: only 2 classification module: the sub-modules are trained to recognize all the enrolled subjects (known users) June, 2006 38 UNIPI

good TAR, FAR, TRR, FRR over subjects enrolled into the system drawbacks: bad FAR and TRR over subjects not enrolled into the system (unknown users, impostors) 5.6.3 Personal classiers In order to improve the performance of the FAR and TAR over unknown subjects (impostors), one classification module was created for each of the enrolled users and for each predefined position. Each sub-module is trained in order to recognize all the other enrolled subjects as impostors. That is, the classifiers are trained to distinguish between the subject and the impostors. Advantages and drawback may be summarized as it follows: advantages: good TAR, FAR, TRR, FRR over subjects enrolled into the system good FAR and TRR over subjects not enrolled into the system (unknown users, impostors) drawbacks: 2 classification modules for each enrolled subject (need more time for training) the time needed for the training step is much more higher than the previous solution 5.6.4 Enrolment State In this section, a detailed architecture of the sensing seat module is described, in terms of subject enrolment/registration. Main purpose of the user enrolment is to add new users to the system using their profile biometric signatures that will be used later in the authentication states of the system. The overall architecture for the user enrolment is illustrated in Fig. 5.7. During the initialisation step, the new user is asked to insert the login information. These data will be stored together with the biometric profile when the enrolment step will be completed. The user is asked to seat in each predefined position as it is defined in the measurement protocol. For each of the predefined positions, the data of the sensing cover are acquired in real-time. The user is asked to stay still until the steady state value is gained for each signal. This process takes a few seconds (2 or 3 seconds) for each of the predefined position. Data are subsequently stored into the PCs RAM and the whole process is repeated until all the predefined positions are taken into account. The acquired data belonging to the new user for all the predefined positions are used to train the classification modules in order to take into account the new user inside the system. The state of the classification modules, the acquired data and the user login information are stored into the database. These information will be used in the authentication step. June, 2006 39 UNIPI

Figure 5.7: Flow chart of the enrolment task Figure 5.8: Flow chart of the authentication task 5.6.5 Authentication State In this section, a detailed architecture of the sensing seat recognition system is presented, in terms of subject initial authentication. In this running state, namely initial authentication, the sensing seat module is used to give access to a user to a secured or controlled area. Initially, a subject presents a new profile signature to the system and claims to be an identity that exists in the system. Then, the sensing seat system compares its signature with the stored profile signature that corresponds to the claimed identity. Using the recognition module, the subject is accepted/rejected. The overall architecture of the recognition system is illustrated in Fig. 5.8. Summarizing, the overall conceptual architecture of the user initial authentication system is quite similar to the architecture used for the enrolment state. However in this case the user must enter the login information choosing among the users already enrolled into the system (i.e. the user claims to be someone). After the data acquisition step for all the predefined positions, the actual biometric profile is available and the needed data of the internal state of the classification modules June, 2006 40 UNIPI

Figure 5.9: Flow chart of the monitoring task are loaded from the database. The analysis proceeds taking into account the data loaded from the database and the biometric profile acquired for the logged user. In this process the idea of the multimodal authentication of the entire HUMABIO project is used recursively. That is, the same biometric data are sent to several classification sub-modules (PCA, LDA, SOM, MLP). Each sub-module performs the analysis in an independent way giving an authentication result (i.e. the recognized user) and an authentication score (i.e. the authentication error level). The authentication output of each sub-module is taken into account by a final classifier in order to obtain a properly weighted authentication result. The final output consists in a user identifier and in a identification error level. If the user identifier recognised by the system is not the same of the login information, than the user is rejected. If the user identifier recognised by the sensing seat system is equal to the login information identifier and the error level is less than a predefined threshold, then the user will be authorised. 5.6.6 Monitoring State In this section a description of the sensing seat system in the monitoring state is described. During the normal work activities the human subject is free to change his position on the seat and the authentication procedure is not applicable in a continuous way. However the sensing seat module is able to continuously detect if a human subject is not seated (i.e. nobody is placed on the sensing seat). This is possible because the nobody user is a user enrolled into the system as well as a normal user. If the steady state is gained and the authentication result returns the nobody user identifier as output, a system warning message should be dispatched in order to make the overall system able to take the appropriate decision. The architecture is similar to the one used for the authentication state. However in this case all the modules are running in real time and the system warning message is scenario-dependant. The architecture of the sensing seat in monitoring state is showed in Fig. 5.9. June, 2006 41 UNIPI

5.7 Classication Software Overview A software interface (DRIVER) is needed in order to perform an initial data filtering and data dispatching to the analysis application. In order to obtain a multitasking architecture, the acquisition task (SERVER TASK) runs in parallel in respect to the analysis task (CLIENT TASK). A message protocol was defined in order to perform the client-server communication. After an initialization step, the server task will communicate new data only when they are available. Currently the server-side HW/SW is able to obtain a sampling rate of 1000 samples per sensor per second. However a sampling rate of 64 samples per sensor per second is sufficient to obtain good results within the sensing seat application. The analysis task (CLIENT TASK) is able to run in real time and to collect new data from the data dispatcher task using a sample-and-hold strategy. This task manages the Graphical User Interface (GUI), the network connection to the database (DB) and the switching between the operating modalities (enrolment, authentication, monitoring). The software application is developed under the Microsoft Visual C++ 6.0 programming environment. Actually the UNIPI Artificial Reasoning Interface (ARI) API is being used in order to perform the sensory system management and the analysis tool processing [9]. During the next months the software porting to the BioSec API will be developed. Each module receives the input vector from the acquisition system. The input vector is an array of voltage information whose elements are the normalised output of the sensing seat system (each element is a floating point value representing the differential of voltage potential of each strain sensor). The sensing seat prototype is equipped with N=36 strain sensors. Each classifier receives the filtered input data (low pass filter with a cut frequency equal to 10 Hz). The system automatically detects when all the signals gain the steady state value in order to trigger the activation of the classifiers. At this time, three classification modules are used during the enrolment, authentication and monitoring states of the system: 1) A Kohonen Self Organizing Map [11, 12]: W W integrate-and-fire neurons fully connected to the input vector (#N strain sensors); W is equal to the number of enrolled users in the case of the global classifier; W is equal to 10 in the case of the personal classifier; Distance-based learning method; Learning rate with decay, learning radius with decay; Labelling procedure for the classification task; During the test phase, the error level is the Euclidean distance between the input vector and the synaptic weight vector of the winning neuron. 2) A Multi Layer Perceptron [10]: Input layer: #N integrate-and-fire neurons connected one-to-one to the input vector (#N strain sensors), linear transferring function; June, 2006 42 UNIPI

Hidden layer: 10 integrate-and-fire neurons, fully connected to the input layer and to 1 bias neuron, sigmoid transferring function; Output layer: #W integrate-and-fire-neurons, fully connected to the hidden layer and to 1 bias neuron; W is equal to the number of enrolled users U in the case of the global classifier; W is equal to 2 in the case of the personal classifier; Learning rate with decay, momentum with decay; Delta-rule learning method; During the test phase, the error level is the Euclidean distance between the activation vector of the output layer and the activation vector built on the basis of the logged user information. 3) A distance classifier: Distance-based classification method: evaluation of the Euclidean distance between the input vector and the data vector belonging to the logged user; The output of this modules is its error level (i.e. the distance described above). 5.7.1 Enrolment State During the enrolment state, according to the conceptual design, the data belonging to new user are collected and subsequently stored into the database. The data set will be rebuild and the SOM and the MLP will be trained again in order to take into account the new user. 500 training epochs are sufficient in order to setup the neural networks. The initial values for the networks parameters are: 1) SOM: Learning rate: 0.9; Learning rate decay: 0.001; Learning radius: 10; Learning radius decay: 0.001; 2) MLP: Learning rate: 0.9; Learning rate decay: 0.001; Momentum: 0.9; Momentum decay: 0.001; June, 2006 43 UNIPI

5.7.2 Authentication State During the authentication state, according to the conceptual design, the data belonging to logged user are collected and the status of the internal classifiers are loaded from the database. The three classifiers will perform the classification task. Each classifier will supply a user identifier and an error level as result of the authentication process. The authentication output of each sub-module will be taken into account by a final classifier in order to obtain a properly weighted authentication result. The final output consists in a user identifier and in an identification error level. If the user identifier recognised by the system is not the same of the login information, the user will be rejected. If the user identifier recognised by the sensing seat system is equal to the login information identifier and the error level is less than a predefined threshold, then the user will be authorised by the sensing seat system. 5.7.3 Monitoring State During the monitoring state, according to the conceptual design, the signals from the sensing seat are acquired in real-time. The status of the internal classifiers are loaded from the database during an initialisation step. The three classifiers will perform the classification task in real time. Each classifier will supply a user identifier and an error level as result of the authentication process. The authentication output of each sub-module will be taken into account by a final classifier in order to obtain a properly weighted authentication result. The final output consists in a user identifier and in a identification error level. If the user identifier recognised by the system is the nobody user identifier, a warning message will be dispatched to the main monitoring system. This strategy is scenario-dependant. 5.7.4 Communication Architecture In the sensing seat system, three main modules must be able to communicate each-other. A communication is established between the sensory system (the sensing cover) and the acquisition board (DAQ). This communication is performed by a 44 pin high-density cable for analogical signals. A communication is established between the DAQ and the software analysis module. The communication is DAQ-dependant. Actually the USB and PCMCIA protocols are available. A communication is established between the software analysis module and the remote HUMABIO database. The TCP-IP protocol and secure socket layer library are currently being used. Moreover, the software analysis module is composed by several running process running in parallel on the same machine. The communication is granted by an internal protocol based on notification messages. 5.8 Hardware specications Physical characteristics for a single strain sensor: June, 2006 44 UNIPI

Dimensions (cm x cm x cm) 0.5 x 20 x 0.01 Volume(cm3) 0.1 Mass (grams) 0.1 Specification for a single strain sensor: Output (Analogue or Digital): Analogue Data rate-bandwidth: 20Hz Sampling rate [Hz]: 1000Hz Quantisation [bits]: 12 Dynamic Range [V]: 0 ; 2 Resolution [V]: 0.01 Frequency range: 0 to 20 Hz Amplitude range: 0 to 2 V Environmental specifications for a single strain sensor: Operating range (Min Temperature to Max Temperature) ºC: -20 ; +70 Storage range (Min Temperature to Max Temperature) ºC: -30 ; +100 Specifications for the data acquisition board front-end: Power supply: 220V @ 50 Hz PC specifications (Minimum/Recommended Requirements): Processor AMD or Intel @ 500 MHz AMD / Intel @ 1 GHz Operating System Microsoft Windows 2000 / Microsoft Windows XP SP2 RAM 256 MBytes / 512 MBytes Hard Disk 1 GByte / 2 GBytes VGA Card Standard VGA / AGP Video Card with OpenGL support Power supply 220 V @ 50 Hz June, 2006 45 UNIPI

6 Measurment Campaign and Experimental Results As the final prototype has been completed, a new measurement campaign was conducted over 40 human subjects, including 20 male and 20 female subjects ranging from 20 to 40 years old and from 50 to 90 Kg. The sensing cover prototype is equipped with 36 strain sensors connected to the a connector plug placed on the left side of the cover. The sensing cover is close-fitting to the truck seat provided by VOLVO. A protective cover is moreover placed over the sensing cover in order to keep the strain sensors in a safe position avoiding any possibility of damages. A 44-pin cable connects the sensing cover to the hardware front-end device that performs a signal conditioning (high-impedance measurement) for each strain sensor channel. The front-end device imposes a reference current and sends the voltages information to the data acquisition board (a National Instrument NI-cDAQ-9172 was used during the tests). The DAQ is connected to the PC through the PCMCIA bus. The high level application contains the software framework described in the previous sections. Once the system is connected, the signal are acquired in real-time with a sampling rate equal to 100 Hz. As described above, the processing application runs asynchronously in respect to the data acquisition process. The experimental setup is shown in Fig. 6.1. During the enrollment step, the GUI asks the new user to insert a nick-name as a reference for the future sessions. The user is also asked to remove wallets and/or coats and to seat in the normal position (with the bottom side and the back side totally adherent to the seat, i.e. the first predefined position). When a steady state is gained for each strain sensor signal, the collected data are stored and the user is asked to seat in the next predefined position (the bottom side is adherent to the seat and the back side is not in contact Figure 6.1: Experimental setup June, 2006 46 UNIPI

a) b) Figure 6.2: Global classification module: each colour corresponds to an enrolled subject; a) KSOM; b) MLP a) b) Figure 6.3: Personal classification module; a) KSOM; b) MLP with the seat, i.e. the second predefined position). When the steady state is reached the collected data are stored and the procedure is repeated for 10 times in order to obtain 10 repeated measurements. Since the steady state for each predefined position is reached in about 2.5 seconds, considering #P predefined position and #M repeated measurements, the enrollment procedure takes M P 2.5 seconds for each new subject. Actually M=10 and P=2 implies an enrollment procedure of 50 seconds plus the delay due to the interaction time between the GUI and the user. During the enrollment task, the processing system runs at 64 FPS. After the enrollment of each subject, the system stores the collected data (belonging to the user for each predefined position and for each repeated measurement) and the classifiers are trained again. The training process tries to run as fast as it can: in a standard PC with a 2.5 GHz CPU, 1GB RAM, the training system runs at ~40000 FPS so that the training process takes less than 0.5 seconds per classification module. 6.1 Data Analysis The global classification module was trained over all the subjects (Fig. 6.2) and the tests were conducted using known subjects and unknown ones (impostors). For each enrolled subject, the personal classifiers were trained on the basis of the data belonging to the subject and to half of the impostors (Fig. 6.3). All the combinations were performed. In order to evaluate TAR, FAR, TRR, FRR and EER, thresholds were used in the KSOM June, 2006 47 UNIPI

TAR FAR TRR FRR EER KSOM (1 st position) 90.4% 5.4% 94.6% 9.6% 2.8% MLP (1 st position) 93.2% 4.3% 95.7% 6.8% 2.2% KSOM (2 nd position) 93.2% 4.0% 96.0% 6.8% 2.0% MLP (2 nd position) 94.3% 3.0% 97.0% 5.7% 1.7% Table 6.1: Acceptance and rejection rates using the personal classification modules. Values are averaged over the classification modules belonging to all the enrolled subjects. Each classification module stores internally the threshold level the corresponds to the equal error rate and in the MLP classifiers: KSOM: the winning neuron has the minimum Euclidean distance between its synaptic weights and the input data; a low distance value (min=0.0) corresponds to a high classification level, while a high distance value corresponds to a low classification level (Error level = distance[ (synaptic weights of the winning neuron), (input data) ]); if the distance value of the winning neuron is higher than the threshold, the system rejects the subject even if he was recognised by the KSOM. MLP: the winning neuron of the output layer has maximum activation layer; a low activation value (min=0.0) corresponds to a low classification level, while a high activation value (max=1.0) corresponds to a high classification level (Error level = 1.0 - (activation level of the winning neuron of the OL)); if the activation value of the winning neuron is lower than the threshold, the system rejects the subject even if he was recognised by the MLP. After a training step with 1000 epochs, the global classification module is able to detect each enrolled subject but fails to correctly classify the impostor subjects. Better results were obtained using personal classifiers as reported in Table 6.1. Fig. 6.4 shows some training and test details belonging to a particular subject and to the first predefined position. Both for the KSOM and the MLP, the training error decay is clearly visible in the top-left plot. The FAR-FRR curve (bottom-left plot) permits to evaluate the EER and to establish the optimal threshold level. The bottom-right plot represents the biometric confusion matrix (BCM). The element BCM 1,1 corresponds to the TAR (blue color), while BCM 2,2 corresponds to the TRR (brown color). The errors are represented by BCM 1,2 (FRR) and by BCM 2,1 (FAR). For the KSOM, the labeled map is shown in the top-center of the figure, where the blue area represents the cluster belonging to the owner subject and the brown area corresponds to the cluster belonging to the impostors. 6.2 Limits of the existing architecture The sensing seat system is able to operate a classification on the basis of the deformation of the sensing cover caused by the human subject when seated on the sensing seat in different predefined positions. Three main problems may occur, making the system unable to correctly perform the classification task: June, 2006 48 UNIPI

a) b) Figure 6.4: Details of a personal classification module belonging to a particular subject; a) KSOM; b) MLP 1. presence of wallet and/or coat 2. changes in the way the same human subject seats in each predefined position 3. changes in the human subject s profile (ageing, changes in weight) For each of these situations, the final prototype system is able to try to solve the problem. The point 1) will result in the human subject signature to be not so far from the signature information stored into the database. Both the enrolment and the authentication procedures are driven by the system, that will inform the user to remove wallets and coat before than starting the session. However, since each classifier of the sensing seat system is able to supply an error level, a threshold should be defined in order to detect if the system does not know if the user has to be rejected. I this is the case, a system warning is supplied through the GUI and the user is asked again to remove wallets and coat. The points 2) and 3) are difficult to be taken into account. However the time-stamp information for each measurement in the signature was taken into account for this kind of situations. We will suppose that the changes (the way of sitting, the weight, the aging) will be small especially if they are considered in short period of time. Once the user is enrolled the time-stamp is saved into the signature for each measurement (related to several repeated measurement for each predefined position). If the user continuously performs the authentication task in the system (let us say, once a month at least) and if he will be correctly authenticated, the new measurements (probably a bit different from the previous measurement) should be saved into the database and the classifiers should be trained again taking into account a LIFO queue of measurements June, 2006 49 UNIPI