Smart Objects for Human Computer Interaction, Experimental Study

Transcription

1 Smart Objects for Human Computer Interaction, Experimental Study Doggen, Jeroen; Neefs, Jef; Brands, Enzo; Peeters, Tom; Bracke, Jerry; Smets, Marc; Van der Schueren, Filip January 31, 2012 Artesis University College of Antwerp, Department of Applied Engineering and Technology Paardenmarkt 92, B 2000 ANTWERPEN Jeroen.Doggen@artesis.be Abstract The fields of embedded systems and networking are being merged, giving rise to the Internet Of Things, built with smart objects that allow people to interact with everyday objects and vice-versa. We developed a cube, equipped with an Arduino based development board, various sensors and an XBee wireless interface, which enables us to develop new methods of human computer interaction. The cube was tested as an input controller for several computer games. The software was released under an open source license. This system will be used to develop new sensor driven applications. Keywords: Smart Objects, Human Computer Interaction, Motion Sensing, Distance Sensing, Arduino. 1 Introduction Advances in technology development, made possible by combining the fields of embedded systems, networking and sensor systems, give rise to new types of applications. Wireless networks of sensor equipped embedded systems, often called smart objects, are connected together to form the Internet Of Things. Household appliances and other consumer products have been equipped with electronics for decades. These electronics contain a lot of useful information about the objects they are embedded in. Most of this information is relayed locally to the users via displays and control lights who then can respond to that information accordingly. Radio-frequency identification (RFID) tags were the first mainstream application of electronic tagging of everyday non-intelligent objects. RFID tags provide static and rigid information about clothing and other supermarket or warehouse goods in general. These tags typically contain a limited amount

2 of information: price, object identification and manufacturing date. In the meanwhile we can find RFID tags with up to 1 MB (active) and 8 kb (passive) of storage space. Near Field Communication technology (NFC) extended the use of RFID by improving and standardising the interaction amongst reader and tag and readers themselves allowing more complex applications [1]. The field of Micro-Electrical-Mechanical Systems (MEMS), where very small mechanical devices allow on chip measurement of motion and rotation of rigid objects, gave rise to straightforward motion related applications, e.g. collision detection [2]. The next logical step in technology development is mixing the fields of information sharing by passive tags and information gathering by active devices. Unlike the data in most of the contemporary RFID tag applications, the information contained in smart objects is very diverse and changes rapidly. These smart objects will not only know what they are, but also where they are [3]; they will sense their environment and communicate with other smart objects to increase sensing efficiency [4] and will interact in new ways with the end-user [5]. 2 Problem Statement Our primary goal is to combine several sensor technologies and data sources to test new methods of human computer interaction. In a previous project, we implemented a customised implementation of a frustrated total internal reflection based multi-touch table [6]. The first idea to improve the functionality of this multi-touch table, was to attach visual masks to all objects. The system could identify the objects by processing the video stream and recognising the masks. The second idea was to make an intelligent object that could deliver the additional context information: the acceleration, rotation and position of the object. By combining these concepts with wireless sensor network technology and by performing on-device sensor preprocessing, we can design new types of applications. The drawback of the visual mask implementation is that the multi-touch table needed a projector and a camera for the mask detection. This makes the implementation bulky, expensive and complex to build. On top of that all objects need to have a visible mask on one or more sides so they can be detected by the camera and software. Recently backlit Infrared (IR) touchscreens removed the need for a camera. But this new technology is still quite expensive, a table sized model would cost you as much as three high-end liquid crystal display (LCD) televisions of the same size. We believe that the smart object solution, implementing sensing and

3 processing technology can be less expensive and is able to increase the possibilities of the setup. Figure 1 shows a mockup of our envisioned implementation. A smart object, equipped with a triple axis accelerometer, is able to detect which side of the object is facing upwards, implementing a gyroscope on this smart object makes the object orientation aware. Well placed distance sensors allow the object to detect its height above the screen when picked up. This removes the need for a projector and camera or an expensive IR backlit display, because no masks need to be detected to determine the object s orientation and movement. The smart object could have features that were previously impossible: e.g. activating different functions depending on the upward facing side. Extra inputs can be derived from the device without the device having to touch the screen, e.g.: tap, pick-up, shake and drop. The possibilities are only limited by the sensors one can attach to such a smart object. Figure 1: Smart Cube for Human Computer Interaction 3 System Design For this experimental study we developed a smart object based on the popular Arduino development platform, which we equipped with an accelerometer, a gyroscope and distance sensors. This platform has gained increased popularity over the last years. The Arduino open-source community has over registered users and an abundance of user submitted libraries. 3.1 System Architecture The complete system consists of two main blocks: a wireless module / cube and a PC equipped with a XBee wireless interface. The movements of the cube are detected using motion sensors. We implemented two types of motion sensors: a triple axis accelerometer and a triple

4 Figure 2: System Architecture axis gyroscope. The basic design uses an analog accelerometer, the signals from the three axes are sampled using three of the eight channels of the 10- bit analog-to-digital converter (ADC) on the ATmega328P microcontroller [7]. The gyroscope on the other hand is a digital one, which sends its data over the Inter-Integrated Circuit interface interface (I 2 C) to the development board. The microcontroller runs the embedded code to process sensor and user events, making the object smart. The microcontroller is connected to the XBee module over the serial interface. Since the interface is not commonly available on computers, we need an extension on this side to connect our smart object to the computer. We connected an XBee module to an XBee shield, which in turn is connected to an Arduino board where the microcontroller has been removed. This module provides a serial port to the computer over the Universal Serial Bus (USB) interface using a Future Technology Devices International (FTDI) chip. Now the interface is available on the computer. On that computer we then run a server application whose task is to forward the serial data towards client applications that make a connection over a Transmission Control Protocol (TCP) socket. The library to connect to the system and several demo applications are described in section and section Hardware Specifications A typical smart object contains five building blocks: CPU, sensors, wireless communication interface, memory and a power source. In this project we chose these blocks considering the following constraints: price, ease of use, flexibility and energy consumption.

5 Figure 3: Partially Disassembled Cube: Sensors, XBee and Microcontroller Board Seeeduino Development Board The price tag has to be low, around $100, to allow production of custom printed circuit board (PCB) variants on the design. The schematics will be available online under an open source license. Since these designs will also be constructed and altered by undergraduate students, we needed a well documented microcontroller with an easy to use yet flexible and powerful development environment. Hence we chose an Arduino compatible development board: Seeeduino. Seeeduino s design is based on the original Arduino, existing software and hardware is 100% compatible, but on the hardware side, it has many improvements, e.g. power efficient surface mount device (SMD) components and extra analog and digital I/O pins. Its most important hardware specifications are: 14 digital I/O pins, 8 analog inputs and a 16 MHz crystal oscillator [8]. XBee Wireless Interface: The system was not designed to be ultralow-power, but it should be able to operate over a period of several days using 2 AA batteries. Hence we chose the MaxStream/Digi XBee low-power wireless communication modules to connect the cube to the basestation [9]. Analog Accelerometer The Freescale Semiconductor MMA7361 accelerometer is a cheap, triple axis sensor that can be configured for ±1.5 g and ± 6 g operation allowing a broad range of applications [10]. Digital Gyroscope The InvenSense ITG-3200 gyroscope is a singlechip, digital output, triple axis MEMS sensor, optimised for human computer interaction. Internally, the chip uses a three channel 16-bit ADC.

6 The chip connects to the outside world over a Fast-Mode I 2 C interface, running at 400 kb/s [11]. Infrared Distance Sensor The Sharp GP2Y0A21YK infrared Distance sensor sends out a pulse of IR light that travels out in the field of view and hits an object. The reflected light creates a triangle between the point of reflection, the emitter, and the detector. The angles in this triangle vary based on the distance to the object. A precision lens directs the reflected light onto various portions of the enclosed linear CCD array based on the angle. The sensor varies its output voltage accordingly [12]. 3.3 Software Libraries The Arduino library provides high level events or sensor data based on the movement of a smart object. The XNA library provides easy access to these events from a C# software development perspective Arduino Libraries The Arduino development environment makes it straightforward to start embedded development, create libraries and share them with the online community. We used an existing software library to interface with the ITG Gyro [13]. Because there was no library available for the MMA7361 accelerometer, we developed our own library and published it on the Arduino Forum [14]. After 9 months we had around 600 downloads of our acceleromma7361 library. The distance sensor library provides a unified and easy to use interface for ultrasonic and infrared distance sensors. The libraries are published as open source projects on Google code [15] [16]. AcceleroMMA7361: Library for retrieving data from the MMA7361 accelerometer. It provides sensor data as a plain binary ADC output, as an absolute output voltage and in reference to earth s gravitation. It also delivers a calibrate function to determine a horizontal reference position, by setting static offsets on the accelerometer axes in case of a slightly tilted starting position at boot time. AcceleroDice: Library which determines the upward facing side of the cube. The code sample in listing 1 shows a minimal application, implemented using this library. In the setup phase, the cube is initialised and calibrated using the AcceleroMMA7361 library. During operation, the system detects which axes the gravitational acceleration is working on and calculates its upward facing side based on averaging and thresholding. This information is updated on each movement.

7 Listing 1: AcceleroDice: Minimal Implementation 1 #include AcceleroMMA7361. h 2 #include AcceleroDice. h 3 #include <Arduino. h> 4 5 AcceleroMMA7361 a c c e l e r o ; 6 A ccelerodice d i c e ; 7 8 void setup ( ) 9 { 10 S e r i a l. begin ( ) ; 11 a c c e l e r o. begin ( 1 3, 12, 11, 10, A0, A1, A2 ) ; 12 a c c e l e r o. setarefvoltage ( 3. 3 ) ; //AREF v o l t a g e = 3.3V 13 a c c e l e r o. s e t S e n s i t i v i t y (LOW) ; // S e n s i t i v i t y = +/ 6G 14 a c c e l e r o. c a l i b r a t e ( ) ; 15 d i c e. begin ( a c c e l e r o ) ; 16 } void loop ( ) 19 { 20 i f ( d i c e. sidechanged ( ) ) 21 { 22 S e r i a l. p r i n t l n ( d i c e. g e t S i d e ( ) ) ; 23 } 24 } AcceleroArrow: Library for using the accelerometer as the arrow keyboard input. The library detects sudden movement and sends an Arrow- Pushed event when a certain acceleration threshold is reached. It stops sending the event when a total deceleration, resulting in the opposite of the first acceleration, is reached. Distance Sensor Library: The analog sensor s output voltage is used to get the corresponding distances from a lookup table. The first version of the lookup table was derived from an approximation of the transfer function based on the sensor datasheet. This approach resulted in inaccurate results, possibly due to inter-sensor differences. To improve our results, we tested several sensors by reading their output voltages while placing an object at known distances ranging from 5 cm up to 80 cm. The summarised results in figure 4 show the average sensor output and two polynomial approximations for the transfer functions: one for the steep part and one for the flat part of the slope. We calculated the remaining points of the 256 byte lookup table using the two transfer functions. This static information is placed in code memory taking the limited amount of random access memory on the ATmega328P into account.

8 Figure 4: Transfer Functions for the GP2Y0A21YK Infrared Distance Sensor XNA Library Microsoft XNA is a set of software tools that facilitate video game development and management. XNA attempts to free game developers from writing low-level code. Our XNA library uses a TCP/IP socket connection to interface with the smart objects. This gives software developers easy access the features of the module, without extended knowledge of the system. It can be used to read the X, Y and Z values, to detect a jump and to generate arrow events. 4 Applications We developed several tools and performed experiments to calibrate the sensors and verify proper system operation while developing the Arduino libraries. The libraries were used to modify 3 in-house developed computer games: a 3-D flight simulator, a 2-D platform game and a Pac-man clone. 4.1 Signal Conditioning Most MEMS inertial sensors are comb based mechanical structures fabricated on silicon wafers. The complex fabrication steps required during MEMS production often result in fabrication deficiencies causing signal differences between individual sensors. These sensor errors have been studied thoroughly [17] and several possible solutions have been proposed [18]. Expensive sensors are often factory calibrated to compensate for these undesirable effects. We implemented a rudimentary software calibration method in the acceleromma7361 Arduino library to take individual sensor differences and mechanical misalignment into account. Based on user-feedback, we decided to implement and compare three simple signal processing algorithms: variable-length weighted moving average filter,

9 Bessel and Chebychev low-pass filters. At first these filters were implemented and optimised in a C# application. In a later version the filters were moved to the microcontroller. As illustrated in figure 5 these simple filters don t have the desired effect when used to filter the accelerometer data. The low-pass filter does not change the signal significantly and the averaging filter introduces to much delay. The next steps in development will involve on-device Kalman filtering and allowing the smart object to adjust signal processing settings based on the application requirements. The library is published as an open source project on Google code [19]. Figure 5: Original data - low-pass filter - moving average filter 4.2 Software Tools SerialMonitor: This monitoring tool was developed to log and analyse sensor data and serial communication. The full system does not need this tool, but it was used extensively during development. SerialForwarder: This application, developed using the processing programming language [20], runs on the XBee-enabled computer. It connects to the serial port and provides two way communication between a TCP/IP socket and the serial port. This allows most network enabled applications to communicate with our cube. 4.3 Demo Applications In the flight simulator, we control the movement of an airplane, flying over a 3-D landscape and passing though rings using the sensor data as a joystick as shown in figure 6. We developed the AcceleroArrow Arduino library to control the Pac-man clone. The 2-D platform game required a new feature on the smart object: jump detection. This was implemented by detecting sudden upward movements, with a direction opposed to earth s gravitational pull, independent of the object s orientation.

10 Figure 6: Space ship game 5 Future Work The next steps in development will be enhancing existing non-smart systems with our cube. Two applications that are currently being developed are a sensor enhanced radio-controlled car and a model train railroad where trains communicate to achieve common goals. Motion sensors are used for collision detection and IR sensors for collision avoidance. To test the reliability and performance of the measuring system, we need to generate movement data under the same mechanical conditions, within an acceptable degree of error. For this goal we plan to put the cube on a turntable with controllable rotation speed, in this way the sensors will complete a periodical motion in a two-axis system. By controlling and measuring the speed of the turntable we can calculate a reference value for the data we can expect from the motion sensors. From this test setup we plan to evaluate error rates for multiple speeds and processing algorithms. The distance sensor library needs some extra fine-tuning to optimise the library for code size. We plan to replace the 256 byte lookup table with a smaller table by performing linear interpolation between a limited set of points. This might be feasible because of the nearly linear relationship between the ADC output and the reciprocal of the distance. This will allow us to perform very rudimentary run-time calibration and thereby automatically supporting various IR distance sensors. Currently, most of the features of the smart objects are static and determined at compilation time. We plan to develop a more extensive set of commands to allow the end-user applications to adjust the on-device processing, e.g.: adjust filter parameters or more complex system behavior. On the hardware side we plan to reduce the power consumption by replacing the XBee module with NRF24L GHz low power transceiver produced by Nordic Semiconductor [21].

11 We plan to extend the sensor library enabling other developers who lack the fundamental signal processing skills to use digital filters in their applications. 6 Conclusion We developed a cube, a smart object, based on the popular Arduino hardware platform. This smart cube, equipped with motion sensors and a microcontroller, processes its sensor inputs and provides movement data as requested by an end-user application. The end-user application is connected over a TCP/IP network to a control server, which in turn is connected to the cube using a wireless ZigBee network connection. We developed software libraries for the MMA7361 accelerometer and the SRF04 and GP2Y0A21YK distance sensors. The source code of these libraries was introduced on the Arduino Forum and is now publicly available under the GNU Lesser General Public License on the Google code website. The cube was tested as an input controller for several different computer games. New features where implemented in the software libraries, based on the needs of these computer games. This system will be used to develop new sensor driven applications and to enhance existing non-smart objects to enable completely new features. From an educational perspective, this design offers a lot of opportunities for student projects. Extended versions will house multiple sensors, implement various signal processing methods and offer many other opportunities. References [1] Neefs, Jef; Schrooyen,Frederik; Doggen, Jeroen and Renckens, Karel; Paper ticketing vs. Electronic Ticketing based on off-line system Tapango, Second International IEEE Workshop on Near Field Communication, pp. 3-8, [2] Jaw-Kuen Shiau; Wei-Yan Luo and Long-Kuang Lee, Three- Axis Electronic Collision Sensor for Airborn Application, 2nd International IEEE Symposium on Systems and Control in Aerospace and Astronautics (ISSCAA), pp. 1-6, [3] Davidson, Pavel; Collin, Jussi and Takala, Jarmo; Application of particle filters for indoor positioning using floor plans, Ubiquitous Positioning Indoor Navigation and Location Based Service (UPINLBS), pp 1-4, [4] Fourati, H.; Manamanni, N.; Afilal, L.; Handrich, Y.; A Nonlinear Filtering Approach for the Attitude and Dynamic Body Acceleration Estimation Based on Inertial and Magnetic Sensors: Bio-Logging Application, IEEE Sensors Journal, Vol. 11, pp , 2011.

12 [5] Siciliano, P.; Leone, A.; Diraco, G.; Distante, C.; Malfatti, M.; Gonzo, L.; Grassi, M.; Lombardi, A.; Rescio, G.; and Malcovati, P.; A networked multisensor system for ambient assisted living application ; 3rd International IEEE Workshop on Advances in sensors and Interfaces (IWASI), pp , [6] Han, Jefferson Y.; Low Cost Multi-Touch Sensing through Frustrated Total Internal Reflection. ; 18th annual ACM symposium on User interface software and technology (UIST 05), pp , [7] 8-bit Atmel 328P Microcontroller, Atmel Corporation. [8] Seeeduino, Arduino compatible development board; seeedstudio.com. [9] XBee/XBee-PRO OEM RF Modules, xbeemultipointmodules.pdf. [10] MMA7361: 3-Axis Acceleration Sensor; Freescale Semiconductor; [11] ITG-3200 Integrated Triple-Axis Digital-Output Gyroscope; InvenSense; [12] GP2Y0A21YK Distance Sensor, Sharp Electronics; http: //sharp-world.com/ [13] Arduino driver for the ITG axis MEMS gyroscope by InvenSense; [14] Neefs, Jef; Library for the Freescale Semiconductor MMA7361 Accelerometer; num= /0. [15] Neefs, Jef; Doggen, Jeroen; Arduino library for the MMA axis MEMS accelerometer; mma7361-library. [16] Doggen, Jeroen; Arduino library for distance sensors; google.com/p/arduino-distance-sensor-library/. [17] Park, M. and Gao, Y., Error Analysis and Stochastic Modeling of Lowcost MEMS Accelerometer, The Journal of Intelligent and Robotic Systems, Vol. 46, pp , [18] S. C. Shen, C. J. Chen, H. J. Huang and C.T. Pan, Evaluation of MEMS Inertial Sensor Module for Underwater Vehicle Navigation Application, International IEEE Conference on Mechanic Automation and Control Engineering (MACE), pp , [19] Doggen, Jeroen; Arduino library for signal filtering; google.com/p/arduino-signal-filtering-library. [20] Processing Programming Language; [21] nrf24l01+ Ultra low power 2.4GHz RF Transceiver; Nordic Semiconductor;