Games and Assistive Technologies for Rehabilitation

Transcription

1 Games and Assistive Technologies for Rehabilitation First Semester Report Fall Semester 2014 Full report By Chris Hesser Corey LeFevre Michael Rowack Prepared to partially fulfill the requirements for ECE401 Department of Electrical and Computer Engineering Colorado State University Fort Collins, Colorado Project advisor(s): Sudeep Pasricha Approved by: Sudeep Pasricha 1

2 ABSTRACT Strokes affect around 700,000 Americans annually, and cerebral palsy (CP) affects 17 million people. These diseases cause upper limb impairments in roughly 40 to 50 percent of the victims. [2] In order to combat the issues brought upon by these debilitating conditions, upper limb rehabilitation is needed. However, rehabilitation can be very tedious; and as a result, patients sometimes give up on the rehabilitation because they do not have the motivation to go through with it. As a result, those who suffer from debilitating diseases lack the quality of life that they once had. The goal of this project is to help improve the quality of life for the people afflicted with these diseases. A unique idea was created three years ago to have computer and electrical engineering students, along with occupational therapy students, use their skills to create an interface for these victims to perform their rehabilitation in a new and entertaining environment. GATOR is a project focused on providing a new type of rehabilitation for patients in need of upper limb rehabilitation. GATOR features games and activities that currently utilize the Leap Motion controller. The games provide positive reinforcement as well as an enjoyable experience for the patient. Developed with JavaScript, HTML5, the suite of games use 2D and 3D game play to give patients the ability to exercise their limbs in an XY plane as well as an XYZ plane. The Leap Motion controller is able to track the user's hand movement in the X Y and Z planes as well as gestures that can interpret grabbing and pointing among others. Up to this point, the project has been successful in accomplishing what Doctors Pasricha and Malcolm have envisioned. Besides some minor issues, the patients enjoy the games. Patients have stated that they enjoy the system and they would enjoy the system more if some improvements were made. After using the system for extended time, occupational therapists believe that patients will be able to perform motor function tests faster than they did before they started using the system. The WOLF motor function test is used to measure the progress that stroke victims are making with their motor skills. With this in mind, it is safe to say that the system can potentially be an effective form of physical rehabilitation; and it can be supplemented to help improve the mobility of these patients afflicted limbs. However, this assumption can only be given after more extensive testing is done. As well as this, it can only be said for patients who have suffered a stroke. The system has yet to be tested by those who have suffered from cerebral palsy (CP), traumatic brain injury (TBI), or spinal cord injury (SCI). In addition, only a small amount of stroke victims have tested the system; and although the system has been effective for stroke patients, it can t be said that the system is effective for all stroke patients. Given that the system has only been tested minimally, future testing needs to be done and the feedback loop with occupational therapy needs to continue. In order to provide coverage for all the different victims who suffer upper limb mobility, more games need to be developed to help people with their specific physical rehabilitation; and more hardware needs to be incorporated to do the same. Besides this, the system does not provide coverage to those who are severely disabled. The next step of the project was to provide a way to help those who are severely disabled and unable to benefit from this system for physical rehabilitation. The system should eventually be able to provide patients with the tools needed to have better control over their environment which will give them a better quality of life. The project introduced methods to do this through the creation of software that utilizes the Emotiv headset which acts as a controller for smart home devices. 2

3 TABLE OF CONTENTS Title 1 Abstract 2 Table of Contents 3 1. Introduction State of GATOR History Initial State Previous Groups Recommendations Requirements and Objectives Patient Requirements Occupational Therapy Requirements Overview of Project Objectives 9 4. Summary of Work Hardware JavaScript / HTML 5 Games Rehabilitation tools Deployment Reports and Statistics for Occupational Therapy Mind Controlled Smart Home Automation Ethics, Potential Market, and Marketability Objectives Analysis Conclusions, Lessons Learned,, and Future Work References 27 Appendix A List of Acronyms 28 Appendix B Project Timelines Appendix C Future Recommendations Acknowledgements 40 List of Tables and Figures Figure 1 - Leap Motion Controller Mounted to Stand 11 Figure 2 - Current Suite of Games 13 Figure 3 - Game Development Time 14 Figure 4 - Leap Frog and Fruit Viking Gameplay 14 Figure 5 - Mirror Therapy Visualization Tool 17 Figure 6 - Gestures 18 Figure 7 - W2P DB management 19 Figure 8 - Python Anywhere Development GUI 19 Figure 9 - Progress Report 20 Figure 10 - Z Wave Smart Devices 22 Figure 11 - Emotiv Epoc + EEG Headset 22 Figure 12 - Patient Using System 23 Table 1 - Previous Teams Work 7 Table 2 Game Development

4 Chapter 1 - INTRODUCTION Every person has had their leg or arm fall asleep. The weird numbness that transitions between tingling and pain is inconvenient at best, but it can be torturous as well. Imagine that feeling not subsiding. Imagine willing a limb to move and only being met with frustration. Imagine having an arm falling asleep forever. This is reality more than a few people, but the range of the body that has fallen asleep for an individual ranges from not being able to move a few fingers to quadriplegia, not having movement in any limb. Because of the wide number of different problems, rehabilitation between rehabilitation patients ranges very dramatically. This is a problem. Health care professionals are very aware of this and create a wide variety of different rehabilitations movements based on an individual s need. The key word in the last sentence is movements. Movements, as in move your arm from point A to point B and repeat. There may be some kind of task or goal, for example, move this pen from point A to point B ; but the underlying task is simple and boring. Those with strong resolve (or maybe easily entertained) will perform these types of rehabilitation without problems. For others, rehabilitation becomes too much. Whether the movements become boring or the pain of rehab becomes a larger burden than accepting the new disability, these therapies do not work for everyone. So how can the problem of boring, methodical, and frustrating therapy be solved? GATOR offers a solution that combines the movements prescribed by doctors to each individual with an exciting new interface that can boost morale and increase patient satisfaction with rehabilitation. Technology has advanced very far since the time of the first rehabilitation movements. Tools like Kinect and the Leap Motion Controller can track body movements. Software is now available that can recognize a face or track eye movements. But despite these new technologies and the knowledge of the human body quickly expanding, the health care industry has difficulties replacing what has worked with what could be better. Finding technology that can overcome this is challenging, and previous teams of the same project faced with this challenge. As new technologies, better computers and devices have emerged, GATOR has adapted and is arriving at a solution that could exit the scene of research and potentially see use in the offices of doctors and therapists. Maybe even in a patient s own home. The GATOR project is approaching these goals because of the unique combination of hardware, software, and web-interfacing. A number of constraints have affected the choices for this combination; and while the current team would love to take credit for all of the work, this framework was due mostly to the advances of the immediate preceding team. After working with CSU s Occupational Therapy department, last year s team decided to use a mobile setup that could be employed using three things: a laptop, a Leap Motion Controller, and access to the internet (see Chapter 2). With such a system, patients can take the system outside of a doctor s office and do rehabilitation in a comfortable home environment. This move created the demand for a set of fun, yet challenging games that run on a computer without demand for lots of processing power. Not a small order! And the previous group may have finished their work, but the current team has inherited these ambitious goals. In addition to the goals set by the previous team, improvements in three main areas needed to be made: game entertainment, therapist feedback, and user inclusion. Chapter 4 illustrates in detail how each of these problems gets addressed. User inclusion specifically means that more types of rehabilitation need to be included in the system to better accommodate many 4

5 different patients. To assess how to make improvements in this area, patient and therapist feedback was carefully analyzed. One of the findings from this was that the previous system was limited by its ease of use and needed a better interface. Many hours were spent increasing the usability of both the front and back end of the website in order to address this problem. The other finding was that the system is limited by the Leap controller and only accommodated upper limb rehabilitation. The future of the system needed an additional piece of hardware to make rehabilitation possible for more people. The initial thought for how to improve the hardware was to invest in the MYO Armband. After finding that the MYO was not a viable solution for what needed to be accomplished (see Chapter 3 Section 1), using the Emotiv headset as a tool for the severely disabled became a primary goal (to see a list of all goals view project plans Appendix B). Another area this group aimed to improve is the overall user entertainment. A major goal was to create not just fun games, but games that a user cannot wait to play again. To do this, the previous game set was modified with the user experience satisfaction as the main goal. Also, a new set of games was added. The new set of games added variety to keep users with different game preferences entertained. One major branch from the previous group s work was developing some 3D games for the new game set. As people with lots of experience with video games, the current group wanted to make a few games that appeal to advanced video game users and felt that 3D games help with achieving this. The final area that needed improvement was the tools for the therapist. Beyond the scores that a patient receives, a therapist had little knowledge about how a patient is progressing. To correct this, the database was expanded to include tracking of other patient statistics including play-time, distance travelled, and other gameplay items. As far as future improvements beyond gameplay statistics, the team hopes to include a system that automatically adjusts difficulty based on how a patient is performing. This will decrease boredom and hopefully create a better custom fit for each patient. The improvements definitely increased the functionality and effectiveness of the system, but there was another hurdle. Bringing the system out of development and into the hands of therapists, or deployment, was loosely implemented but not perfected. Chapter 3, Section 4 outlines this deployment phase more closely. After exploring a number of options, including a number of web hosts, two deployments were chosen in the form of a CSU web server and a SaaS deployment. Some worked, some gained less attention as they were explored more deeply; but currently, there are two promising ways to deliver the same benefits to therapists making the system flexible and adaptable. Flexible, adaptable, robust, fun, and personalized would be adjectives that should be strongly associated with the project if it is successful. In order to meet the requirements of the customers an AGILE software development framework was employed. The scrum framework consisted of Chris Hesser, Corey LeFevre, Michael Rowack, Dr. Malcolm and his graduate students, Dr. Pasricha, Mrs. Notaros, and a number of industry and scholastic professionals. There is no doubt that the project would be floundering without these people; their expertise and opinions are quintessential in advancing GATOR to a professional level of accomplishment. 5

6 Chapter 2 State of GATOR Section 1 History The GATOR project has been being developed for the last four years by the electrical and computer engineering department at Colorado State University as well as the occupational therapy group at Colorado State University. Over these last four years, the project has been supervised by Dr. Sudeep Pasricha and Dr. Matt Malcolm. For the last four years, the goal has remained the same. The goal has, and will continue to be, providing an enjoyable interface for patients to perform their physical rehabilitation. The project is also able to provide a cheaper alternative for physical rehabilitation. It can reduce the amount of time patients spend with their physical therapist, and it increases the quality of care for these patients. The project initially started with rehabilitation in mind, but the interface was very different. The project was able to provide physical rehabilitation by utilizing object tracking with web cameras. A physical object would be viewed by the web camera, and the object motion would be translated into a game where the object was used to play the game.[3] Games like whack a mole and pong were created. This project iteration was very successful in that patients would use objects to control the on screen game environment while using their afflicted limbs; and based on proper physical rehabilitation movements, the game would reward them. This proved to be good for physical rehabilitation; and because of this first iteration, the project would be taken up as a continuation project. With the idea of improving the project continually, the next step was to transition from web cameras to smart mobile devices like tablets and smart phones.[4]this new technology was adopted because of their growing popularity and their ability to be used in so many different ways. Unfortunately, this iteration of the project was not as successful as hoped for. The problem was that the mobile devices were not able to accommodate the resource needs of the game, and the physical interface was not beneficial for physical rehabilitation. However, based on prior success of the project and the ability to help people so much, the project continued. The project found its most success in the third year. The project worked with two different Motion tracking devices: the Leap Motion controller and the Microsoft Kinect. The Kinect was unfortunately hard to utilize; however, the Leap Motion controller was a suitable fit as it met the majority of requirements for physical rehabilitation.[5] It was very successful in tracking the hands of the victims. With this tracking, the students were able to create an interface for physical rehabilitation. They created JavaScript games with HTML5 that rewarded successful physical rehabilitation. The system was used for preliminary testing of physical rehabilitation for patients early this year. This most recent version of the project has been adopted by this year s group, and it is currently being improved upon and is also being used for the occupational therapy department. Section 2 Initial State In the summer of 2014 a new team took over the GATOR project from the previous year s team. The previous team found a successful motion tracking device and developed web based games. A python controlled web based interfaced was used to associate the web based games with an SQL database for patient information and setting. The team was successful since they met their defined requirements. However at takeover the new team had to complete the 6

7 recommendations outlined by the previous group and address their work in order to make it usable. The following figure outlines the previous teams work. Table 1 Previous Teams Work Name Movement Type Description Usability Maze Left and right. Push and pull. Maze is a game featuring a mouse Extremely difficult. Diagonal sweeping. Circular Movements. Pattern based. that the user controls and the goal is to have the mouse move through the obstacle path without colliding. Meteors Left and Right. Meteors has the user controlling a space ship that s primary goal was to dodge the incoming meteors. Good. Pirates Cove Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. Pirates cove is a game featuring a ship that the user controls. The user tries to move the ship around a body of water with the goal of collecting as many gold coins as possible. Water Drops Left and Right. Water Drops has the user holding a virtual cup. With the cup the user tries to collect as many water drops as possible. Whack-a-Mole Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. WAM has the user controlling a mallet and hitting as many moles as possible. Web2py NA The web2py interface was only a testing interface for the group. They did not have it hosted online, so only local deployments were possible. Besides this the deployment was hard coded to only include local paths rather than dynamic paths. Leap Stand NA The Leap Stand was handmade and as a result reproduction was difficult, therefore a new stand was needed. Broken Unable to use what so ever. Extremely difficult The software was usable however it did not meet any patient requirements. Good Little changes were needed to meet patient requirements. Section 3 Previous Groups Recommendations Extremely difficult. Broken. Good. Extremely difficult. Extremely Difficult. The previous group recommended that the new group continue with the web2py framework since their entire codebase was written with the web2py backend. Changing framework would require porting the system or starting from new. Continuing with the AGILE software development method was highly recommended. The previous group iterated that it was imperative to continually develop and change development in order to meet the ever evolving requirements of the occupational therapists. Enhancing and fixing the inherited set of games was also highly recommended. As outlined in table 1 the games inherited were promising however many of them were faulty and did not meet the patient or therapist requirements. The previous group also recommended implementing the ability for therapists to assign therapy routines to their patients. 7

8 The last recommendation was to implement games that were visually stimulating. Adding in more games was also recommended. New games would provide the ability to make the system available to patients in different stages of rehabilitation. They also would give the ability to implement more motions for physical rehabilitation. Chapter 3 Requirements and Objectives Section 1 Patient Requirements A patient that plans on using the system can expect multiple benefits from the system and it is through these benefits that we expect a patient to choose this particular system for rehabilitation. First and foremost the major benefit of using this system is the potential for enjoyable rehabilitation. Enjoyable rehabilitation can be measured through continued use of the augmented reality games. Continued use can be defined as a patient using the system until he or she regains motor skills that are comparable to that which the patient had before they were afflicted by their particular debilitating condition. Continued use can also be defined as indefinite usage occurring multiple times per week for 30 minute to one hour intervals. Besides the benefit of an enjoyable rehabilitation system a patient using the system requires the benefit of an affordable and convenient rehabilitation system. Considering that the average cost of an occupational therapy session can cost anywhere from $50 to $400[11], an affordable system can be defined at roughly $150. This price can be chosen as a baseline since it is on the lower end of the average for a session of occupational therapy. Based on the demographic that the system is intended for ease of use is a major requirement. Ease of Use corresponds to a multitude of factors. The system has to be easy to set up. The hardware cannot be too complex where it does not work if it is set up slightly incorrectly. As well as this anyone should be able to use it in a standard environment. A standard environment consists of access to a standard power source and high speed internet access. This standard environment is based on assisted living centers [12]; this environment is expected to largely consist of the majority of the users of this system. Section 2 Occupational Therapy Requirements The occupational therapists choosing to use the system are looking for multiple benefits as well. The system needs to be able to increase their productivity. This means that the therapist needs to be able to have multiple patients using the system and benefiting from rehabilitation; while this is taking place the therapist does not have to be present. In order for this to be possible the therapist needs to be able to prescribe certain games or therapy routines to their patient while on the fly. They also need to be able to monitor patient progress so that they can diagnose the current rehabilitation and modify it. In order to monitor the progress of the patients the occupational therapists need to have statistics that help them understand how the patient is doing in many different categories. They need to know if range of motion is increasing, if the patient is moving more, if the patient is moving faster, if the patients endurance is increasing, etc... Modification on the fly is also necessary. Therapists need the capability to adjust the difficulty settings, sensitivity settings, patterns, etc to ensure that the patient s rehabilitation is designed properly. 8

9 Section 3 Overview of Project Objectives Utilization of a cheap and effective motion tracking device. Enjoyable games o Visually stimulating o Mini games o Achievements o Power ups o Progression Games featuring many different movements for physical rehabilitation Statistics o Stats for patients to illuminate their progress o Stats for occupational therapists to interpret total progress Cloud based deployment Tools to increase therapist productivity Object oriented code to allow for major modifications (specifically to make a controller change simple) Create a software package to allow for mind controlled smart home automation Chapter 4 Summary of Work Section 1) Hardware A. Initial Hardware and Investigation Hardware selection was made fairly easy because the system was inherited from the previous group. The system was set up with the Leap Motion controller, and this piece of hardware met the incoming requirements due to its capability to track hands, fingers, and recognize gestures. Gestures include making a fist, pointing a finger, as well as many others. Although the capabilities of the Leap Motion controller met requirements, investigation on other pieces of hardware was conducted. The hardware investigated was the Google Glass headset and the MYO wristband. After investigating these devices, it was determined that implementing motion tracking games on Google Glass would be very difficult; the hardware would not be powerful enough to support the necessary motion tracking requirements. After initial investigation, an initial investment was made for the MYO wristband development kit. The MYO wristband appeared like it could be a good solution to provide more coverage as it was touted as being capable of tracking more complex gestures than the Leap Motion controller. The MYO wristband, like the Leap Motion controller, is also very portable. It was able to be charged before usage over a USB charger. It was also wireless and communicated with the PC via Bluetooth. The wristband s major benefit over the Leap controller is that it is able to directly translate muscle activity in the forearm into gestures based on muscle movements. For patients who are unable to perform proper hand movements because of their condition, this would be a great addition to the project. Unfortunately, the MYO wristband did not meet expectations or the necessary hardware requirements. Still in the development phases at Thalmic Labs, the gesture recognition was inflexible and lacked precision. So despite the 9

10 wristband being able to sense gestures, it was too primitive, and it was ultimately decided that waiting for the firmware to improve was not worth enough to integrate the device into the suite of games and tools. Another major reason for straying away from the MYO wristband development kit was because the company did not provide access to raw muscle signals. This was detrimental to for development because it would not be possible to develop custom gestures. In the future, when the MYO wristband has increased functionality and is able to provide developers raw muscle movement data, this option may be reconsidered. B. Leap Motion Controller Currently, it can be said that the Leap Motion controller is working well based off of therapist and patient feedback; meaning that the patients are able to learn how to use the controller quickly and the controller provides a smooth experience for their physical rehabilitation. The Leap controller is quick to set up, it is very portable, it is also unfortunately very easy to set it up incorrectly. Setup consists of rolling out a keyboard mat, mounting the Leap via Velcro onto a stand and plugging it into the computer. The major problem is that if the height of the stand is incorrect gameplay can suffer significantly. One of the requests of the occupational therapy group is that they want patients to be able to take the system home and use it throughout the week. The physical therapists want patients to be able to use the system for about 30 minutes twice a day. The Leap is more than capable for operating for these periods of time successfully. The only problem noticed in this regard occurs over an extended period of time i.e. over six hours the Leap Motion controller can overheat. This is not normally a problem because the therapy sessions span only a couple hours each day. The Leap Motion controller is very successful at capturing patient movement when it is setup correctly. It has a resolution of 10 um.[6] As a result, it is able to accurately capture information like average velocity, max velocity, twitch speeds, gestures, grab strength, etc. These measurements and stats have been very helpful in giving therapists the ability track the patient s rehabilitation and the ability to give patients a tangible idea of how their actual movement is progressing as they are playing games. These abilities of Leap Motion provide the ability to exploit more movements for physical rehabilitation within the games. The biggest downfall of the Leap Motion controller is that it is not meant to be mounted to a stand. The Leap Motion controller is designed to be sitting on top of a desk or similar environment. When it is mounted, the Leap controller attempts to render a hand that is palm side up even though the patients hand is actually palm side down; this can create issues with tracking. Sometimes the Leap alternates between the two settings rapidly and can create an issue where the rendered hand is flipping back and forth between upside down and right side up; this switching results in faulty movements and confusing gameplay. In order to mitigate this issue, recent firmware update were utilized that is adapted for head-mounted functionality. As well as implementing head-mounted functionality in the games, games were designed with the KiwiJS library which allowed for improved tracking with its custom Leap plugin; this will be discussed more in depth in section 2. The head-mounted functionality is not meant for a completely inverted leap controller, but it has improved the hand orientation recognition. On top of that, by placing a flat black matte keyboard mat on the desk area, the infrared sensors are very successful at differentiating the hand from the mat and can reproduce the patient s actual movement with better accuracy. The group this year found a new stand for the Leap Motion controller in the form of a 10

11 folding LED desk light. This new stand is pictured in figure 1. This light met requirements and that is why it was chosen. The light allowed for adjustable height, a small profile, and it was under $30. Using glue to adhere Velcro strips onto the light and the Leap a playing stand was produced that was comparable to the previous stand however it was better since the height could be adjusted ( which is needed for games that use up and down movement). It has been mentioned through the report that the Leap Motion controller can and is very successful when set up properly. Set up properly in theory is something that should be very easy to do, but many problems have been encountered where patients do not always consider the how the Leap is meant to track and how the inverted position affects the usability. As a result, they have occasionally neglected to roll out the playing mat with the system. As well as this, during transfer, the stands have been adjusted; and, as a result, the playing height of the stand is not set correctly. When the stand height is not set correctly and without the playing mat, movement is not rendered correctly. In order to supplement this, a troubleshooting guide was created which includes pictures and documentation on what to do when encountering a certain problem. It was observed during patient testing that when the system performs, Figure 1 Leap Motion Controller Mounted to Stand albeit not perfectly, patients do not always refer to the troubleshooting guide and instead attribute the poor motion tracking to the fact that the system is still in the prototyping phase. One recommendation noted below in appendix C is that the group continuing this project should implement a new technology such as the Intuigine nimble controller. This controller would meet the same requirements as the Leap controller however it would eliminate the need for a stand while also providing superior motion tracking. The group this year was unable to work with this technology as it won t be available until Q C. Patient Experience with Leap When the system has performed as expected the patients have enjoyed using the system. However the patients initially are very unsure of how to use the system. Typically the patients using the system have been older than 60 and haven t been versed with technology. As a result there has been a learning curve; typically the patients aren t capable of playing the games correctly until they have played all of the games at least once. Once the patients get passed the learning curve they haven t had any problems. It seems that all the patients learn how it tracks their hand in less than 20 minutes which is ideal and the majority of patients have tried to challenge themselves once they understand how the tracking works. One patient named Betty would ask if objects in game could be launched in certain areas so that she would be forced to move her hand to the corners of the tracking areas. The biggest problem with tracking is that some patients whose condition severely affects them make a combination of a clenched fist however their wrist will stick up and a finger will 11

12 also stick out. This hand stretches the capabilities of the Leap controller as the controller will lose tracking since it is looking for a regular hand. As a result the occupational therapist occasionally would need to help the patients stretch out their hand; this could potentially present a problem if a patient was using the system at home and had no one to help them. Section 2) JavaScript / HTML 5 Games A. Overview The goal of GATOR is to create fun and entertaining games that patients enjoy playing. Therefore, a major focus was on providing a variety of fun and bug-free games. Being a continuation, work from the previous team was inherited and their work was improved upon. The previous group created five different games written in JavaScript and HTML5 (these games are outlined in table 1). These games provided a good starting library of games. However massive improvements were needed for the inherited games. These improvements are detailed in table 2. The reasoning behind choosing JavaScript and HTML5 is simplicity and portability. A major requirement was creating a system that could run effectively on a variety of machines with little additional requirements. Last year's group describes the choice: In order to meet this requirement the programming language needed to be interpretive. This means that the code does not need to be compiled before it can run; it is simply interpreted at runtime. This cuts down on the need to compile the code numerous times for various different platforms. Given this requirement we narrowed down our programming language options to either Python or JavaScript. Our final decision was made when we considered the requirements of each language. In order for us to use Python we would need to ensure that the end user had a compatible version of Python installed on their system. Also, Python would require us to manually integrate the Leap Motion Application Program Interface (API) and any other development libraries. On the other hand, JavaScript does not require any special installation or dependency integration on the users end. Its only requirement is that the user has an internet connection and a web browser. Given these facts we chose JavaScript as our primary game development language.[5] Besides this, it was concluded from preliminary testing with the Leap SDK that using JavaScript as the primary coding language was beneficial for Leap development. The Leap Motion API is developed in various languages; however, the JavaScript API is very simple to use and easy to develop with. There are only two requirements for using Leap with a JavaScript game. The code needs to include the Leap.js library which includes the functions needed to initialize the controller, interpret data, implement gestures, etc. In addition to this, the Leap Motion control panel software and updated controller firmware is needed on the computer in order to interact with the device. The Leap Motion controller firmware is needed in order for communication to be established properly between the end user and the Leap API. While the Leap is mostly compatible with all browsers that utilize web sockets, Google Chrome provides the best support. While loading over https in other browsers, it needs to run scripts in an unsafe manner which besides being unsafe can also be a hassle. 12

13 B. JavaScript Libraries Using JavaScript as the primary coding language can be a pretty vague statement. While games can be created from pure JavaScript, there is a plethora of different libraries that provide already written functions and features. The previous group chose to use the open source CreateJS library. CreateJS has functions that allow for image creation, manipulation, and animation as well as the ability to add sounds, and music, etc.[7] By default, in the beginning of development and redevelopment, CreateJS was used. However, there were major problems that were associated with using CreateJS. The biggest problem was that CreateJS did not interact well with the Leap controller. CreateJS is resource hungry when preforming draws. The Leap games are constantly animating the object being controlled by the Leap controller. As a result there are many animations that are not rendered which results in poor movement. Using CreateJS resulted in code that was not Figure 2 Current Suite of Games read friendly, and the functions are obfuscated. This resulted in code that does not make much sense when you go back and look at it a couple weeks later. When designing games with animations, and many objects it is typical to program in an object oriented manner. In order to develop in an object oriented style a new library was needed. In order to produce new games that were similar to the fun, addictive style android games, a powerful object oriented library was chosen as the primary development library. The KiwiJS library was chosen because it had the benefits that the CreateJS library had, it is open source, and it features many examples and documentation. As well as this it is a frequently used HTML 5 game engine that is used for the production of mobile games that people are familiar with. Choosing Kiwi provided the ability to work with a cutting edge HTML 5 game engine as well as the ability to create recognizable games for the patients. Kiwi is very fast, and it uses WebGL rendering.[8] The object-oriented approach makes the creation of games much easier as well. So rather than having all your code in one place which is how the CreateJS games were done, the complex code could be built into objects instead. For example, in Fruit Viking there is a weapon which has its own class and functions; and the weapon is then instantiated in the top level class. This approach is considered dynamic object oriented programming and it makes the code easier to understand. 13

14 C. Game Development Using Kiwi.JS the current group was able to create Leap Motion versions of familiar games like Fruit Ninja, Flappy Bird, Doodle Jump, Space Invaders, and Pong. One of the primary goals (to see a list of all goals view project plans Appendix C) was to create more entertaining games than the current set. This is to take nothing away from the previous group as they only had a year and were setting up the environment and beginning the development of the overall system. Since this year s group was expanding on the system, a goal was to take what they had done and make it better by adding to it. By using a more modern engine, games were created that were very similar to their android counter parts (without as many features); but these games were created to still be fun while utilizing movements of physical rehabilitation. Using an engine like Kiwi.JS, familiarity was gained and new game Weeks Game Development with Kiwi vs Other Games (sorted in order of dev date) development eventually became Figure 3 Game Development Time rapid (see figure 3). Quick development gave the ability to provide patients with a large suite of games to choose from. This was important because it gave the patients more choices of games thus they would not get bored by playing the same games over and over; this was a patient requirement. Creating new games also introduced different types of physical rehabilitation made available to the patient. With this being the overall goal, the more physical rehabilitation that the system can provide the better it is. Figure 4 Leap Frog and Fruit Viking Gameplay 14

15 D. Outline of Game Development The following table contains information in the form of movement type, benefits, features, and descriptions on all the games that were developed and modifications if any to the inherited games. As well as that it contains the development time and dates for games as well as the type of engine used. Table 2 Game Development Name Movement Type Benefits Description Improvements (if older game) / Features Alien Invaders Left and Right (dodging hand must keep moving entire time) Break Out Left and Right ( ball tracking) Dolphin Run Fruit Viking Gestures Leap Frog Up and Down Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. Pitch and Yaw. Closed Fist. Full palm. Left and Right (controlled movement must move to exact position) Increased range of motion. Improved hand eye coordination. Controlled movement. Increased range of motion. Critical decision making. Improved hand eye coordination. Triceps Stimulation. Increased endurance. Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement. Increased hand strength. Improved endurance. Hand eye coordination. Increased hand eye coordination. Improves Focus and critical decision making. Alien Invaders was modeled after space invaders. The gameplay is very similar. Break out is brick breaker style of game. The game features many power ups which forces the user to change their strategy as they play. Dolphin Run is similar to floppy bird, the user tries to get the dolphin past as many obstacles as possible. User has to cut as much fruit as possible. User matches their hand with the oncoming gesture. User controls a frog and must guide it up by hopping to different Lilly pads. Bonuses, power ups. Different firing modes. AI fires based on user location to force constant movement. Bonuses and power ups. Also added in progression in the form of multiple levels. First game to implement the up and down movement style. First game to feature a power up. Fruit projectiles can be set to launch into areas where mobility improvements are needed. Features the ability to incorporate any movements requested by occupational therapy. This game was modeled after doodle jump specifically to mirror the addictive style of game play. Development Time (Date week) engine 1 week (March 15 W3) KIWIJS 2 weeks (March 15 W2) KIWIJS 3 weeks (October 14 W1) KIWIJS 2 weeks (October 14 W4) KIWIJS 3 weeks (November W4) Play Canvas 3 weeks (November 14 W1) standard JavaScript 15

16 Pong Maze Meteors Pirates Cove Water Drops Whack-a- Mole Push and Pull. (controlled movement must move to exact position) Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. Left and Right. (controlled movements must dodge meteors, no sweeping) Closing fist. Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. Left and Right. Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based. Increased hand eye coordination. Increased range of motion. Triceps stimulation. Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Increased range of motion. Improved hand eye coordination. Controlled movement. Vision field improvement. Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement. Improved hand eye coordination. Increased range of motion, vision field improvement. Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement. Classic Pong. User has to perform the movements that are required to progress through the mazes. Meteors has the user controlling a space ship that s primary goal was to dodge the incoming meteors. User controls a pirate and collects coins that are dispersed in the ocean based on a reoccurring pattern. Water Drops has the user holding a virtual cup. With the cup the user tries to collect as many water drops as possible. WAM has the user controlling a mallet and hitting as many moles as possible. AI capable of ramping up during game, paddle spin, paddle speed increase. This game was entirely rebuilt to improve tracking performance. New motions were added, more rewarding gameplay was created through scoring and diverse environments. Modifications include: closed fist controls rockets. Ship spin, meteor destruction, implementation of tween objects rather than single animations. This game was entirely rebuilt to improve tracking performance. A mini game was created within the game for the first time. Increased visual stimulation. Fixed tracking problems. Created new animations. Implemented Tweens (replacing single animations) allowed for many water drops at once rather than one at a time. NA 4 weeks (December 14/ January 15) standard JavaScript 3 weeks (October 14 W1) CreateJs NA 1 week (April W1) KIWIJS NA NA E. Feedback The new games at this point in time have received very positive feedback. One man named Jerry enjoyed hunting before his condition. During his rehabilitation his favorite game was alien invaders and it was during this game he would always put forth his best effort. Another patient referred to Leap Frog as the Pilates of her rehabilitation. She enjoyed chasing her high 16

17 score and she appreciated the need for controlled movement since her stroke caused ataxia which resulted in jerky movements. A large majority of the patients would request to play Fruit Viking when the therapists would give them the option to pick a game to play for the last game in their therapy regime. Another positive response was recorded at the E-Days presentation. It is believed that if ones product can hold the attention of a child it is entertaining. During the event many school children demoed the games. The favorite games of the school kids were all the new games. The kids were trying to compete with each other on games such as Alien Invaders, Leap Frog, Pong, and Dolphin Run. Based on preliminary feedback and observations it can be said that the new games have inspired motivation in the rehabilitation which was a major requirement in that it was a requirement of occupational therapy and the patients. Section 3) Rehabilitation Tools A. Mirror Therapy Tool One of the tasks given was to create a tool that embodied the mirror therapy technique that is used in physical therapy. The mirror technique requires that a patient hide their affected hand in a box. A mirror is then positioned so that the patient views their good hand when looking at the location of their hidden hand. The patient is then told to move both hands in the same way; and to the patient, it appears that both of their hands are doing the same action. This technique is used because studies have shown that damage to the central nervous system can cause instances of paresis, neglect, or phantom pain. The mirror creates an illusion of normal movement and sensation on the affected side of the body which can stimulate the sensory processing that is attributed to that limb. While not always successful, this therapy tool is sometimes beneficial, and the therapists recommended that a similar tool should be included on the website. The Leap Motion controller comes with its own visualization tool that served as the visual inspiration for this mirror visualization tool that was being developed. The Leap Motion visualizer is a tool that renders the hand movement as seen by the Leap controller. However, the Leap Motion visualizer does not provide an actual hand model when it renders the movement. The team s goal with the tool is to modify it to have heightened sensitivity; when a patient moves their afflicted hand a tiny bit on the screen, the movement is amplified. The current Leap Motion visualizer only provides the standard sensitivity, but future coordination with the occupational therapy team will help determine the Figure 5 Mirror Therapy Visualization Tool 17

18 proper settings at which to amplify the movement and mimic the traditional mirror therapy illusion properly. To build the tool, open source software was found online from a source at MIT, this software was utilized and modified in order to accomplish what the occupational therapists were looking for. The tool developed uses a rigged hand model and renders the Leap Motion activity. As well as that, the code has been adjusted to heighten the sensitivity to produce the correct illusion that the therapists are looking for. Currently, the mirror visualizer tool is working well; but the biggest problem with it comes back to the problem that the Leap Motion controller is not meant for top down usage. Since it is not meant for top down usage, the tool can only be used when the Leap Motion controller is sitting atop the desk or table or playing environment. The problem with this is the patients typically do not have the arm strength to elevate their hand above the Leap controller for an extended period of time. B. Gestures Another major tool was one of the games that was developed. Although gestures is labeled a game it does not follow the traditional routes that the other games follow. Gesture s was developed specifically for training and regaining ability in the hand. The patient needs to utilize muscles that were afflicted by their condition in order to be successful with the tool. Gestures features four different movements currently. There is an open palm, a closed fist, a palm left open hand, and a palm right open hand. However besides these movements integration of Figure 6 Gestures new movements is very easy. Using an object oriented approach in the play canvas engine made integration of these new movements simple. In order to add in a new movement the hand position can be defined and then the patient needs to match it. The capabilities of this tool make it very powerful based on the requirements of the occupational therapists. This tool eventually can be used to test the response, coordination, and range of motion of patients. C. The Help Page A help page was built to include all the necessary information that is needed to play the games. This information was contained in a readme that documents how the system should be set up, how the system works if you are a therapist or a patient, etc As well as this it contains the download link for the Leap software. A mailer script was created in the python controller as well. This mailer script was created so that anyone who fills out the form can contact the GATOR team directly with comments, concerns, feedback etc In order to stay on top of valuable information the database was modified to store anonymous comments. 18

19 Section 4) Deployment The project was deployed on the web with Web2Py (W2P). This was chosen by the previous group, and it was overall a good choice. Web2Py features a suitable backend for the needs of the project. It features a very simple database for patient, game, and data management (see figure 7). Using Web2Py also makes it very easy to deploy updates and games to the system. The biggest benefit of using Web2Py is that is easy to setup communication between the database and the games which is very important because games need to bet set up correctly based on therapists settings. Adding new settings based on therapist input in a timely manner is also very important. Web2Py does not feature global hosting [9]; it only features local hosting which, if need be, can and was used in the beginning of this project. However, global hosting is needed in the long run so the project was deployed in two different settings. Multiple options were discussed for deployment, and the two settled on were hosting an Apache server as well as hosting on the Python Anywhere server. Google Apps Engine was also investigated, but this proved to be very difficult because it required porting the system design into a new format. Python Anywhere was chosen as the primary deployment method because of the ease to integrate Web2Py onto their server. Integration is made very easy because PA features an online bash terminal, a point and click file management system, and a push button deployment recipe (see figure 8). These things are all very important because updates are constantly being deployed; and depending on the update, the update needs to be available to the client instantly. Currently, free version of Python Any Where is being utilized which is not Figure 8 Python Anywhere Development GUI Figure 7 W2P DB management necessarily the best way; however, at the moment, it is meeting the necessary requirements. The reason an upgrade should eventually be considered is because it gives the designers more CPU time on the terminal as well as more storage space and 19

20 the option to name the URL that your site is accessed on. The CPU time that you are allocated is 100 seconds which adds up pretty fast depending on what you are doing in the terminal. The second development chosen was to host a server on the CSU engineering network. This server provides the option to adjust files and make updates while avoiding the Python Anywhere interface, and provides the ability to take files straight from the repository at BitBucket to the server. This is very convenient as updates can be directly updated to the server as soon as BitBucket is updated, and it is all push button using a git pull. This option is a secondary option; and up to this time, it has been used more for exploratory purposes rather than primary deployment. This is because the actual performance of the server did not meet the requirements and it did not have as many of the preferred features; and deploying on this server has been problematic because, on the CSU engineering network end, there has been resets which have set back work; and there has been bugs that have made server access a hassle. On top of the problems experienced with this deployment, if this deployment is chosen, CSU engineering will require a monthly fee to host this server. It is recommended that the project should maintain the Python Anywhere deployment as it was designed specifically for projects like GATOR; it is easy to maintain, and Python Anywhere features an excellent support community. Section 5) Reports and Statistics for Occupational Therapy A. Progress Report Initially all the therapist and patient had access to was a graph at the end of the game showing them their scores for those particular games. Is the score a good way for the patient to understand if they are making progress? The answer to this question was, no, it definitely is not a good way to track user progress. The reasoning behind this thought is that tracking and/or viewing a graph with recent scores for a particular game greatly varies depending on the game, the difficulty, and it really only shows if a patient is getting better at a game. This might be because they learn the rules or how to manipulate the Leap and is not always directly linked with improved muscle activity. The point is, score tracking is good for the user to know; but it is not the end all be all way to track progress. In order to successfully track progress of a patient, it was necessary to understand what Figure 9 Progress Report 20

21 progress is for someone who has suffered a stroke, TBI, SCI, or suffers from CP. Progress in the long run means increased mobility in the upper limbs. Increased mobility is not something that happens overnight. A major goal was to create a way to show the patients their progress, but do so in a way that is encouraging and realistic. It was determined that in order to track progress of increased mobility, it was necessary to track indicators of mobility. The specific indicators chosen include velocity of the hand, distance traveled during gameplay, and maximum velocity. With these indicators therapists can see if the patient is moving their hand more or maybe if they are moving their hand faster and have that direct link between game and progress. B. Weighted Score As well as this, it was important to give the patient a better way to track progress. In order to do this, the concept of a weighted score was used. The score is divided by the max possible score, and when this was not applicable the score was placed into a bin where it was weighted. This weighted score takes into account score, which essentially measures game success, difficulty, and the distance traveled during the game. The weighted score isn t an end all be all measure of progress however it is a better solution to track successful movement. C. Patient Data Stored in Excel Another tool was given to the occupational therapists in the form of excel spread sheets for particular patients. A therapist in the system can access their patient history by simply clicking download patient history and they will then have access to all the patients data since that patient has been using the system. This tool initially wouldn t have been very useful. The database needed to be rebuilt to include valuable information like distance moved, average velocity, maximum velocity, etc Once it was rebuilt the corresponding information could be presented to the therapist in a meaningful manner. Section 6) Mind Controlled Smart Home Automation A. Overview The primary goal of creating software for a mind controlled smart home was to improve the quality of life for those who suffer from debilitating conditions that prevent them from controlling necessary everyday objects within their home. The vision was to create software that acted as middleware between an electrooculography (EEG) headset and smart devices. The user of the EEG headset could think of turning lights on for instance and then the middleware would process the signal, and based on the thought associated with that signal the middleware would send out a corresponding signal to activate the smart lights. B. Z Wave Smart Devices Throughout the life of the project funds were being raised to accomplish this portion of the project. The fund raising took considerably longer than initial expectations. As a result by the time funds were raised there was less than a month left in time for development. In order to deal with the limited time for development the team decided to work with Z Wave smart devices. Z 21

22 Wave devices are all built by the same manufacturer and as a result interacting with devices in the Z Wave mesh network is quite simple. Through the use of the open Z Wave SDK one can interact with a Z Wave transmitter which can then interact with the Z Wave smart devices. It was because of the simple communication techniques associated with Z Wave that Z Wave smart devices were chosen. Figure 10 Z Wave Smart Devices Z Wave is a wireless communications protocol that is specifically designed for home automation. The home automation is typically controlled through remote control applications. The Z Wave devices communicate via low power RF radio and Z Wave is based off of the IEEE standard. C. Emotiv EPOC + EEG Headset The Emotiv Epoc + headset was chosen as the EEG headset for the smart home automation. The primary reason for choosing this particular headset was because it met the outlined requirements for an EEG headset. The specific requirements for the headset were as follows. The headset needed to be relatively cheap while providing accurate EEG output. Relatively cheap meaning that it provided Figure 11 Emotiv Epoc + EEG Headset competitive performance to other EEG headset while being priced under $1000. The Emotiv headset is able to capture brain activity using 14 EEG channels.[10] The expansion will be a piece of software (specifically an application program interface) that is able to capture the user brain activity recorded by the Emotiv headset and act as an interface between user and smart devices around the home. Something simple like a light switch becomes an impossible tool to use if affected with quadriplegia. By linking brain activity to a smart light, a light becomes a tool again. The Emotiv headset features raw EEG data which allows for the creation of custom thought recognition. As well as this the headset features a very powerful training program. The Emotiv software provides the ability to train a thought and associate it with a command. D. Current Work By combining open-source OpenZWave software with the Emotiv API functionality, the ability to assign a thought to a device and control devices without moving was established. While 22

23 fundraising for the device was a challenge, the linking of the two prebuilt software bases was relatively simple. This being said, a customized interface for users and possibly some machine learning concepts could be very useful for a future team trying to make this system viable for patient testing. The software interface in combination with smart devices makes tasks that once were hard because of the movement required into simple thought assignment; things like controlling the television, locking the doors without having to get up, turning the lights off, and the list goes on are all viable uses. The system will provide people with some of the functionality that they lost as a result of their debilitating condition. Section 7) Ethics, Potential Market, and Marketability A. Ethics Ethically there were two primary concerns. The first and foremost concern was to create a safe database. The point of this was to protect the medical data and information of the patients who are using the system. As well as this, a goal was to ensure that there was no copyright infringement. The new team inherited code for the project that used portions of code from previous work and or examples. With this in mind, if the code is ever released to the public, it is imperative that have the consent of the original designers is given before release. However, in order to avoid this nearly all code developed is original and unique. The code that is not is used from an open source or it is generic; as well as this, the majority of the art has been created to avoid the hassle of finding open source art. B. Potential Market and Marketability This project is able to deliver a cheap and effective form of physical rehabilitation. The physical rehabilitation is in the early stages of becoming a proven method to increase upper limb mobility for victims of stoke, CP, TBI, and SCI. With the number of victims from the previously mentioned diseases exceeding over 20 million, it is hard not to recognize the potentially massive market. Based on feedback received from Occupational Therapy, this appears to be something that these people, who suffer from a loss of mobility in their upper limbs, are very interested in. The system is overall very cheap to implement; and for someone without a computer, Figure 12 Patient Using System the entire system including a compatible computer can cost around $300.Regular physical rehabilitation can cost around $150 to $200 a session. With the system roughly costing two sessions of physical therapy, it can be assumed that many victims would be willing to give it a try. The system can help reduce the amount of time patients need to spend with the therapist, and they can still perform the therapy exercises in a guided 23

24 environment further reducing their total expenditure on physical therapy. Eventually, the system could be used in a subscription environment where the users would have to pay a small $5-$10 fee per month while they are actively using physical rehabilitation. Based on the preliminary results of beta testing, the project should be set for some time next year to get a better understanding of the potential market. Conducting a study with ten occupational therapists each with 3-5 patients who they think are able to benefit from the system would be ideal. If occupational therapists decide to get behind the system, then marketing the system potentially as a kick starter could begin. As well as this, the occupational therapists that choose to support the system could potentially help us in marketing the system by recommending it as a supplemental activity to aid in their physical rehabilitation. Chapter 5 Objectives Analysis This chapter analyzes the objectives that were set out to be accomplished by spring semester Some of the accomplishments were not necessarily planned in the beginning but were created as the needs of the occupational therapy group changed. Ultimately, the surprise changes helped in creating a better product. Besides those, the major objectives corresponding to the project timelines (appendix B) are either completed or were recommended to be discontinued. A primary objective this semester was to create enjoyable games for the patients. This objective was achieved and the results were far more promising than anticipated. This was because many of the patients enjoyed the games and requested the new games. The games were even quite popular among young people who had experience playing modern video games. The creation of these new video games was made possible through the use of the Kiwi JS library. Through the creation of a large suite of games it was possible to keep patient enjoyment at a high level. By creating a more diverse set of games, it was possible to provide more rehabilitation exercises. The diverse set of games also is able to keep the patient entertained for a longer period of time. A recommendation of the previous group was to improve on the inherited games by visually, functionally, and audibly modifying them; and this objective was accomplished in the form of Maze, Water Drops, and Pirates Cove. In addition to this, the inherited games were improved upon by adding progression in some cases and new features in other cases. This made the games more enjoyable. Specifically, in the Maze game, added different levels so that the users can experience different patterns simultaneously. For the meteors game, rockets were added that can be fired by making a fist thus giving the patient the ability to control the in game environment more. Pirates Cove was completely remade in KiwiJS to provide smoother gameplay. A mini game was also added to increase patient engagement. In Water Drops the singular animations were removed and tweens were added to increase the amount of water drops and make the game much more challenging as well as visually stimulating. In the other games, new patterns were added, the controls were smoothed out, and different functionality was added to help increase the physical rehabilitation benefits and make the game a more fulfilling experience. Also another objective was accomplished by using the KiwiJS library. The code was developed for the KiwiJS games in an object oriented style. This allowed for rapid development as outlined above. As well as this through the object oriented design future groups will be able to inherit the project and develop faster which will lead to less turn down time. 24

25 One of the main objectives was to use the Leap Motion controller to gather stats about the patient while they were playing the games. When the project was inherited, the system was only able to provide the patient and the therapist with a progress page showing a graph of the users score and the previous scores for that particular game. The project was greatly expanded in terms of improving therapist productivity. User score is not the best way to keep track of progress (as mentioned in Chapter 3 Section 5). The Leap was utilized to gather stats such as average hand movement and velocity. Now the current statistics help show the patient and the therapist the progress that the patient is making. When the project was inherited it was recommended to investigate a way to deploy the system via the web. This was accomplished by utilizing python anywhere. Through python anywhere the system is accessible to anyone with high speed internet, and the system can be modified by the developers rapidly and therapists can view or change patient information on the fly. Chapter 6 Conclusions, Lessons Learned, and Future Work Based on the current design and the testing that has occurred, it is believed that the GATOR system can eventually be used as major supplement to physical therapy and potentially an alternative to physical therapy. Currently, the system has shown promising results in the form of user enjoyment and progression noted by the occupational therapy group. Currently, the system is in a working state. This means that all the features and games that are built, designed, and implemented into the system are in a state that is capable of providing effective physical rehabilitation. The current design is validated through numerous iteration of self-testing and ready to undergo further testing by the occupational therapy group. Future work needs to focus on expanding the capability and usability of the system. Included in recommendations for continuation are adding control devices, therapist tools, customized data analysis, and more games. One important addition that should be considered is to dynamically recommend games for the patient and inform therapists when little progress is being made. With this addition, the system settings can be modified bi-directionally: the therapist can prescribe areas/muscles the patient needs to work on, but the system can also recommend games/difficulties that will utilize these muscles effectively. The ultimate goal is to automate some less specialized therapist activities. By automating the therapist s role, the time a therapist needs to spend doing monotonous tasks with the patient will decrease thus reducing the cost incurred by the patient. In addition to therapist automation, games should provide more feedback to the patients in an entertaining and enjoyable way in the form of a reward and achievement system. The current reward system will be improved upon in multiple plays. The reward system would provide more than just a message saying "good job" and might include achievements, level progression, and potentially alter game play as users become familiar (or bored) with the controls. Expanding playability to the current set of games is as important as creating new games. Single level games with no progression make for very boring gameplay and are not viable for playtime durations more than a few minutes. By adding multiple levels to all the games that feature change in audio, visuals, challenges, patterns, etc. similar to retro video games that were developed for the original Nintendo Entertainment System. In part, some of this progression was achieved like with Pirate s Cove, but there needs to be more. The games will not likely feature highly advanced gameplay like many modern counterparts, however games can 25

26 still feature the addictive gameplay associated with games built on the Nintendo Entertainment System. In the past, the team considered adding further audio and visual stimulation, but after interaction with patients, complicated or numerous effects can cause mental fatigue depending on a patient s condition. Any additions here must be very precise as to satisfy a large number of patient needs. During gameplay motivation to the patients would be provided as simple noises or effects. A faster rate of progression depending on skill might also be an effective motivator. The game Fruit Viking demonstrates this: when a patient is doing well they are rewarded with a war hammer that allows them to destroy more fruit at once; a future goal is to provide functions like this, at a minimum, within numerous games. Improving upon the current system is needed, but there is also a need to maintain it. Next semester s group should continue to support occupational therapists and their patients. While the current team has tried to ensure a robust system, usability bugs are still existent and will continue to arise as the testing continues. Bugs cause patients to become discouraged with the system, and this is a critical problem to consider as patient satisfaction has great impact on future testing. Maintaining the current standard for feedback is also important in order to can make changes and alterations to the system as if problems arise. As frequency of patient testing increases there should also be a corresponding increase in the feedback received and handled. The final objective is to provide users who are unable to benefit from rehabilitation a major expansion to the current system so as to help them and improve their quality of life. Specifically, this system should expand to aid victims of SCI and TBI as well as any others who have severe upper limb impairment. While this portion of the project was relatively small, only having a month to implement it, it is believed that a smart home model would be an important proof of concept for future assistive technology research. A model of this was successfully built using the Emotiv headset, Z-Stick controller, and two Z-wave devices. Future investigation of the capabilities of the Emotiv headset will be necessary. This might include researching best use techniques, signal analysis of EEG data, and new user testing. Learning to fine tune controls using the Emotiv headset and exploiting the EEG data is a future goal as well. But even before accurate and reliable controls for smart devices are found, the system should receive guidance from a medical group specializing in severe disability. Teaming up with a therapist/patient group will prove effective resources for facilitating useful development. Similar to the current GATOR system, a quick prototyping loop will be needed to prevent testers and therapists from getting discouraged. The goal last semester was to create a system to feature fun, robust games that are effective for physical rehabilitation. Hopefully it can be seen that this goal was not only accomplished, but expanded upon with the inclusion of the Emotiv headset system. With the strong base of development provided, it is expected that future groups create equal if not more ambitious goals. 26

27 REFERENCES [1] Cerebral Palsy International Research Foundation (2014) facts-about-cerebral-palsy[online] 5 December 2014) [2] The Stroke Center at University Hospital, Newark, New Jersey (2014) Stroke Statistics [online] 5 December 2014)] Nash, Wallace, Thornton. (2013)Serious Games for Upper Limb Rehabilitation Fall Report.. Colorado State University [online] (Accessed: 5 December 2014) [3] Poore, Tevfik. Augmented Reality Games for Neurological Rehabilitation. Colorado State University.(2012)[online] (Accessed: 3 December 2014) [4] Campuzano, Olson, Vlahinos. Augmented Reality Games for Upper-Limb Rehabilitation.Colorado State University. (2013)[online] (Accessed: 3 December 2014) [5] Nash, Wallace, Thornton. (2014)Serious Games for Upper Limb Rehabilitation Spring Report.. Colorado State University [online] (Accessed: 3 December 2014) [6] Leap Motion (2014) Leap Motion Documentation [7] CREATEJS (2014) CreateJS documentation [8] KiwiJS (2014) KiwiJS documentationhttp:// [9] Web2Py (2014) Web2Py Documentation [10] Emotiv Systems (2014) EmotivEpoc Documentation [11] Occupational Therapy Cost (2014) Cost Helper Health [12] Brookdale Senior Living Solutions (2013) What role does technology play in assisted living? 27

28 Appendix A List of Acronyms API Application Program Interface CP Cerebral Palsy EEG - Electrooculography LMC Leap Motion Controller PA Python Any Where SCI Spinal Cord Injury SDK- Standard Development Kit TBI Traumatic Brain Injury W2P Web2Py 28

29 Appendix B Project Timelines 9/16 Phase 1 October 3 Provide Sturdy Base for Patient Trials Simple game development o One tangible/playable game delivered based on java script old games o Fruit ninja Website Correction/enhancement o Contact us, feedback, readmes, etc o Codes for therapist registration ( non changing) o Dynamic codes for patient registration to prevent data base floods o New splash pages between graphs and end of game o Setting for therapist to enter affected limbs o Flag in games to ensure correct limb usage(cheater flag) o Specifically used for in home therapy and also for stat keeping and tracking o Start Clock and Report Clock Clinical trials (starts Thursday) o Get feedback and adjust based on feedback o Set a second meeting and or more Basic Leap tracking for therapist statistics o o Distance traveled Max stats(field day type) Game enhancements o o Add better visuals non 3d Add better sounds Phase 2 October 24 Enhance Patient and Therapist Experience Game enhancements (basic) non visual/ non performance related o Add levels to create advancement feeling but base it off current difficulty One tangible/playable game delivered based on java script old games o Fit your hand into the puzzle piece? Meeting with therapy group o Understand success/failures/likes/dislikes/etcc. Create report Discuss new innovations and changes Therapy Program assignment for patients o Therapist bubble selections for areas of need MYO basic Development and stat generation for muscle tracking(tentative on delivery date of the MYO) First set of Torque games Website Correction/enhancement(tentative on first deliverable) o New achievement system in place o Updates for home therapy session delivered to therapist at login Phase 3 November 21 Automate Therapy Experience MYO advanced development o o o Statistics generation Muscle exertion and force Muscles utilized and muscles not being used Meeting with Therapy group o Continue creating a usable product for the therapy team (feedback loop) 29

30 Dynamic therapy scheduling based on statistics and MYO statistics o o Auto difficulty adjustments Generation of new therapy routines based on current progress and stats Goal is to create routines where patient is working everything that therapist wants Advanced leap tracking for therapist statistics o o o Finger twitch speed Wrist twitch speed Etc MYO game development o Make current set of games compatible with MYO worst case make at least 2 to 3 games compatible If it is not possible to integrate games with both controllers, we will try to remake the games with MYO or adjust games to be compatible with MYO Phase 4 December 8(tentative) Final Fall Product Delivery Second Set of Torque games (date may change based on 561 reqs) Cumulative Report Stats (IP through the three phases, however we want it to contain all features by this date) Report on clinical trials, success and failures 30

31 10/21 31

32 11/6 32

33 12/11 33

34 5/4 34