An Innovative Approach for Elderly Behavioral Analisys by adopting enabling IoT Technologies

Transcription

1 An Innovative Approach for Elderly Behavioral Analisys by adopting enabling IoT Technologies Luigi Patrono, Piercosimo Rametta, Ilaria Sergi Dept. of Innovation Engineering University of Salento Lecce, Italy {luigi.patrono, piercosimo.rametta, Ruben Mulero, Aitor Almeida DeustoTech Deusto Institute of Technology University of Deusto Bilbao, Spain {ruben.mulero, Abstract As the average age of the citizens increases, cities must provide new services for the emerging problem. The City4Age problem aims to provide meaningful interventions to address the problems related to Mild Cognitive Impairment and Frailty in elderly citizens. As part of the City4Age project we have developed a flexible and scalable data capturing and management infrastructure which combines both the Internet of Things and Linked Open Data paradigms. A proof-of-concept validation illustrates how data are collected, managed and computed by the proposed system to make them available for MCI and frailty risk detection algorithms and for third parties. Keywords Internet of Things; Embedded System; Ontology; Big Data; BLE; Sensors; Behavioral Analysis; Cloud; I. INTRODUCTION The increase in life expectancy as a result of the progress of social well-being, is becoming one of the greatest social and economic challenges for European societies in the 21st century. In fact, as reported by the European Commission, by 2025 more than 20% of Europeans will be over 65s, with an increasing number of over 80s 1. Aging citizens, however, are at greater risk of social vulnerability and exclusion due to the onset of physical and cognitive disorders related to age, such as Mild Cognitive Impairments (MCI) and Frailty. This implies significant negative consequences for the quality of life of both elderly people and their formal and informal caregivers. Because older people have different healthcare requirements, health systems will need to adapt so they can provide adequate care and avoid, in the long term, the risk of unsustainability for social-health care systems 2. Therefore, ageing could also become an economic concern for all the citizens in Europe. The definition of innovative solutions to improve the wellbeing of the elderly population (and of their informal caregivers) is one of the major challenges faced by various international research projects [1]. They address the problem by developing frameworks based on Information and Communication Technologies (ICT) to monitor elderly behaviors and to provide corrective interventions after the analysis of detected data The goal of the NACODEAL [2] project is to enhance the quality of life of elderly people by using ICTs. It offers solutions to support elderly people during their daily tasks, while keeping them involved in present society and new technologies. These solutions are driven through Augmented Reality (AR) technologies implemented in a portable device that gives them instructions of how to proceed during different activities of daily living. This device allows elders with memory disorders to continue living an independent life. The ALFA [3] project deals with the application of technologies in supporting people (both in homecare and residential settings) with early stage dementia, by stimulating their cognitive functions and fostering their independent living. By means of three different and integrated technologies, i.e. (i) visual stimulation of mirror neurons in Alzheimer patients, (ii) an interactive agenda or diary and (iii) a movement monitoring system based on wearable sensors, people with dementia will be able to improve or sustain their cognitive functions. The DOREMI [4] project aims to develop a systemic solution for older people, which can prolong their functional and cognitive capacity. The project combines multidisciplinary research in the areas of serious games, social networking, Wireless Sensor Network, activity recognition and contextualization and behavioral pattern analysis. By recording and monitoring information about elderly daily activities, it is possible to track the user performance over long periods, providing early warning of signs of malnutrition, physical and cognitive deterioration. The NITICS [5] project provides a holistic and expandable platform offering a set of basic and task oriented services based on ICT, such as localization of personal objects, localization and movement pattern analysis of elderly and disabled people inside their homes, monitoring with body and environmental sensors, a multimedia bi-directional platform to support daily activities; augmented-reality system to assist users in finding the objects, etc.). The system is used to detect unpredicted or abnormal behavior and to trigger actions in case of need, so care providers can promptly intervene. The BREATHE [6] project, instead, aims to provide a rich platform for improving the quality of life of informal caregivers at all levels. Among its functionalities, it provides an AAL system deployed at elderly s home which is based on a visual monitoring system for recognizing a person, an event, or an action; the integration of video data and information acquired

2 by other sensors can facilitate scene analysis and behavioral pattern recognition. The BREATHE platform provides status reports that informal caregivers can share with doctors at the health system, smart progressive learning support that provides guidance when needed, connection to social networks in order to avoid isolation, and support for the participation of social care professional working behind the scene. The brief overview presented above shows that the hardware technologies involved in the monitoring of elderly people s behavior are heterogeneous, ranging from wireless sensor networks, to wearable devices, portable interactive devices and vision systems. Thus, in this context, there could be the risk of adopting solutions based on expensive sensors and devices, sometimes characterized also by high power consumption. Another issue to consider in this case is the management of a large amount of data. This process involves: (i) the modeling of data handled by the system, with the proper level of abstraction, (ii) the data gathering process from heterogeneous sources, and (iii) the data integration and enrichment phase to give them a semantic meaning shareable among the other system components (e.g. reasoner and/or risk detection modules). To address these and other issues, the City4Age - Elderlyfriendly City services for active and healthy ageing project 3, a research and innovation project funded by the European Commission under the Horizon 2020 Programme, aims to create an innovative framework based on ICT tools and services in order to enhance early detection of risk related to MCI, and to provide personalized intervention helping the elderly population to improve their daily life and promoting positive behavior changes. It also includes six Pilot sites to test the outcomes of the research, which are located in Athens, (GR), Lecce (IT), Birmingham (UK), Madrid (ES), Montpellier (FR) and Singapore. This paper presents the work partially done within the City4Age project and it exploits our experience acquired in the field of Ambient Assisted Living (AAL) [7][8][9]. In particular, it describes the first two layers of the architecture used to collect and manage data related to elderly people and their daily activities, both in indoor and outdoor scenarios. These data are then used by upper layers of the system to recognize behavioral changes in the elderly and eventually trigger proper interventions. The work has dealt with three main aspects. Firstly, the main concepts involved in the data capturing phase have been modeled with a quite high level of abstraction, avoiding that the whole system handles raw data with uncertain interpretations. Secondly, a personal data capturing system for collecting data related to elderly people has been developed [10][11]. It exploits innovative technologies enabling the Internet of Things (IoT) to create an unobtrusive, low-cost and low-power sensing infrastructure that abstracts the heterogeneity of physical devices and communication technologies. The third component is a data management architecture that combines a high-performance REST application programming interface (API) and a Linked Open Data (LOD) API. The REST API allows to easily manage large quantities of data, while the LOD API maps the 3 information in the database to OWL[12], providing semantic meaning to the stored data and making them easier to share. The interaction of these three logical components will be shown through a proof-of-concept example. The paper is structured as follows. In Section II the data modeling adopted in this context is explained. Section III presents the overall system architecture, whereas all implementation details are given in Section IV for the Personal Data Capturing System and in Section V for Personal Data Store and Management System. Finally, in Section VI a proofof-concept explaining how the system works is presented and in Section VII the conclusions are drawn. II. PERSONAL DATA MODELING A. From GEFs to LEAs The top-down approach adopted by the Modelling and requirements for the City4Age risk detection scenario work package (WP) of the City4Age project led to the definition of a set of geriatric factors (GEFs) and geriatric sub-factors (GESs) as quantitative indicators of the MCI/Frailty risk associated to an elderly person. These indicators derive from the most commonly used tools in current geriatrics practice, that measure MCI and frailty based on behavior and human activities monitoring. A partial list of the defined GEFs and GESs is shown in Table I. The numerical values of GEFs and GESs result from the aggregation of data with a lower level of abstraction and a larger basin of sources. Therefore, in order to address issues related to heterogeneous data sources, low level technologies, semantic interpretation and so on, the City4Age project has defined the notion of Low-level Elementary Actions (LEAs). A LEA is the finest grain atomic information used to detect behaviour of elderly people. It relates to start/stop events of user basic actions and contains additional information about time and position of the action that is being taken. All this Table I Excerpt of GEFs and GESs list Geriatric Factors Geriatric Sub-factors Motility Basic activities of daily living Instrumental activities of daily living Socialization Cultural engagement Environment Health Physical Health Cognitive Walking, Climbing stairs, Still/moving, Moving across rooms Bathing and showering, Dressing, Selffeeding, Going out Ability to cook food, Housekeeping, Laundry, Phone usage, New media communication, Shopping, Transportation Paying/receiving visits, Attending senior centers, Attending other social places Visiting cultural or entertainment places, Watching TV, Reading Quality of housing, Quality of neighborhood Falls, Weight, Weakness, Pain, Appetite, Exhaustion, Quality of sleep, Visits to doctors, Visit to health related places Abstraction, Attention, Memory, Mood

3 information is enveloped in the defined Common Data Format (see Section II.B) and sent to the upper layer of the City4Age Platform. LEAs can be grouped in the following macro-categories: Person LEAs: for tracking user states about motility, like standing, moving, walking, etc., but also for collecting data about the usage of smartphone for calling and the number of visits payed or received; Home LEAs: for tracking user movements inside his/her home environment; for collecting data about the usage of home appliances and furniture, like fridge, TV, washing machines, cabinets, etc.; for monitoring ambient parameters, like temperature, humidity, noise, etc.; City LEAs: for tracking user movements, both inside monitored places in the city being part of the City4Age pilot scenarios (shops, offices, pharmacies, etc.) and in outdoor spaces in the city (streets, parks, etc.); for tracking the interaction of user with public transportation systems. LEAs are collected as soon as they happen and sent to the Personal Data Store and Management System for further elaboration, such as the execution of Activity Recognition algorithms. Since a large number of LEAs occurrences can be generated during the day, the City4Age project has introduced the concept of Measure as a daily indicator that synthetizes a set of occurrences of a given LEA. For example, by analyzing all LEAs related to the entering and exiting of the user in a room (such as the bathroom), it can be computed how many times the user went to bathroom in a day and the average time of permanence. These are typical examples of Measures, generated on a daily basis, that make sense from a geriatric point of view to assess changes of behavior relevant for MCI/Frailty. Starting from these and other daily Measures, GEFs and GESs indicators can be computed in order to define a risk profile of each elderly person on a monthly basis. The exact algorithms used to compute GEFs from Measures are out of the scope of this document. Table II LEA's Common Data Format (CDF) Property Description action Name of the action user ID of the user involved in the action. pilot Identifier of the Pilot that uploaded the action location Location where the action is executed. position Geographical position at which the action took place timestamp Date and time at which the action was executed. payload Object containing additional sub-properties, as needed to further describe the action. rating Value defining the uncertainty of the inferred action (1.0: certain, 0.0: unreliable) extra Object containing additional, Pilot specific information that Pilots may want to add. Table III Measure's Common Data Format (CDF) Property Description user ID of the user to which the measures relate. pilot Identifier of the Pilot that uploaded the Measure interval_start Timestamp of the start of the time interval to which the measure relates. interval_end Timestamp of the end of the time interval to which the measure relates. Note: the following property duration can be used in alternative to this one interval_duration Duration of the time interval to which the measure relates. Note: the previous property interval_end can be used in alternative to this one. payload Set of measures values. extra Object containing additional, Pilot specific information that Pilots may want to add. precisely, it deals with the Personal Data Capturing System (PDCS) and the Personal Data Store and Management System B. The Common Data Format (CDF) The concept of Common Data Format has been introduced with the aim to define a data object which can be used within the City4Age Platform to exchange data and information with a uniform and shared meaning, hiding all technological low-level details. In this way, data gathered by different devices, can be treated in the same manner, avoiding concepts misalignment and loss of knowledge. The CDF, along with a well-defined and shared vocabulary of LEAs and Measures labels, provides a first level of abstraction with respect to raw data generated from sensors. There exist two different CDFs, shown Table II and Table III, used to transmit LEAs and Measures to the Personal Data Store and Management System. III. SYSTEM ARCHITECTURE The work presented in this paper is focused on the first two layers of the City4Age Platform, shown in Fig.1. More Fig.1. System Architecture

4 (PDSMS). The main task of the Personal Data Capturing System is to gathers raw data from sensors spread in physical environments (independently of both their specific technologies and communication protocols) and process them to calculate LEAs and Measures to be sent to the Personal Data Store and Management System. The PDCS is internally composed of two main logical blocks. The Local Environment Building Block (LEBB) provides a modular set of software components (generally installed on smartphones or embedded devices acting as gateways), which are able to communicate with different sensing technologies according to the respective standards and protocols in a uniform way. This capability abstracts the heterogeneity of the physical devices and provides a high degree of expandability to include upcoming technologies. The LEBB core logic translates raw data into LEAs and send them, through a well-defined REST APIs, to the Cloud Building Block (CBB). It is in charge to complete the CDF data object if any other information is missing (like the ones related to user profile), since the CBB has access to a wider range of information. Furthermore, the CBB performs other computations in order to calculate Measures based on the given LEAs. Finally, the CBB is in charge of sending both LEAs and Measures to the PDSMS. The Personal Data Store and Management System (PDSMS) integrates the data received from the IoT infrastructure and provides a semantic meaning to that data following the Linked Open Data paradigm. This process also enriches the gathered data applying spatial and temporal knowledge eliciting rules, which improve the semantic knowledge easing consequent inference and querying processes. The PDSMS is composed by two modules. The REST API, which allows to manage large quantities of data in an efficient manner and the Linked Open Data API, which performs the semantization process over the stored data. IV. PERSONAL DATA CAPTURING SYSTEM WP3 The Personal Data Capturing System (Fig.2) is in charge of collecting large quantities of data that can be detected from the surrounding environment through a sensing infrastructure, both in the home and in the city environments. Relevant types of collectable data can be roughly grouped into the following categories: User motility: data related to user body activities, such as motion, rest, sleep, walking, etc. Indoor/Outdoor localization: data involved in the process of determining the position of the user inside a private or public indoor place, such as user s homes or shopping malls, pharmacies, churches, etc., or data related to the position of the user in outside places, like streets, parks, etc. Ambient parameters: data concerning the quality of living condition in indoor and outdoor environments, like temperature, humidity, luminosity, weather conditions, etc. User/Environment interaction: data related to user interaction with the surrounding environments, especially with home appliances (TVs, HVACs, etc.) and public services, for example public transportation. Several technologies can be potentially involved in the process of gathering data for the categories listed above, therefore the first step has been to analyze sensing requirements expressed by the six Pilot sites being part of the City4Age project. Thanks to a continuous interaction with all of them, it emerged that the User motility detection process is based on two main solutions: the use of wearable devices, like BLE wristband (Lecce), smartwatch (Birmingham, Singapore and Montpellier), and the use of the MEMS motion sensors available on the smartphone (Athens and Madrid). In these cases, both public available Apps and APIs or custom Apps can be used. Vision systems, like Kinect, can be used for walking pattern recognition (Montpellier and Singapore). Typical outputs of this module are the BODY_STATE_START/ BODY_STATE_STOP LEAs indicating the timestamp when the user enters and leaves a particular body state (i.e. still, walking, sleeping, etc.). Regarding the Indoor/Outdoor localization, the indoor home monitoring is based on two main solutions, one based on BLE beacons interacting with smartphone or wristband (Lecce and Madrid), and one based on motion and contact sensors (Montpellier and Singapore). BLE beacons are also used to monitor indoor public places in the city (Athens, Birmingham, Lecce, Montpellier and Singapore), being this technique more precise and reliable than the one based on geo-located POIs. POIs definition, in fact, is a technique also used for this purpose, but it does not provide the certainty that the elderly people is actually inside the place of interest. Moreover, the definition of POIs and the interaction with smartphone s GPS receiver, is the most adopted solution to track elderly people position in outdoor environment (all Pilots). All of these types of event can be captured by triggering couples of POI_ENTER and POI_EXIT LEAs, indicating the location type and/or the GPS coordinates. Several and heterogeneous technologies are Fig.2. Personal Data Capturing System (PDCS) architecture

5 involved in the User/Environment interaction detection. For example, the activity of meal preparation can be inferred by using vibration, motion and contact sensors installed on furniture and tools (FURNITURE_OPEN/CLOSED LEAs), like in Montpellier and Singapore; the usage of public transportation means can be detected by beacons at bus stops and Wi-Fi connections within buses (TRANSPORT_ENTER/EXIT LEAs), as in Madrid; the usage of home appliances can be detected by using BLE Smart Plugs (Lecce) or with unobtrusive smart meters (APPLIANCE_ON/OFF LEAs). Finally, Ambient parameters are gathered mainly through wearable sensors (Lecce) or domestic weather stations (AMBIENT_REPORT LEA). By analyzing all Pilots scenarios, two main solutions are used as gateway, for gathering data from the sensing infrastructure and sending them to a local platform for a first stage of elaboration. The first solution uses the smartphone, which interacts with physical devices mainly through BLE connection and relays data by using its cellular data connection. The second solution, adopted by Montpellier and Singapore Pilot sites, uses a wired home gateway, which receives data from the home sensing infrastructure and relays them through a DSL or cellular data connection. Both types of gateways implement the LEBB module inside them. In all cases, instead, the CBB is implemented in a (local or Cloud) server, locally managed by Pilots managers, which directly interact with the PDSMS with the communication protocol described above. V. PERSONAL DATA STORE AND MANAGEMENT SYSTEM The second main element of the City4Age infrastructure is the PDSMS 4 (See Fig 3.), which acts as the central repository, providing a single access point for the stakeholders and semantizing the received information. The PDSMS has three main objectives: (i) to enable its city to interact with the repository using the desired level of abstraction, (ii) to give semantic meaning to the stored information in order to make it more usable for third parties, and (iii) act as a flexible and scalable access point for all the information generated by the City4Age project. The PDSMS is divided in two components: the REST API, which provides the access point for the data gathered by the Data Capturing System and the Linked Open Data API, which allows to enrich the information stored in the repository with semantic meaning and to share it with third parties. The REST API provides a scalable and flexible single access point to the cities integrated in the City4Age system. The PDSMS is able to manage the data from the different cities that take part in the project, integrating the data gathered from thousands of users in those cities. It defines three different modes of data integration: Low-level Elementary actions (LEAs), Activities (which are complex events composed of several LEAs which take place in a period of time) or Measures. As the different cities have different requirements and already deployed software and hardware infrastructure, each city can integrate with the PDSMS using different levels of abstraction. Some of the pilots of the City4Age project already have an Activity Recognition mechanism deployed in their cities and they share the detected activities. Others need the City4Age software infrastructure to help them with the activity recognition and choose to share the LEAs. Once the data is stored in the PDSMS it is processed by the LOD API to align it with a network of domain ontologies. This network of ontologies [13] describes the user data, the geriatric risks, the behavior models and the geriatric interventions. To do that we use D2RQ 5 to transform the data stored in PostgreSQL to OWL/RDF, using a mapping file that describes how this transformation is done. That semantized information is then processed using the Jena Rule Engine 6, making explicit the temporal and spatial relationships in a process similar to the one defined in [15] and [15]. The final ontology is made available to third parties using Fuseki 7, which is a RDF server that provides HTML, REST and SPARL endpoints (an example of the HTML endpoint can be seen in Fig. 4). The access privileges to this endpoint are controlled with Apache Shiro 8, in order to assure the authentication and authorization. VI. PROOF-OF-CONCEPT In this section, a use case of the City4Age Project is presented, in order to provide an example of the kind of data produced and managed by City4Age that can be used by upper architectural layers to perform MCI risk analysis. The first use case is related to the analysis of the activities of daily living (ADL) performed by the user. One of the ADLs Fig. 3. Personal Data Store and Management System (PDSMS) architecture

6 in several cities. Currently the system is being deployed in five European cities (Lecce, Madrid, Montpellier, Athens and Birmingham) and in Singapore. ACKNOWLEDGMENT We acknowledge the contributions of all partners of the City4Age Consortium: many of the ideas and concepts described in this work are due to them. The City4Age project has received funding from European Union s Horizon 2020 research and innovation programme under grant agreement No Fig. 4. Location information in the Linked Open Data API that are monitored in the City4Age project is the housekeeping. A reduction on the housekeeping activity is related to certain MCI and frailty risks. In order to detect these risks and provide meaningful interventions, the two types of LEAs are used, location related LEAs and appliance usage LEAs. To detect the location of the user (ROOM_ENTER and ROOM_EXIT, an example is available in [11]) BLE beacons have been installed in the user house. To detect the appliance usage, a smart-meter is used to analyze the power consumption for inferring which appliances are used. The LEBB captures these sensor data, transforming the raw data into meaningful LEAs. Then the CBB integrates the LEAs using the Common Data Format and sends it to the PDSMS. The REST API of the PDSMS receives the LEAs and stores them in the central data repository. These LEAs are processed by the City4Age algorithms. An activity detection algorithm ascertains the activities that the user is performing and a behavior modelling algorithm evaluates the user behavior comparing it to his previous one. The multivariate risk detection algorithms then analyze if the current behavior is related to any MCI and Frailty risk. If the user behavior is considered as a risk, the intervention module proposes an intervention in order to address it. This information inferred by the City4Age algorithms is also stored in the PDSMS. The Linked Open Data API then maps it to OWL/RDF and the semantic reasoner provides additional semantic information, enriching the dataset. All this information is then shared by the Linked Open Data repository, providing several endpoints for third parties. VII. CONCLUSIONS In this paper, we have presented the data management infrastructure of the City4Age H2020 project. This infrastructure is composed of the first two layers of the general project infrastructure: the Personal Data Capturing System (PDCS) and the Personal Data Store and Management System (PDSMS). The presented infrastructure is flexible enough to be deployed in different scenarios, supporting thousands of users REFERENCES [1] CTS_FOCUSSING_ON_PEOPLE_WITH_NEURODEGENERATIVE_ DISEASES.doc [2] NACODEAL Project, [3] ALFA Project, [4] DOREMI Project, [5] NITICS Project, [6] BREATHE Project, [7] L. Mainetti, L. Patrono, A. Secco, I. Sergi, (2017) An IoT-aware AAL System to Capture Behavioral Changes of Elderly People, Journal of Communications Software and Systems, vol. 13, No. 2, (ISSN: ) [8] L. Mainetti, L. Patrono, A. Secco, and I. Sergi, An IoT-aware AAL system for elderly people, 1st International Multidisciplinary Conference on Computer and Energy Science, SpliTech 2016, Split (Croatia), 13 July [9] L. Mainetti, L. Manco, L. Patrono, A. Secco, I. Sergi, R. Vergallo, R. (2016), An ambient assisted living system for elderly assistance applications, IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, PIMRC, art. no [10] L. Mainetti, L. Patrono and P. Rametta, "Capturing behavioral changes of elderly people through unobtruisive sensing technologies," th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, 2016, pp [11] A. Almeida, G. Azkune, L. Mainetti, V. Mighali, R. Mulero, L. Patrono, P. Rametta, and I. Sergi, An AAL system based on IoT Technologies and Linked Open Data for elderly monitoring in Smart Cities, 2nd International Multidisciplinary Conference on Computer and Energy Science, SpliTech 2017, Split (Croatia), July 2017 (accepted). [12] P. Hitzler, M. Krotzsch, B. Parsia, P. F. Patel-Schneider, and S. Rudolph, Owl 2 web ontology language primer, W3C recommendation, vol. 27, no. 1, p. 123, [13] David Bujan (2017). RISKC4A, INTERC4A, COMBONT and CITY4AGE vocabularies. URL: [14] Almeida, A., & López-de-Ipiña, D. (2012). A distributed reasoning engine ecosystem for semantic context-management in smart environments. Sensors, 12(8), [15] Almeida, A., & López-de-Ipiña, D. (2012). Assessing ambiguity of context data in intelligent environments: Towards a more reliable context managing system. Sensors, 12(4),