Introduction to Mobile Sensing Technology Kleomenis Katevas k.katevas@qmul.ac.uk https://minoskt.github.io
Image by CRCA / CNRS / University of Toulouse
In this talk What is Mobile Sensing? Sensor data, but what can we actually sense? Sensor Fusion. Mobile Sensing on Android and ios. Continuous Sensing Platforms. Mobile Sensing in Research.
NFC WiFi GPS Ambient Light Sensor Barometer Camera(s) Bluetooth Magnetometer Microphone(s) Water Sensor Proximity Sensor Gyroscope Accelerometer Motion Coprocessor
Satellite Navigation Systems It calculates the location (latitude and longitude), altitude, and speed of the device based on the distance from at least four satellites (trilateration). Average accuracy is ~3m. Not suitable for indoor positioning. GPS is the most popular one, but others are available or coming in the future (GLONASS, COMPASS, Galileo, IRNSS).
Wireless networking technology. Can been used as an indoor positioning system (IPS) by triangulating the Received Signal Strength Indication (RSSI) of other Wi-Fi hot-spots. Some mobile devices combine GPS, cell tower triangulation and WiFibased location to improve accuracy. Read Survey of Wireless Indoor Positioning Techniques and Systems by Hui Liu et al.
Wireless networking technology for exchanging data over short distances. (~10 meters for Class 2 radios) Can also be used for indoor positioning. Very useful for spatial-temporal data gathering (participant mobility and social interactions). Bluetooth LE (BLE), marketed as Bluetooth Smart, has very low battery consumption.
ibeacon Apple s implementation of Bluetooth Smart, used as an indoor positioning system. Classifies the location of the device as Immediate, Near or Far. All devices can be ibeacon transmitters, receivers (or both), but the technology is also open to third-party hardware. Also available in Android using thirdparty libraries.
ibeacon Data Format Company ID (Apple: 0x004C) UUID (16 byte hex) Minor (uint 0-36535) Type (always 0x02) Major (uint 0-36535) Measured Power (rssi in 1m distance)
Estimote Indoor Location
Eddystone Google s answer to ibeacon. Fully open-source specification (Apache v2.0) with source code available for Android and ios. Three frames: Eddystone-UID (similar to ibeacon ) Eddystone-URL (broadcasts a URL) Eddystone-TLM (broadcasts telemetry information)
Some terminology Six degrees of freedom (3D Translation & 3D Rotation)
Magnetometer Magnetic field sensor. Tells you the actual orientation of the device relative to the magnetic north (not the true north). Three axis magnetometer: x, y, z (East, North, Up)
Magnetometer Magnetic field sensor. Tells you the actual orientation of the device relative to the magnetic north (not the true north). Three axis magnetometer: x, y, z (East, North, Up)
Accelerometer A sensor that measures the force of acceleration of the device. 3 axis accelerometer: x, y, z (Surge, Heave, Sway) Can also calculate Pitch and Roll (but not Yaw!) using trigonometric calculations. Also used to understand the device orientation, by measuring the acceleration caused by the gravity.
Accelerometer A sensor that measures the force of acceleration of the device. 3 axis accelerometer: x, y, z (Surge, Heave, Sway) Can also calculate Pitch and Roll (but not Yaw!) using trigonometric calculations. Also used to understand the device orientation, by measuring the acceleration caused by the gravity.
Gyroscope 3 axis gyroscope: x, y, z (Pitch, Roll, Yaw) Tells you how much your device is being rotated over time. Less computationally expensive Pitch and Roll. Also provides Yaw.
Demo time
Comparing the three sensors + - Magnetometer No drift problem (depend on magnetic north) Noisy. Hard to calibrate. Not accurate for rotation detection. Tilt compensation. Accelerometer No drift problem (depend on earths gravity) Noisy, especially in high frequencies. Mixes linear acceleration and gravity. Gyroscope Smooth reading even in high frequencies. Great dynamic response. Large errors due to the drift problem.
What is sensor fusion? The use of multiple sensors so that they compensate each other s weaknesses. Users (developers) do not want to know about sensors. What they really want is accurate and responsive 3D translation, 3D rotation and orientation.
Accelerometer and Gyro Fusion (Sachs, 2010)
Accelerometer and Gyro Fusion Integration (Sachs, 2010)
Sensor Fusion (Sachs, 2010) Gyroscope Orientation 3D Rotation Accelerometer Gravity
Sensor Fusion (Sachs, 2010) Integration Gyroscope Orientation 3D Rotation Accelerometer Gravity
Magnetometer and Gyro Fusion (Sachs, 2010)
Sensor Fusion (Sachs, 2010) Gyroscope Orientation 3D Rotation Magnetometer Direction Accelerometer Gravity
Gravity and Linear Acceleration Linear Acceleration = Accelerometer Data - Gravity In most cases, it is very hard to separate Gravity from Linear Acceleration as gravity changes really fast. High pass filters do not work very well.
Sensor Fusion (Sachs, 2010) Gyroscope Orientation 3D Rotation Magnetometer Direction Accelerometer Gravity Linear Acceleration 3D Translation
Sensor Fusion (Sachs, 2010) Gyroscope Orientation 3D Rotation Magnetometer Accelerometer Direction Gravity Double Integration Linear Acceleration 3D Translation
Motion Coprocessors Apple s M7 / M8 / M9 chip. Continues measures motion data from Magnetometer, Accelerometer, Gyroscope and Barometer sensors. Automatically performs activity detection: stationary, walking, running, automotive, unknown. M8 also provides distance and elevation using the Barometer sensor. M9 can also track Pace and Cadence. Apps can query data of the past 7 days when loaded. Does not provide the raw data.
Sensing Motion in ios and Android Fused data of: Raw and Calibrated data of: - Accelerometer - Gyroscope - Magnetometer - User Acceleration - Gravity - Attitude - Rotation rate - Magnetic field
Mobile Sensing on Android Sampling Rate in motion sensors is not consistent. Timestamp is not consistent. Lack of a motion coprocessor, only Pedometers. (this will change very soon ) Android fragmentation problem. Bluetooth Smart (BLE) is only fully supported in Lollipop.
Sampling rate on Android Samsung Galaxy S2 Google Nexus 4 FASTEST 100 Hz 200 Hz GAME 50 Hz 50 Hz UI 15 Hz 30 Hz NORMAL 5 Hz 15 Hz
Mobile Sensing on ios Running a process in the background is (almost) not possible. Access to some sensors is limited (e.g. WiFi, NFC). Restrictions in the App Store.
SensingKit: A Multi-Platform Mobile Sensing Framework Platform Characteristics SensingKit Server Works in Android and ios mobile systems. Captures Motion, Location, Proximity, Environmental data. Power efficient using Bluetooth Smart (4.0). Easily extensible using a modular design. Automated time sync and data processing on the server. Available in open-source under the GNU LGPL v3.0. Sensor Modules Model Manager Data Communication Module Internet Web Services Data Plug-in System Data Extraction For more info, check www.sensingkit.org
SensingKit: Battery Consumption Power (%) 100 90 80 70 60 50 40 iphone 5S Idle Accelerometer (100Hz) Gyroscope (100Hz) Magnetometer (100Hz) Device Motion (100Hz) Location (Best Accuracy) ibeacon Broadcast (1Hz) ibeacon Scan (1Hz) ibeacon Scan + Broadcast (1Hz) Microphone (44100.0 Hz) 30 20 10 0 00 03 06 09 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 Time (hours) Battery consumption of SensingKit running on an iphone 5S.
Detecting Group Formations
141 142 143 144 145 146 147 148 149 } P1 P1 P1 P2 P3 P4 P5 P6 P1 P1 P1 P2 P2 5.72 21.78 29.07 39.95 30.60 P2 P2 P2 P3 P3 5.72 15.45 29.06 19.67 25.41 P3 P3 P3 P4 P4 21.78 15.45 4.60 3.94 1.22 P4 P4 P4 P5 P5 29.07 29.06 4.60 10.49 2.43 P5 P5 P5 P6 P6 39.95 19.67 3.94 10.49 3.04 P6 P6 P6 30.60 25.41 1.22 2.43 3.04 P1 P2 P3 P4 P5 P6 P1 0 0 0 0 0 P2 0 0 0 0 0 P3 0 0 1 1 1 P4 0 0 1 0 1 P5 0 0 1 0 1 P6 0 0 1 1 1 150 time (sec) (a) Weighed Adjacency Matrices (b) Binary Adjacency Matrix (c) Graph
Results Confusion Matrix
Thank you! David Sachs, Sensor Fusion on Android Devices: A Revolution in Motion Processing, Google Tech Talk, February 2010. Retrieved from https://www.youtube.com/watch?v=c7jq7rpwn2k. Kleomenis Katevas, Hamed Haddadi, Laurissa Tokarchuk, Richard G. Clegg, "Detecting Group Formations using ibeacon Technology", 4th International Workshop on Human Activity Sensing Corpus and Application (HASCA2016) in conjunction with UbiComp 2016, September 2016, Heidelberg, Germany. Kleomenis Katevas, Hamed Haddadi and Laurissa Tokarchuk, SensingKit: Evaluating the Sensor Power Consumption in ios devices, 12th International Conference on Intelligent Environments (IE'16), September 2016, London, UK. Kleomenis Katevas, Hamed Haddadi and Laurissa Tokarchuk, Poster: SensingKit -- A Multi-Platform Mobile Sensing Framework for Large-Scale Experiments, Extended abstract, ACM MobiCom 2014, Maui, Hawaii, September 2014.