PDR Benchmark: View Models and Case Studies

Transcription

1 PDR Benchmark: View Models and Case Studies Takeshi Kurata AIST, Japan University of Tsukuba, Japan

2 Outline Introduction View Models Case Studies Conclusions and Future Works 2

3 PDR Business Launched in Japan This Year Collaboration among NTT Docomo, ZDC, AKM, and AIST. A PDR driven Indoor navigation mobile app was released in Apr, (320 areas including subways and underground shopping arcades all across Japan) Collaboration between MegaChips and AIST. Sensor Hub LSI Frizz was released in Jan, 2015 and MegaChips plans to sell 10 million Frizz products in (Power-aware motion coprocessor optimized for PDR) M. Kourogi and T. Kurata, Personal positioning based on walking locomotion analysis with self-contained sensors and a wearable camera. ISMAR2003, pp , (2003). 3

4 Global Trend on PDR PDR R&D players have rapidly exposed all over the world on and after Acquired by InvenSense Acquired by InvenSense Acquired by Qualcomm Trusted Positioning (Canada) Movea (France) CSR (UK) Acquired by Audience SFO Sensor Platforms (USA) TRX (USA) Indoo.rs (USA) 4

5 Standardization on PDR Benchmarking PDR related R&D is highly active worldwide: Necessity for sharing common measures. Description of the performance should be unified in spec sheets and scientific papers. Different measures from absolute positioning methods such as GNSS, Wi-Fi, and BLE are required for PDR, which is a method of relative positioning. PDR Benchmark Standardization Committee was established in 2014 as a platform of the grassroots activity. 5

6 Support Organizations Asahi Kasei Corporation, INTEC Inc., MTI Ltd., KDDI R&D Laboratories, Inc., KOKUSAI KOGYO CO., LTD., SHIBUYA KOGYO CO., LTD., Koozyt, Inc., SITESENSING, inc., Sugihara Software and Electron Industry Co., Ltd. (SSEI), Information Services International-Dentsu, Ltd. (ISID), Hitachi, Ltd., MULTISOUP CO.,LTD., Milldea, LLC, Murata Manufacturing Co., Ltd., MegaChips Corporation, Recruit Lifestyle Co., Ltd. (Ushida), RICOH COMPANY, LTD., Aichi Institute of Technology (Kaji), Akashi National College of Technology (Arai), Kanagawa Institute of Technology (Tanaka), Keio University (Haruyama, Kohtake, Nakajima), University of Tsukuba (Kurata), Tokyo Institute of Technology (Okada), Nagoya University (Kawaguchi), Niigata University (Makino), Ritsumeikan University (Nishio), National Institute of Advanced Industrial Science and Technology (AIST) (Kurata, Kourogi), HASC: Human Activity Sensing Consortium, LISRA: Location Information Service Research Agency (29 organizations in Japan as of August, 2015) 6

7 Activity Examples in Japan Kickoff meeting of the committee (2014/05/22) 1 st seminar in Location Business Japan (LBJ) (2014/06/11) Organized session in HCG (Human Communication Group) symposium (2014/12/17) 2 nd meeting of the committee (2015/05/19) 2 nd seminar in LBJ (2015/06/13) 7

8 How to standardize PDR benchmark Discussing Datasets and Benchmark indicators with the following aspects: Intrinsic factors: sensor offset, sensitivity (scale factor), etc. Extrinsic factors: Human factors: How to hold/wear the device, How to walk (Gait), Types and amount of actions besides walking, Walking while looking at navigation or not, etc. Environment factors: Geospatial conditions (Route, Shape of walkable area, length without absolute positioning), Environmental conditions (EMF, Atmospheric pressure, Temperature), etc. Initial conditions: Initial orientation configuration, sensor calibration, etc. Whose view?: Evaluation purely for PDR, or for PDR as a part of Hybrid positioning Benchmark Indicators + Datasets 8

9 View Model A view model (in systems engineering, software engineering, and enterprise engineering) is a framework which defines a coherent set of views to be used in the construction of a system architecture, software architecture, or enterprise architecture. Viewpoint: Specification for an individual view View: Representation of a whole system from the perspective of a viewpoint Wikipedia: ISO/IEC/IEEE 42010:2011, Systems and software engineering - Architecture description ISO/IEC , Information technology - Open Distributed Processing - Reference model: Overview 9

10 View Model Enterprise viewpoint, which is concerned with the purpose and behaviors of the system as it relates to the business objective and the business processes of the organization Information viewpoint, which is concerned with the nature of the information handled by the system and constraints on the use and interpretation of that information Computational viewpoint, which is concerned with the functional decomposition of the system into a set of components that exhibit specific behaviors and interact at interfaces Engineering viewpoint, which is concerned with the mechanisms and functions required to support the interactions of the computational components Technology viewpoint, which is concerned with the explicit choice of technologies for the implementation of the system, and particularly for the communications among the components Peter Shames, Joseph Skipper. "Toward a Framework for Modeling Space Systems Architectures, NASA, JPL ISO/IEC :1998 Information technology Open Distributed Processing: Reference Model Part 1: Overview. 10

11 View Model Enterprise viewpoint, which is concerned with the purpose and behaviors of PDR benchmark as it relates to the benchmark objective and the benchmark processes of the organization Information viewpoint, which is concerned with the nature of the information handled by PDR benchmark and constraints on the use and interpretation of that information Computational viewpoint, which is concerned with the functional decomposition of PDR benchmark into a set of components that exhibit specific behaviors and interact at interfaces Engineering viewpoint, which is concerned with the mechanisms and functions required to support the interactions of the computational components Technology viewpoint, which is concerned with the explicit choice of technologies for the implementation of PDR benchmark, and particularly for the communications among the components Peter Shames, Joseph Skipper. "Toward a Framework for Modeling Space Systems Architectures, NASA, JPL ISO/IEC :1998 Information technology Open Distributed Processing: Reference Model Part 1: Overview. 11

12 Matrices of Viewpoints and X Properties/ Features/ Constraints Enterprise viewpoint Precision Accuracy Efficiency Representativeness Completeness Strictness Repeatability Organization, Process on how to get benchmark results Transparency Traceability Information viewpoint Computational viewpoint Engineering viewpoint Technology viewpoint Datasets (Intrinsic factors, Extrinsic factors, Initial conditions) Benchmark indicators (Reliability, Temporality, Variety) 12

13 Matrices of Viewpoints and X Properties/ Features/ Constraints Enterprise viewpoint Whose Information view? viewpoint Precision Accuracy Efficiency Representativeness Organization, Process on how to get benchmark results PDR R&D player Completeness Strictness Repeatability Hybrid Positioning R&D player (PDR-tech user) Transparency Traceability End-User Computational viewpoint Engineering viewpoint Technology viewpoint Datasets (Intrinsic factors, Extrinsic factors, Initial conditions) Benchmark indicators (Reliability, Temporality, Variety) 13

14 Case Studies: Technology Mapping Geospatial Techs (Positioning, Navigation) Social Cyber Physical (Ubiquitous, Wearable, IoT) PDR Benchmark MAR Tracking Benchmark MAR: Mixed and Augmented Reality 14

15 TrakMark WG The TrakMark working group (WG) was established in 2009 as a subsidiary organization of the Special Interest Group on Mixed Reality (SIG-MR), the Virtual Reality Society of Japan (VRSJ) to standardize benchmark schemes that permit objective evaluation and comparison of diverse registration and tracking methods. 15

16 16 Universities, Institutes, Companies (R & D on Tracking) Tracking method development Management of Web, ML, etc. Datasets TrakMark WG Dataset consolidation Aggregation of tracking methods Benchmarking Promotion of researches or products based on the benchmark Universities, Institutes, Companies (Tracking-tech users) Benchmark survey Image sequence 3D Model Benchmark results Ground truth Property Distribution of Datasets, Usage Log analysis Aggregation of benchmark results and its publication Investigation of Benchmark indicators Official benchmarking International WS organization Aggregation of feedback Distribution of Datasets Dissemination Tracking-method choice Development of systems and products using the tracking methods Towards international standardization De-facto standardization Certification (logo mark based on benchmark results) Proposal for ISO (ISO/IEC NP 18520)

17 ISO/IEC NP 18520: Scope of Project Contents (1) Dataset Image sequence Ground truth of camera intrinsic/extrinsic parameters Option Ground truth of image feature correspondences Object/Environment 3D model data (Shape, Appearance, Film Studio Lighting, Motion) Package Depth, Sensor data, NAIST Campus Package Additional objects, etc. Conference Venue Package 17

18 Scope of Project in NP: (1) Dataset Metadata (property for dataset mapping and ease of retrieval) Scenario (Indoor/Outdoor navigation, Tabletop, Industrial) Motion Type (Translation, Rotation, Walking, Handheld, Vehicle) Image Quality (Resolution, Defocusing, Motion blur) Format Image: JPEG, PNG, or Raw (Recommended: Lossless format) 3D model: X3D, Collada, etc. Ground truth and Metadata: XML, X3D, JSON, etc. 18

19 Scope of Project in NP: (1) Dataset Metadata (property for dataset mapping and ease of retrieval) Scenario (Indoor/Outdoor navigation, Tabletop, Industrial) Motion Type (Translation, Rotation, Walking, Handheld, Vehicle) Image Quality (Resolution, Defocusing, Motion blur) 19

20 Scope of Project in NP: (2) Benchmark Indicators Reliability indicators Projection error of virtual objects (points) Position and posture errors Reprojection error of image features Temporal indicators Latency Frame rate (Frequency, Computational cost (time)) Variety indicators Number of datasets used for benchmarking Variety of properties in datasets used for benchmarking To prevent fine tuning for some specific datasets 20

21 Scope of Project in NP: (3) Process Process to make benchmark results How to get benchmark indicators How to choose dataset (number, property) Process to share benchmark results 21

22 Metaio GmbH (acquired by Apple Inc. in 2015) Metaio datasets for template-based tracking algorithms Metaio dataset for sensor-aided visual handheld camera localization Metaio had also provided benchmarking service for template-based tracking algorithms with the corresponding datasets. S. Lieberknecht, S. BenHimane, P.G. Meier, and N. Navab, A dataset and evaluation methodology for template-based tracking algorithms. ISMAR2009, pp , D. Kurz, P.G. Meier, A. Plopski, and G. Klinker, An Outdoor Ground Truth Evaluation Dataset for Sensor-Aided Visual Handheld Camera Localization. ISMAR2013, pp ,

23 Metaio: Benchmarking Inertial Sensor- Aided Localization and Tracking Methods Degradation model for synthesized gravity Noise (Gaussian distribution) Delay between image capture and acceleration measurement No temperature hysteresis Kurz, D., Lieberknecht, S., and BenHimane S., Benchmarking Inertial Sensor-Aided Localization and Tracking Methods, TrakMark

24 HITLAB NZ HITLAB NZ have published the source code of the Mobile Augmented Reality Visualisation and Evaluation Toolkit and data sets to support reproducible research. Examples in HITLAB NZ Datasets (Left) and example result on comparison between sensor-based tracking and sensor-fusion-based tracking (Right). S. Williams, R. Green, and M. Billinghurst, Transform Flow: A Mobile Augmented Reality Visualisation and Evaluation Toolkit, In Proc. IVCNZ

25 Tracking competition in ISMAR 2015 ISMAR 2015 is held from Sep 29 to Oct 3 in Fukuoka, Japan. PRMU (SIG on Pattern Recognition and Media Understanding) algorithm contest committee and ISMAR 2015 Tracking Competition committee including TrakMark WG are collaborating for organizing tracking competition in ISMAR Off-site and on-site competitions are prepared (Off-site competition would be the first attempt for ISMARs). TrakMark WG and ISO/IEC NP mutually get feedbacks though this event. Previous tracking competitions: 25

26 PDR Benchmark Case Example: Mega Chips Corp. Route How to hold At Front/Back As Portrait/Landscape 4 In hand/shoulder bag 5 calling 26

27 27

28 28

29 29

30 PDR Benchmark Case Example: AIST (2014) Experimental conditions: Four subjects (male: 2, female: 2) Two routes (Outdoor: 350 m, Indoor: 242 m) How to hold: changed in the same manner Hold in a hand (watching), In a chest/trouser pocket, In a handbag held by an arm, and put to his/her waist. Noise: Synthesized noise added. AIST-PDR: the following 2x2 settings Moving direction estimation: On Moving direction estimation: Off EMF: On PDR-1 PDR-3 EMF: Off PDR-2 PDR-4 M. Kourogi and T. Kurata, A method of benchmarking on pedestrian dead reckoning and its evaluation. HCG Symposium 2014 in Japan. 30

31 PDR Benchmark Indicator In this case, the longer, the more important. Error (m) 2.6% Weight 2.3% 2.7% Weight Function 2.63% Walking Distance Walking Distance 31

32 Evaluation Results PDR-1 [%] PDR-2 [%] Indoor route PDR-3 [%] PDR-4 [%] [rad/s] [rad/s] [rad/s] [rad/s] Outdoor route [%] [%] [%] [%] [rad/s] [rad/s] [rad/s] [rad/s] Moving direction estimation: On Moving direction estimation: Off EMF :On PDR-1 PDR-3 EMF :Off PDR-2 PDR-4 32

33 Conclusions and Future Works View Models and Case studies were introduced. No like/dislike, Boost up!! Various factors in PDR benchmark should be projected into matrices of views and {properties/features/constraints/stakeholders} PDR Benchmark Existing methodologies X PDR-specific factors cf. In TrakMark: Projection error of virtual objects Repeatability: Temperature Hysteresis, MF, etc. Representativeness: How to hold, Route shape, etc. 33

34 FYI SIMNET (IEEE Std.): not benchmark but DR description in ISO IEEE Std : Annex B: Dead reckoning definitions and algorithms in IEEE Standard for Distributed Interactive Simulation - Application Protocols 34