Collaborative Effects of Distributed Multimodal Sensor Fusion for First Responder Navigation Jim Kaba, Shunguang Wu, Siun-Chuon Mau, Tao Zhao Sarnoff Corporation Briefed By: Jim Kaba (69) 734-2246 jkaba@sarnoff.com
Overview Sensor-independent distributed multimodal sensor fusion framework for improving navigation accuracy Theoretical & analytical models and simulations of how performance scales versus Different algorithms Different sensor error models Varying numbers of users Practical navigation algorithm and system design Excellent implementation characteristics Capable of achieving predicted error reduction effects Collaborative Error Reduction Effects -- Exploit the collaborative nature of team-oriented operations Teamwork Effect Enables firefighters operating in groups to achieve better navigation accuracy than when operating individually Anchor Effect Flexible use of minimal numbers of Deployable Anchor Nodes as RF navigation reference beacons Reset Effect Immediate opportunistic accuracy gains under common conditions 2/2 28 Sarnoff Corporation
Motivation: GPS-denied Navigation GPS is not always available Weak signal, corrupted signal, no signal Urban Canyons, Indoors, Underground Jammed and/or spoofed Significant challenges or complications Cluttered RF & magnetic environment Size, Weight and Power (SWaP) and Cost issues Mobility Pragmatic operational use and deployment constraints for certain applications Hostile operating environment compounded by mission constraints 3/2 28 Sarnoff Corporation
Target Application: Locatus System Design Goals Dismounted Warfighter Navigation in a fully GPS-denied Environment Provide absolute & relative 3-D position to ground forces for: Individual warfighter navigation Team coordination with neighbors (fireteam, squad, platoon) Position status updates to command / headquarters Support Military Operations on Urban Terrain Outdoor urban areas, building interiors, subterranean environments and open areas Maintain strict position accuracy 2m SEP after traveling 2km over 8hrs =>.12 % system error while GPS-denied Man portable; size, weight and power restrictions < 4cm 3 & < 1kg per unit, battery powered Associated pragmatic operational constraints Strong match to Firefighter requirements 4/2 28 Sarnoff Corporation
Locatus System Overview a multimodal position/navigation system for GPS-denied environments* Primary Subsystem Components Inertial Navigation System 3-axis accelerometer, rate sensors, magnetometers, pressure altimeter Inter-node Distance Measurement Inter-node Data Communications Network RF Ranging, MEMS Inertial Navigation & Ad Hoc Network Relay Functionality RF Ranging & Ad Hoc Network Relay Functionality *Illustrated as a man-portable position/navigation system for GPS-denied urban operations. System approach and algorithms can be applied to heterogeneous combinations of aircraft, vehicles, UAVs, UGVs, munitions, etc. Ad Hoc Network Relay Functionality INS provides baseline displacement estimates Inter-node ranges bound the growth of inertial position error over time /2 28 Sarnoff Corporation
Each firefighter wears a Mobile Locator Node Inertial Navigation System + RF Ranging & Comms Concept of Operation Deployable Anchor Node(s) positioned around the periphery of the operational area (optional, mission-permitting) RF Ranging + Comms Provides a framework of RF beacons at known locations Firefighters move into the GPS-denied operational area INS tracks 3-D location (with some error) RF Ranging Radios track inter-node (Anchors and Firefighters) distances (with some error) Locatus system fuses inertial and RF position estimates, appropriately weighted for operational & error characteristics Firefighters move deeper into the GPS-denied operational area Warfighters deploy additional Anchor Nodes as needed to maintain RF measurement connectivity with external framework Throughout the incident response 3-D position estimate is maintained with bounded (but growing) error as the incident progresses (with time, distance, constrained operational areas) Situational Awareness (navigation capability and accountability) is provided throughout incident for firefighters and command structure 6/2 28 Sarnoff Corporation
Sarnoff s Technical Approach Continuous, real-time fusion of error-prone position estimates from multiple technologies to achieve accurate absolute & relative location, navigation & mapping capabilities Distributed fusion of multiple position / location / navigation modalities Multiple sensors on each warfighter/platform and across multiple warfighters/platforms Exploit the complementary nature of sensor s operational characteristics and errors Absolute vs. relative, localized vs. distributed, range unlimited vs. tethered, long-term drift vs. short-term random error Use dynamic and continuous cross-modal feedback to bound system errors Detect & minimize errors, constrain measurements Achieve graceful (vs. catastrophic) degradation of system performance over time & distance as dictated by the mission Use dynamic Bayesian (belief) network to build a graphical model of inferred node location Intractable joint probabilistic distribution of all node locations & uncertainties is factored into a combination of simpler local distributions 7/2 28 Sarnoff Corporation
Multimodal Fusion Algorithm Overview Family of Inference Methods Issues: Performance Computation Overhead Communications Overhead Scalability Reliability Fault-tolerance distributed centralized Arrows connote added assumptions/constraints I: iterative E: extended K: Kalman F: filter S: smoother N: non-parametric BP: belief propagation PF: particle filter I online distributed optimization Gaussian Approx. neighbor info I distributed EKF Single iteration online joint optimization Gaussian centralized IEKF Single iteration Approx. neighbor info Approx. neighbor info centralized EKF Gaussian+ linearization Markovian Markovian online BP non-parametric Markovian N-frame joint optimization Gaussian centralized IEKS Single iteration centralized EKS batch BP non-parametric NBP distributed EKF NBP-PF online batch 8/2 28 Sarnoff Corporation
iekf Fusion Algorithm Data Flow Formulated as a rapidly converging, iterative, message-passing algorithm Nodes individually estimate their position & certainty using localized sensors (INS, altimeter) Nodes exchange their estimates with neighbors Nodes refine their estimate using neighbor s estimates and inter-node range constraints Nodes repeat the exchange/refine cycle multiple times 9/2 Comparable performance to centralized/optimization algorithms 28 Sarnoff Corporation
Multimodal Fusion: Implementation Benefits Provides accurate Absolute and Relative Positioning Using real-world sensors under real-world conditions Sensor flexibility Robust to varying inertial sensor noise/performance models Easily extensible to include additional individual and distributed sensors Handles asynchronous sensor inputs Tolerates inaccuracies of inter-node range constraints Achieves low overall position error even with low-accuracy ranging Accurate positioning even as RF range measurement accuracy fluctuates due to obstructions and multipath interference in urban environment Tolerates variations in inter-node ranging interval Minimal effect on the absolute system accuracy, minor (short term) effect on relative accuracy Superior implementation characteristics: Algorithm is naturally fully distributed Scalable, robust to dynamic changes in # of users Iterative algorithm rapidly converges in steady state operation Computation and communication are inherently localized to nearest-neighbors Communication load is well within the Kbps constraints of tactical military radios /2 28 Sarnoff Corporation
The Teamwork Effect The Locatus Teamwork Effect enables warfighters operating in groups to achieve significantly better navigation accuracy than when operating individually Opportunistic Peer-to-Peer Ranging Constrains INS Drift Range estimate between two warfighters serves as a Wireless Tether between them and bounds their otherwise independent drifts Using multiple inter-asset range estimates constrain INS drift further Teamwork Effect holds as team size varies Single pair to large groups i.e. Position accuracy improves by a factor n for an n-node group General performance prediction guideline for distributed multimodal fusion 11/2 28 Sarnoff Corporation
Absolute Position Accuracy 2 2 1 INU-only (n=2) INU olny dis EKF dis OPT cent EKF cent OPT - 2 (n=) INS Stds: σ dx =.2(m), σ dy =.2(m), σ dz =.1(m), MSSI std σ d = 1m 2 2 1 (n=4) 2 (n=12) 2 2 1 (n=6) 2 (n=14) 2 2 1 (n=8) 2 (n=16) 2m SEP Error vs. Time for varying team sizes Tight grouping of performance results for four fusion algorithms Centralized vs. Distributed; 2 1 2 1 2 1 2 1 Optimization vs. EKF Substantial (2-3 X) performance increase even with a small size network Systematic error reduction with increasing network size 12/2 28 Sarnoff Corporation
The Teamwork Effect: Simulation Validation RMS Err (m) RMS Err (m) 3 2 1 8 6 4 2 cent ekf: rms err cent ekf: fitted line dist ekf: rms err dist ekf: fitted line frame # =.2.4.6 frame # = 4.2.4.6 6 frame # = 14 4 2.2.4.6 12 frame # = 8 6 4.2.4.6 INU: σ dx =.2m, σ dy =.2m, σ dz =.1m, MSSI: σ d =1m 8 frame # = 23 6 4 2 8 6 4.2.4.6 frame # = 9.2.4.6 8 frame # = 32 6 4 2 6 4.2.4.6 12 frame # = 68 8.2.4.6 Error vs. Network Size at 1-minute intervals RMS Err (m) 1 frame # = 77.2.4.6 1 frame # = 86.2.4.6 14 12 frame # = 9 8 6 4.2.4.6 1 frame # = 4.2.4.6 RMS Err (m) 1 frame # = 113 1 frame # = 122.2.4.6.2.4.6 1/sqrt(n) 1/sqrt(n) 1 frame # = 13 frame # = 14 1.2.4.6.2.4.6 1/sqrt(n) 1/sqrt(n) 13/2 Validation of the Teamwork Effect error scaling law 1/sqrt(n) reduction in system error with a team of n collaborating nodes Robust to variations in multimodal fusion algorithm Robust to variations of inertial sensor noise model (not shown) 28 Sarnoff Corporation
Relative Position Accuracy 3m SEP INU-only (ZUPT sqrt(d) inertial drift): Relative position error exceeds 3m SEP after 2 km & 4 hrs GPS-denied m SEP 1m SEP Multimodal Fusion: Relative position error bounded to a constant level < 1m SEP after 2 km & 4 hrs GPS-denied Even with high-error INS exceeding 4m a relative error ~2-3m SEP achieved (not shown) 14/2 28 Sarnoff Corporation
The Anchor Effect Deployable Anchor Node Reference beacon deployed at fixed location Zero INS drift error: position estimate (and error) remains constant Anchor point for mobile nodes whose position estimates degrade with time/distance Deployed opportunistically (pre- or during mission) as stationary wireless tethers and communication relay nodes Self-calibration of deployed nodes based on best location estimate available at the time of deployment The use of even a single Deployable Anchor Node can increase system accuracy by a factor of 2 to 3 The use of two Deployable Anchor Nodes can bound absolute system error to <1m SEP Contrast with classical Time Difference of Arrival (multilateration) and Time of Arrival (trilateration) approaches that require at least 4 constraining measurements 1/2 28 Sarnoff Corporation
The Anchor Effect: Simulation Validation 2 2 1 (n=2) inu only dist w/o anchor dist w 1 anchor dist w 2 anchors 2 2 1 (n=4) nmcrs=, INU: σ dx =.2(m), σ dy =.2(m), σ dz =.1(m), MSSI: σ D =1(m) 2 2 1 (n=6) 2 2 1 (n=8) Absolute Position Accuracy Shown 2 (n=) 2 (n=12) 2 (n=14) 2 (n=16) 2 1 2 1 2 1 2 1 Unaided (ins-only) Operation RF-aided Operation 1 Anchor 2 Anchors 1 Anchor 2-3X performance improvement 2 Anchors Constant, low level error 1-2m SEP 16/2 28 Sarnoff Corporation
The Reset Effect Locatus continual refinement of position estimates can result in estimates that improve, rather than degrade, over time The inaccurate position estimates of Mobile Locator Nodes are reset to a lower level in the middle of a mission even after those errors have grown Provides immediate, rather than gradual, improvement of the estimate accuracy Reset Effect conditions: a) A Mobile Locator Node whose position error has grown establishes contact with a stationary Deployable Anchor Node. The mobile node s position estimate, and its confidence in accuracy of that estimate, will improve b) A Mobile Locator Node whose position error has grown establishes contact with another Mobile Locator Node with a better location certainty c) An individual or a small group of Mobile Locator Nodes merges with another group of Mobile Locator Nodes, forming a larger group allowing for collectively better error performance (e.g. due to the Teamwork Effect ) The Reset Effect is demonstrable for Deployable Anchor Nodes and large groups of Mobile Locator Nodes The Reset Effect enables teams of warfighters that split into subgroups and merge some time later to regain the same level of positioning performance as if they had not split 17/2 28 Sarnoff Corporation
The Reset Effect: Simulation Validation Simulation results support the Locatus Reset Effect by which system errors can actually decrease over time 2 2 1 (n=2) inu only dist w/o anchor dist w 1 anchor dist w 2 anchors 2 2 1 Anchor(s) deployed at f=3, INU: σ dx =.2m, σ dy =.2m, σ dz =.1m, MSSI: σ d =1m (n=4) 2 2 1 (n=6) 2 2 1 (n=8) Example: Encounter with Anchor Node(s) 2 2 1 (n=) 2 2 1 (n=12) 2 2 1 (n=14) 2 2 1 (n=16) Opportunistic encounter with 1 or 2 anchor nodes resets error to a lower level Unaided Operation RF-aided Operation Contact with Anchor Error Reset 1 and 2 Anchors 18/2 28 Sarnoff Corporation
The Reset Effect: Simulation Validation Simulation results support the Locatus Reset Effect by which system errors can actually decrease over time 8-node team splits teams operate independently Two 4-node teams merge nmcrs=, INS Stds: σ dx =.2(m), σ dy =.2(m), σ dz =.1(m), MSSI std: σ d =1 (m) Example: Team Split/Merge Absolute Error 1 dynamical nodes n=8 n=4 Error Reset n=8 n=4 n=8 2 4 6 8 12 14 Relative Error relative RMS err (m) 1 1 1 2 4 6 8 12 14 2 4 6 8 12 14 19/2 2 4 6 8 12 14 Affects not just error rate (curve slope), but absolute level as well! 28 Sarnoff Corporation
Summary System Design & Multimodal Sensor Fusion Framework Sensor independent Excellent implementation characteristics Exploits the collaborative nature of team-oriented operations Expected performance trends & behaviors Teamwork Effect Firefighter position accuracy can improve through collaboration with other mobile nodes without additional infrastructure requirements Anchor Effect When reference beacons can be used, they can be used In minimal numbers With flexibility, deployed at any time Reset Effect Position accuracy need not be constantly degrading Improvements over time can be expected 2/2 28 Sarnoff Corporation
End of Brief Jim Kaba (69) 734-2246 jkaba@sarnoff.com