Adaptable C5ISR Instrumentation Mission Command and Network Test Directorate Prepared by Mr. Mark Pauls U.S. Army Electronic Proving Ground (USAEPG) 21 May 2014 U.S. Army Electronic Proving Ground Advanced Electronics & Network Systems Mission Command & Network Test Directorate US Army Electronic Proving Ground 2000 Arizona Street Fort Huachuca AZ 85613
Distributed T&E Instrumentation Challenge C5ISR systems have evolved into very rapid Commercial-off-the-shelf (COTS) technology procurements Increasing use of these new and rapidly evolving technologies presents a challenge to the Test and Evaluation (T&E) community Shrinking procurement cycle forces the T&E cycle to also decrease To adequately verify these systems a robust instrumentation package must be developed and implemented to proved stimulation/simulation, data collection, reduction, and analysis Solution Non-platform specific instrumentation suite of software tools that are embedded and LVC C5ISR system instrumentation modifications and innovative without the lengthy procurement process Instrumentation platform must support a wide array of standard and custom input/output (I/O) functions. Software tools must be reprogrammable to interpret the data streams for C5ISR system under test. Meet constraints of size, weight, and power (SWaP) 2
Shrinking Test Technology Resources Constrained Budgets Shrinking Number of Test Events/Increased Complexity Test Completion & Successful Metrics DOE Driven events to gain the most possible from a single event Aging T&E Work-force Limited Academia T&E Professional Programs 3
Shift Left Embed Instrumentation Early Shift - Left Current T&E Cycle Estimate 70% of the defects could be discovered pre-milestone B Find defects early and avoid the unnecessary schedule and cost over runs Left shift in testing takes care of the ailments as they occur and ensures minimal after effects. Improve Acquisition Outcomes Dr. C. David Brown T&E in Defense Acquisition and Dr. Steve Hutcheson Shift Left 4
Problem Definition Test and Evaluation Legacy Solutions are expensive Do not support the Cradle to Grave Do not standardized performance measurement Do not support lifecycle of Systems Testing Hardware-based solutions Do not leverage Systems Computing Resources Customized to supporting a single system test Increase risk with the employment of new complex systems Cumbersome and require maintenance 5
Operational Instrumentation View Spectrum Monitoring Sim/Stim Data Collection Data Reduction/Analysis 6
The Big Picture Integrated Data Reduction and Analysis Multiple Sources of Data Embedded Agents VM Collectors Seamless Instrumentation Infrastructure Test Control Common DB Meta Data Data collected enterprise-wide must be fused for comprehensive analysis Testing Advantages Legacy Single unique test instrumentation Golden Eye Solution Light weight hardware with installed software Embedded Solutions Software embedded per milestone B Life Cycle Testing Improved Acquisition Systems 7
Instrumentation Approach Visualization Simulation Analysis Collection 8
Solution What is embedded instrumentation? Why should we embed and virtualize instrumentation? 9
Next Generation Tools Machine Learning Embedded Test Tools - Software Development Kits (SDKs) Use of SDK s to develop collection and analysis threads Use of SDK s to validate networks Use of SDK s to ensure compliance Use of SDK s to support a wide array of data standard 10
Application Programming Interface (API) Development of standard API s will ease the work of programming graphical user components Over time the developed API s would form a library that includes specifications for routines, data, structures, object classes and many other variables API technology enables programmers to build hybrid analytics solutions The use of functional API has the potential to transform unstructured data into intelligence 11
Machine Learning API s Selection of the Best Model Statistical Based Models Biased based on model used to develop the API Limits of the model curve not infinite Human Genome Models Un-biased based on repetition of occurrence A lot like the human brain learns as things occur 12
Embedded Stimulation SUT Embedded Stimulator Virtual Machine SUT Embedded Stimulator Virtual Machine Virtual Environment P H Y S I C A L N E T W O R K Virtual Switch SUT Embedded Stimulator Instrumentation SUT Embedded Stimulator IMS Network Traffic Stimulation Single server platform Multiple embedded stimulators Multiple virtual environments Stimulate message/network traffic Between different virtual systems Controlled Environment 13
.STAX..HTTPS. Network Effects Emulation System Network Effects Latency, Packet loss, Packet corruption, Packet duplication, Packet reordering, Bandwidth limitation NE2S provides a Cyber testing capability by simulating a wide range of computer and network-based effects that can be centrally managed and controlled during a training and/or test event NE2S is a TRMC program managed by the Joint Coalition Warfare Office/J7. The program is at the TRL 6 maturity level Tester Web browser SSH with public key authentication NE2S Master Station Distributed Network Access Switch Host Based Effects Generator (Software) CPU Effects Mem Effects Net Effects ` Mission Command Platform ` Mission Command Platform Host Effects Blue Screen, Erratic Mouse, Keyboard Lock, Forced Reboot, Sound Damage, Monitor Disruptions, CPU Effects, Memory Effects Tester Starship / StarGEN 14
Distributed Test Capability EPG-JBLM PEO-C3T & DCGS Fort Dix established network Joint Staff J6 (C4AD) EPG JITC Redstone Test Center 1: Cyber & IA Test Facility 2: Realistic Battlefield Emitters 3: Test Events: NetWarrior, Manpack, Prophet 4: Large Range, controlled airspace, controlled RF spectrum 5: Jamming & other EW. Fort. Hood Operational Test Command & Central Technical Support Facility 15
Leveraging Academia Partnerships for Research Concepts Network Topology / QoS Data Mining DATA ACQUISITION PREPROCESSING FEATURE EXTRACTION FEATURE SELECTION DATA MINING INTERPRETATION & EVALUATION Network packet trace (log) Log ->Flow Transformed data Reduced data Patterns Knowledge Characterize network behavior via statistical analysis Detect significant changes/events by comparing adjacent time windows Identify root cause for changes 16
Challenges Identifying S&T Gaps for Next Generation Embedded Test Tools Verification and Validation of Embedded Test Tools Adapt Policy to allow Co-Use of DT/OT Embedded Tools for Life Cycle Testing Testing Community/PM s agreement to embedded test tools for life cycle testing Embed Cross-Domain solutions 17
Questions? Electronic Proving Ground Phone: (520)538-8043 Mr. Mark Pauls mark.k.pauls.civ@mail.mil 18