Prototyping Vision-Based Classifiers in Constrained Environments

Transcription

1 Prototyping Vision-Based Classifiers in Constrained Environments Ted Hromadka 1 and Cameron Hunt 2 1, 2 SOFWERX (DEFENSEWERX, Inc.) Presented at GTC 2018

2 Company Overview SM UNCLASSIFIED 2 Capabilities image processing / computer vision applications for US government customers Number of Employees around 700, most with MS/PhD Main locations Chantilly, VA Dayton, OH Carlsbad, CA Kihei, HI Seattle Colorado Springs So. CA Area LA El Segundo List cities? Ann Arbor New England Valley Forge Dayton IAI Office Location Denver St. Louis DC Area IAI Work Location IAI Future Work Location Albuquerque Las Cruces Carlsbad San Diego IAI HQ, Chantilly Ft. Belvoir Charlottesville (DC Area) Dahlgren Conference Center Conference Drive Chantilly, Center Drive, VA Suite 100, Chantilly, (703) VA PAX River

3 SOFWERX SOFWERX performs collaboration, ideation and facilitation with the best minds of Industry, Academia and Government. SOFWERX can also conduct rapid prototyping and rapid proof of concepts from ideation discovery. Run by DEFENSEWERX (formerly the Doolittle Institute) Located in Tampa, FL

4 Background requirement to track usage of tank ammunition Commanders asked for an automated means of tracking and reporting the firing of the Abrams main gun Location Timestamp Type of ammunition used Various other means of tracking the ammunition unacceptable due to wear & tear, etc. Computer vision solution

5 Context Loader (1) pulls 120mm round from cabinet (5) and loads it into main breech (3) Source: unattributed on multiple websites, appears to be scanned pages from a book

6 Concept Vision-based classifier Camera Processor GPS and SATCOM links No impact on tank s systems Mounted somewhere inside cabin

7 Collecting Training Data Raspberry Pi 2B (900 MHz) 1 GB RAM RPi camera board v2 8 MP = 3280x2464 5V USB battery pack (12 hours) Python script to take and write images to SD card as quickly as possible (~1 Hz) Source: adafruit.com

8 Collecting Data - RPi

9 Collecting Data static photos Compact Nikon digital camera Resolution 4610 x 3460 Slightly over 1000 photos per class Wide range of background scenes

10 Collecting Data Day 2: added GoPro to tank commander s GPS extension eyepiece HD video can be matched to RPi quality in post-processing

11 Early network Initial comparison runs of Caffe and TensorFlow on stock GoogLeNet (Inception v1) Caffe trained using DIGITS software; TF trained using python Remainder of this talk will only discuss TF Initially treated as Image Classification 4 classes No need to label bounding boxes Runs faster than object detection We never more than one object in scene Trained on a DevBox-1 (4x TITAN X)

12 Why use old version of GoogLeNet? Network MAC (million) Parameters (million) Inception v Inception v (?) Inception v VGG ResNet AlexNet

13 PREDICTED = Early results (sanity check) Model was confidently wrong Averaged results of 25% mini-batches: TRUTH M829A1 M830 M830A1 M1028 TOTAL ACC % M829A % M % M830A % M % TOTAL

14 Augmented training data CATALYST tool Noise background Transparent on top of tank scene background

15 Re-training baseline model Still treating as image classification ~10,000 images per class Switched from DIGITS to manual

16 Misclassified images No longer deciding that everything is an M829A1 Mistakes now due to orientation, possibly also due to shadowing

17 Better results 99% accuracy on synthetic imagery, 76% on action shots Need to incorporate real imagery in next model Good enough to switch focus to deployment on Raspberry Pi To build TF on RPi, relied heavily on excellent guide in: Makefile needed for RPi can be found at: es.txt

18 RPi struggled to keep up Need to catch a specific 3s critical window over many hours of movement in scene Evaluated several approaches Frame grabs High accuracy, low false positives, but too slow (1/4 fps) Darknet/YOLO video Could not run it usefully on RPi Possibility of hardware trigger from cabinet door opening: discarded due to complexity Just sending imagery to server for processing there

19 RPi struggled to keep up Need to catch a specific 3s critical window over many hours of movement in scene Evaluated several approaches Frame grabs High accuracy, low false positives, but too slow (1/4 fps) Darknet/YOLO video Could not run it usefully on RPi Possibility of hardware trigger from cabinet door opening: discarded due to complexity Just sending imagery to server for processing there

20 TF model_pruning Attempted to simplify network down to an RPi level /python/tf/contrib/model_pruning/pruning Exploit sparsity of large model TensorFlow model_pruning Threshold & mask Prune, train(100), repeat pb reduced from 87.4 MB to 22.4 MB Sacrifice ~3% model accuracy for ~60% speedup Still only getting ~1/2 fps on RPi

21 MobileNets Very different approach Small-dense models vs large-sparse [pruned] model (same number of calcs) Depthwise-separable convolutions followed by 1x1 pointwise convolution = 1/8 the MAC of a regular convolution Depending on settings for W and resolution, pb size ranged from 16.7 MB down to 1.9 MB (!) Peak accuracy was still around 75%

22 Size on disk (MB) = MobileNets tradeoff space Resolution W Width multiplier only affected MAC, not parameters count

23 Accuracy = MobileNets tradeoff space Resolution W W made a bigger impact than R (W 0.5, R 192) accuracy fell off quickly

24 Latest TF model results on Raspberry Pi 2B Model Accuracy Fps on RPi 2B GoogLeNet / Inception v ~1/4 model_prune(googlenet) 0.73 ~1/2 MobileNet ~1/2 MobileNet ~1 MobileNet ~1 Frame size = 320x240 Possible issues other than CPU processing: camera data bus

25 Jetson TX2 GPU hardware + cudnn + TensorRT 3 Conclusion: TX2 is far overpowered for the application requirements No latency or processing issues at all 24 fps YOLO accuracy: pretty good anecdotally

26 TensorRT 3 Optimization engine for Caffe/TF models running on NVIDIA GPU Layer and tensor fusion and elimination of unused layers; FP16 and INT8 reduced precision calibration; Target-specific autotuning; Efficient memory reuse Source =

27 Next steps Taylor criteria ranking ¼ size, 3x faster, 2%-5% accuracy loss? Sparse MobileNets? fp16, int8, maybe even fixed-point (quantized)? RGB - YCbCr? Reduced image resolution? TF object detection (not just image classification) Updated dataset Draw boundaries on still images by hand using LabelImg CATALYST generated bonding boxes on the synthetic images Convert to TFRecords Optimize for speed/accuracy tradeoff Video again: SSD, F-RCNN on Jetson

28 Conclusions Visual classification is feasible in daylight conditions NIR camera or other night vision needed for dark conditions Pruning reduced network by 3X RPi 2B could only handle ~1 image/sec, even with extensive compression and optimization tf.model_prune = best accuracy TF MobileNets = best speed Jetson TX2 exhibited no practical limits in this application TensorRT 3