Neural Networks The New Moore s Law Chris Rowen, PhD, FIEEE CEO Cognite Ventures December 216
Outline Moore s Law Revisited: Efficiency Drives Productivity Embedded Neural Network Product Segments Efficiency and Productivity in Embedded Neural Networks Breadth of Neural Networks: Vision and the Pixel Explosion Speech Natural Language Efficiency, Productivity and Neural Networks The Embedded Systems Innovation Space The Evolution of Electronic Design and Cognitive Computing
Moore s Law Revisited: Efficiency Drives Productivity Moore s Law : number of transistors in a [economically viable] dense integrated circuit doubles approximately every two years Dennard Scaling Density: Transistors and gates per unit area Speed: Gate delay Impact of scaling L by α (<1)!!! α Power: Energy per switch α! Dennard benefits hit by limits of voltage scaling How is Deep Learning like Moore s Law? Compound effect of cost and performance scaling has revolutionized electronics Calculations/$ improved ~1 1 in 5 years Excess efficiency largely responsible for processor and software revolution
Embedded Neural Network Product Segments Autonomous Vehicles and Robotics Monitoring and Surveillance Human-Machine Interface Personal Device Enhancement Vision Multi-sensor: image, depth, speed Environmental assessment Full surround views Attention monitoring Command interface Multi-mode ASR Social photography Augmented Reality Audio Ultrasonic sensing Acoustic surveillance Health and performance monitoring Mood analysis Command interface ASR social media Hands-free UI Audio geolocation Natural Language Access control Sentiment analysis Mood analysis Command interface Real-time translation Local bots Enhanced search
The Pixel Explosion Computing and communication driven by new data in/out CMOS sensors trigger imaging explosion 99% of of captured raw data is pixels (dwarfs sounds and motion) 1 1 sensors x 1 8 pixels/sec = 1 18 raw pixels/sec 2E+1 1.8E+1 1.6E+1 1.4E+1 1.2E+1 1E+1 8E+9 World Population Three-year sensor population Rapid growth of vision-based products and services Starting 215: more image sensors than people 6E+9 4E+9 2E+9 New Age: Making sense of pixels requires computer cognition 199 1995 2 25 21 215 22
1 categories 12 species of dogs ImageNet Top-1 Error % Computer vision is big, obvious NN domain Many related tasks: classification, localization, segmentation, object recognition, captioning Huge computation in embedded inference Vision is fundamentally hard!! Example: ImageNet Classification: ImageNet Top-1 Error % Vision 5 Rapid Progress on Accuracy 4 3 2 1 211 212 213 Year 214 215 216 Models Getting More Manageable 5 4 3 2 1 5,, 1,, Model Coefficients 15,, Tibetan mastiff Shih-Tzu Norwegian elkhound ImageNet Top-1 Error ImageNet Top-1 % Error % Optimization Doubles Efficiency Bounded Compute Load 525.5 4 25 3 24.5 2 ResNet 5,11 1 24 Cadence 23.5 5 1 15 2GMACs per image 4 GMACs 2 6 25 8
Automated speech recognition (ASR) pipeline: Moving to unified end-to-end trained neural network RNN or Hidden Markov Model Waveform to spectral samples Spectral samples to phonemes Year Word Error Rate on Switchboard ASR 1995 21 215 4% 2% 6.3% Tuske et al, Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR RNN or Hidden Markov Model Phonemes to words She sells seashells by the seashore Speech: From sounds to words
Natural Language Real time natural language interpretation crucial for rich human-machine interfaces But how do we automatically find word meanings?? Powerful approach: Automatically learn high-dimension vector embedding (N =3-6) from the word usage in large data-sets Words whose vectors are close have strong relationship Different dimensions reflect different kinds of word relationships: good: better good: fine good: bad good: product Cool application: vector arithmetic reveals complex relationships: Translation goes NN: Full sentence-at-a-time translation With [Google Neural Machine Translation], Google Translate is improving more in a single leap than we ve seen in the last ten years combined - Barak Turovsky V( king ) V( man ) + V( woman ) V( queen ) See: Mikolov et al, Efficient Estimation of Word Representations in Vector Space Source: https://research.googleblog.com/216/9/a-neural-network-for-machine.html
Efficiency and Neural Networks 1, Neural Network Platforms Efficiency: Conventional wisdom - deep neural networks much less efficient than hand-tuned feature recognition methods (but more effective) Convolutional neural networks allow High parallelism Low bit resolution Structured, specialized architectures Manageable memory bandwidth ~1x energy improvement over GP CPU may compensate for efficiency gap GMACS Per Watt 1, 1 1 GP CPU Vision DSPs Embedded GPUs CNN engines Vision + NN DSPs FPGAs 1 1 1, 1, 1, Vision DSP core 1 Vision DSP core 1 cluster Vision DSP core 2 Vision DSP core 2 cluster Embedded GPU core cluster Data Center GPU 1 cluster Embedded GPU cluster FPGA 1 FPGA 2 Data Center GPUs Convolutional neural network (CNN) engine CNN engine cluster Data Center GPU 2 cluster GMACs
Productivity and Neural Networks machine learning technology is on the cusp of eating software -Alex Woodie Productivity: Training neural networks often much easier than coding of application-specific features Neural network methods routinely perform better than best manual methods Massive educational shift underway to make machine learning a basic CS skill MOOCs: Andrew Ng s Stanford Machine Learning course: 1, students registered. Hype still exceeds reality Productivity Impact at Three Levels: 1.Programming: Trained NN replace hand-design of feature detectors 2.Applications: Rich machine learning frameworks make opens sophisticated NN to non-experts 3.Business: Deep Learning methods are enough better and faster to influence the whole tech sector. We are in the first phase, when very rapid productivity gains become possible in parts of the economy that are simply too small to affect the overall numbers [T]he tech sector should experience greater productivity gains over a greater range of businesses, potentially nudging measured productivity upward. -- The Economist, July 23,214
Improved algorithms Moore s Law Implementation Efficiency The Embedded Systems Innovation Space Deep Learning Innovation Productivity
The Evolution of Electronic Design Cognitive Computing will be a long-term driver for electronics Circuit sim Custom layout Extraction/DRC 3-state sim Logic synthesis Std Cell P&R Static timing Formal verification Optimizing compiler RTOS Debugger/profiler MP programming App dev framework Computing Processor Based: AssemblyèHLLèEcoSystemèOpen Source Digital Circuits: discreteèttlèrtlèip Reuse Analog Circuits: tubeèdiscreteèicèip Reuse Parallel stochastic network training Network structure synthesis Auto data labeling & augmentation Cognitive 193 194 195 196 197 198 199 2 21 22 23 24
neural network technology and applications