GESTURE RECOGNITION WITH 3D CNNS

Size: px
Start display at page:

Download "GESTURE RECOGNITION WITH 3D CNNS"

Transcription

1 April 4-7, 2016 Silicon Valley GESTURE RECOGNITION WITH 3D CNNS Pavlo Molchanov Xiaodong Yang Shalini Gupta Kihwan Kim Stephen Tyree Jan Kautz 4/6/2016

2 Motivation AGENDA Problem statement Selecting the best classifier Online gesture detection and classification Demos 2

3 MOTIVATION 3

4 GESTURE IS NATURAL FORM OF COMMUNICATION 4 photo.elsoar.com

5 SAFE INTERFACES bmw.com

6 IN NEED FOR VIDEO RELAY SERVICES

7 leapmotion 7

8 PROBLEM STATEMENT 8

9 PROBLEM STATEMENT No special devices Single commodity sensor: Gesture recognition Skeleton tracking Gaze estimation Kinectv1 Head tracking SoftKinetic 9

10 PROBLEM STATEMENT Understanding gesture concepts We do: We don t: Classifier Thumb up Classifier Wave hand Hand model fitting and tracking * 10

11 PROBLEM STATEMENT Understanding gesture concepts We do: We don t: Classifier Thumb up?????? Classifier Wave hand Hand model fitting and tracking * 11

12 SELECTING THE BEST CLASSIFIER 12

13 SELECTING THE BEST CLASSIFIER VIVA CHALLENGE 2015 organized by UCLA 19 classes, 8 subjects Driver and passenger RGB + Depth from Microsoft Kinect 885 gestures in total 13

14 SELECTING THE BEST CLASSIFIER VIVA CHALLENGE 2015 organized by UCLA 19 classes, 8 subjects Driver and passenger RGB + Depth from Microsoft Kinect 885 gestures in total Gesture example: Slide 2 fingers left 14

15 SELECTING THE BEST CLASSIFIER VIVA CHALLENGE 2015 organized by UCLA 19 classes, 8 subjects Driver and passenger RGB + Depth from Microsoft Kinect 885 gestures in total Gesture example: Zoom out 15

16 SELECTING THE BEST CLASSIFIER VIVA CHALLENGE 2015 organized by UCLA 19 classes, 8 subjects Driver and passenger RGB + Depth from Microsoft Kinect 885 gestures in total Gesture example: Rotate CCW 16

17 RGB Depth Prediction SELECTING THE BEST CLASSIFIER 3D Convolutional Neural Network ReLU ReLU 3D convolution and max-pooling 3D convolution and max-pooling 3D convolution and max-pooling 3D convolution and max-pooling Softmax 17

18 RGB Depth SEGMENTED GESTURE CLASSIFICATION Training error 3D CNN Back propagation update 18

19 SELECTING THE BEST CLASSIFIER First result HON4D 1 HOG 2 3D-CNN Testing set 58.7% 64.5% 48.3% Training set 99.9% Classification accuracy, higher better 1 Oreifej and Liu. HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences, CVPR, Ohn-Bar and Trivedi, IEEE Trans. on Intelligent Transportation Systems,

20 SELECTING THE BEST CLASSIFIER IMAGENET VIVA 1.5 M examples 885 examples Recent success in deep learning benefited from large data 20

21 RGB Depth SELECTING THE BEST CLASSIFIER Training error 3D CNN Back propagation update 21

22 RGB Depth SELECTING THE BEST CLASSIFIER Training error Data augmentation 3D CNN Back propagation update 22

23 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 23

24 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 24

25 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 25

26 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 26

27 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 27

28 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Original Temporal augmentation Generating new training data Augmented 28

29 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Temporal augmentation Generating new training data 29

30 SELECTING THE BEST CLASSIFIER Data augmentation Spatial geometric transformations Temporal augmentation Generating new training data flip 30

31 SELECTING THE BEST CLASSIFIER AUGMENTED VIVA 0.3 M examples 885 examples 31

32 SELECTING THE BEST CLASSIFIER Official challenge results NVIDIA (3D-CNN) No data augmentation 48.3 HOG+HOG HON4D 58.7 Dense Trajectories 54 HOG3D 44.6 Harris-3.5D Classification accuracy, higher better 32

33 SELECTING THE BEST CLASSIFIER Official challenge results with data augmentation NVIDIA (3D-CNN) HOG+HOG HON4D 58.7 Dense Trajectories 54 HOG3D 44.6 Harris-3.5D Classification accuracy, higher better 33

34 SELECTING THE BEST CLASSIFIER Speed NVIDIA (3D-CNN) 110 GPU +250 cudnnv HOG+HOG2 50 HON4D Dense Trajectories CPU HOG3D 3 Harris-3.5D FPS, higher better 34

35 SEGMENTED GESTURE CLASSIFICATION Start of the gesture time Gesture End of the gesture Classification Decision Decision after gesture ends introduces latency 35

36 ONLINE GESTURE DETECTION AND CLASSIFICATION 36

37 ONLINE GESTURE CLASSIFICATION Start of the gesture time Gesture Classification End of the gesture Decision Decision before gesture ends improve feedback and user experience 37

38 3D CNN 3D CNN global motion descriptor ONLINE GESTURE CLASSIFICATION R3DCNN Connectionist Temporal Classification (CTC) softmax softmax softmax RNN RNN RNN Forward recurrence only Detection and classification 109M parameters CTC for training only local motion descriptor 8 frames Video server 38

39 ONLINE GESTURE CLASSIFICATION Training loss function Labeling dynamic gestures is difficult Labeling per frame is ambiguous Input: Labels: Loss function: Per frame negative log likelihood 39

40 ONLINE GESTURE CLASSIFICATION Training loss function Sequence based training is the solution Input: Sequence: nothing slide right nothing slide left - nothing Loss function: Connectionist Temporal Classification (CTC) by A. Graves et al. 40

41 ONLINE GESTURE CLASSIFICATION Italian sign language recognition Chalearn2014 challenge held in 2014 RGBD videos of 20 Italian sign language 13K gestures 20 subjects 41

42 ONLINE GESTURE CLASSIFICATION Italian sign language recognition Classification accuracy (%) % Improvement in accuracy By seeing only 41% Pigou et al.* 3D-CNN 3D-CNN CTC of gesture *L. Pigou et al. Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video 42

43 ONLINE GESTURE CLASSIFICATION Italian sign language recognition 35% Improvement in accuracy By seeing only 41% No pre- or post-processing of gesture 43

44 ONLINE GESTURE CLASSIFICATION Car interfaces In-house database Media player, navigation, phone 20 subjects, 25 gestures More information at CVPR

45 ONLINE GESTURE CLASSIFICATION Car interfaces In-house database Human 88 Media player, navigation, phone Ours subjects, 25 gestures More information at CVPR2016 C3D idt SNV Two stream CNN 66 HOG+HOG

46 ONLINE GESTURE CLASSIFICATION Latency is critical Suitability of hardware for inference: IMAGE CLASSIFICATION CPU GPU VIDEO CLASSIFICATION CPU GPU 46

47 ONLINE GESTURE CLASSIFICATION Scalability NVIDIA TX1 - for embedded solutions Credit card GPU in your pocket Our R3DCNN takes only 30% of GPU 47

48 CONTRIBUTIONS Data augmentation helps a lot to deep learning R3DCNN are the best for sign language and gesture recognition CTC helps a lot for video sequence learning Scalable enough to run on NVIDIA TX1 48

49 April 4-7, 2016 Silicon Valley Deep Learning Data Augmentation CTC

50 April 4-7, 2016 Silicon Valley THANK YOU JOIN THE NVIDIA DEVELOPER PROGRAM AT developer.nvidia.com/join

Wadehra Kartik, Kathpalia Mukul, Bahl Vasudha, International Journal of Advance Research, Ideas and Innovations in Technology

Wadehra Kartik, Kathpalia Mukul, Bahl Vasudha, International Journal of Advance Research, Ideas and Innovations in Technology ISSN: 2454-132X Impact factor: 4.295 (Volume 4, Issue 1) Available online at www.ijariit.com Hand Detection and Gesture Recognition in Real-Time Using Haar-Classification and Convolutional Neural Networks

More information

Continuous Gesture Recognition Fact Sheet

Continuous Gesture Recognition Fact Sheet Continuous Gesture Recognition Fact Sheet August 17, 2016 1 Team details Team name: ICT NHCI Team leader name: Xiujuan Chai Team leader address, phone number and email Address: No.6 Kexueyuan South Road

More information

SIMULATION-BASED MODEL CONTROL USING STATIC HAND GESTURES IN MATLAB

SIMULATION-BASED MODEL CONTROL USING STATIC HAND GESTURES IN MATLAB SIMULATION-BASED MODEL CONTROL USING STATIC HAND GESTURES IN MATLAB S. Kajan, J. Goga Institute of Robotics and Cybernetics, Faculty of Electrical Engineering and Information Technology, Slovak University

More information

In-Vehicle Hand Gesture Recognition using Hidden Markov Models

In-Vehicle Hand Gesture Recognition using Hidden Markov Models 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) Windsor Oceanico Hotel, Rio de Janeiro, Brazil, November 1-4, 2016 In-Vehicle Hand Gesture Recognition using Hidden

More information

23270: AUGMENTED REALITY FOR NAVIGATION AND INFORMATIONAL ADAS. Sergii Bykov Technical Lead Machine Learning 12 Oct 2017

23270: AUGMENTED REALITY FOR NAVIGATION AND INFORMATIONAL ADAS. Sergii Bykov Technical Lead Machine Learning 12 Oct 2017 23270: AUGMENTED REALITY FOR NAVIGATION AND INFORMATIONAL ADAS Sergii Bykov Technical Lead Machine Learning 12 Oct 2017 Product Vision Company Introduction Apostera GmbH with headquarter in Munich, was

More information

VISION-BASED gesture recognition [1], [2] is an important

VISION-BASED gesture recognition [1], [2] is an important 1038 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 20, NO. 5, MAY 2018 EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition Yifan Zhang, Member, IEEE, CongqiCao,Jian Cheng, Member, IEEE,

More information

Deep Learning. Dr. Johan Hagelbäck.

Deep Learning. Dr. Johan Hagelbäck. Deep Learning Dr. Johan Hagelbäck johan.hagelback@lnu.se http://aiguy.org Image Classification Image classification can be a difficult task Some of the challenges we have to face are: Viewpoint variation:

More information

Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN

Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN Recognizing Gestures on Projected Button Widgets with an RGB-D Camera Using a CNN Patrick Chiu FX Palo Alto Laboratory Palo Alto, CA 94304, USA chiu@fxpal.com Chelhwon Kim FX Palo Alto Laboratory Palo

More information

Lecture 19: Depth Cameras. Kayvon Fatahalian CMU : Graphics and Imaging Architectures (Fall 2011)

Lecture 19: Depth Cameras. Kayvon Fatahalian CMU : Graphics and Imaging Architectures (Fall 2011) Lecture 19: Depth Cameras Kayvon Fatahalian CMU 15-869: Graphics and Imaging Architectures (Fall 2011) Continuing theme: computational photography Cheap cameras capture light, extensive processing produces

More information

Research on Hand Gesture Recognition Using Convolutional Neural Network

Research on Hand Gesture Recognition Using Convolutional Neural Network Research on Hand Gesture Recognition Using Convolutional Neural Network Tian Zhaoyang a, Cheng Lee Lung b a Department of Electronic Engineering, City University of Hong Kong, Hong Kong, China E-mail address:

More information

Camera Model Identification With The Use of Deep Convolutional Neural Networks

Camera Model Identification With The Use of Deep Convolutional Neural Networks Camera Model Identification With The Use of Deep Convolutional Neural Networks Amel TUAMA 2,3, Frédéric COMBY 2,3, and Marc CHAUMONT 1,2,3 (1) University of Nîmes, France (2) University Montpellier, France

More information

Semantic Segmentation on Resource Constrained Devices

Semantic Segmentation on Resource Constrained Devices Semantic Segmentation on Resource Constrained Devices Sachin Mehta University of Washington, Seattle In collaboration with Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi Project

More information

Detection and Segmentation. Fei-Fei Li & Justin Johnson & Serena Yeung. Lecture 11 -

Detection and Segmentation. Fei-Fei Li & Justin Johnson & Serena Yeung. Lecture 11 - Lecture 11: Detection and Segmentation Lecture 11-1 May 10, 2017 Administrative Midterms being graded Please don t discuss midterms until next week - some students not yet taken A2 being graded Project

More information

The Art of Neural Nets

The Art of Neural Nets The Art of Neural Nets Marco Tavora marcotav65@gmail.com Preamble The challenge of recognizing artists given their paintings has been, for a long time, far beyond the capability of algorithms. Recent advances

More information

GESTURE RECOGNITION FOR ROBOTIC CONTROL USING DEEP LEARNING

GESTURE RECOGNITION FOR ROBOTIC CONTROL USING DEEP LEARNING 2017 NDIA GROUND VEHICLE SYSTEMS ENGINEERING AND TECHNOLOGY SYMPOSIUM AUTONOMOUS GROUND SYSTEMS (AGS) TECHNICAL SESSION AUGUST 8-10, 2017 - NOVI, MICHIGAN GESTURE RECOGNITION FOR ROBOTIC CONTROL USING

More information

Tiny ImageNet Challenge Investigating the Scaling of Inception Layers for Reduced Scale Classification Problems

Tiny ImageNet Challenge Investigating the Scaling of Inception Layers for Reduced Scale Classification Problems Tiny ImageNet Challenge Investigating the Scaling of Inception Layers for Reduced Scale Classification Problems Emeric Stéphane Boigné eboigne@stanford.edu Jan Felix Heyse heyse@stanford.edu Abstract Scaling

More information

Introduction to Machine Learning

Introduction to Machine Learning Introduction to Machine Learning Deep Learning Barnabás Póczos Credits Many of the pictures, results, and other materials are taken from: Ruslan Salakhutdinov Joshua Bengio Geoffrey Hinton Yann LeCun 2

More information

Scene Perception based on Boosting over Multimodal Channel Features

Scene Perception based on Boosting over Multimodal Channel Features Scene Perception based on Boosting over Multimodal Channel Features Arthur Costea Image Processing and Pattern Recognition Research Center Technical University of Cluj-Napoca Research Group Technical University

More information

3D Interaction using Hand Motion Tracking. Srinath Sridhar Antti Oulasvirta

3D Interaction using Hand Motion Tracking. Srinath Sridhar Antti Oulasvirta 3D Interaction using Hand Motion Tracking Srinath Sridhar Antti Oulasvirta EIT ICT Labs Smart Spaces Summer School 05-June-2013 Speaker Srinath Sridhar PhD Student Supervised by Prof. Dr. Christian Theobalt

More information

GPU ACCELERATED DEEP LEARNING WITH CUDNN

GPU ACCELERATED DEEP LEARNING WITH CUDNN GPU ACCELERATED DEEP LEARNING WITH CUDNN Larry Brown Ph.D. March 2015 AGENDA 1 Introducing cudnn and GPUs 2 Deep Learning Context 3 cudnn V2 4 Using cudnn 2 Introducing cudnn and GPUs 3 HOW GPU ACCELERATION

More information

GESTURE BASED HUMAN MULTI-ROBOT INTERACTION. Gerard Canal, Cecilio Angulo, and Sergio Escalera

GESTURE BASED HUMAN MULTI-ROBOT INTERACTION. Gerard Canal, Cecilio Angulo, and Sergio Escalera GESTURE BASED HUMAN MULTI-ROBOT INTERACTION Gerard Canal, Cecilio Angulo, and Sergio Escalera Gesture based Human Multi-Robot Interaction Gerard Canal Camprodon 2/27 Introduction Nowadays robots are able

More information

Semantic Localization of Indoor Places. Lukas Kuster

Semantic Localization of Indoor Places. Lukas Kuster Semantic Localization of Indoor Places Lukas Kuster Motivation GPS for localization [7] 2 Motivation Indoor navigation [8] 3 Motivation Crowd sensing [9] 4 Motivation Targeted Advertisement [10] 5 Motivation

More information

Vision with Precision Webinar Series Augmented & Virtual Reality Aaron Behman, Xilinx Mark Beccue, Tractica. Copyright 2016 Xilinx

Vision with Precision Webinar Series Augmented & Virtual Reality Aaron Behman, Xilinx Mark Beccue, Tractica. Copyright 2016 Xilinx Vision with Precision Webinar Series Augmented & Virtual Reality Aaron Behman, Xilinx Mark Beccue, Tractica Xilinx Vision with Precision Webinar Series Perceiving Environment / Taking Action: AR / VR Monitoring

More information

arxiv: v1 [cs.ce] 9 Jan 2018

arxiv: v1 [cs.ce] 9 Jan 2018 Predict Forex Trend via Convolutional Neural Networks Yun-Cheng Tsai, 1 Jun-Hao Chen, 2 Jun-Jie Wang 3 arxiv:1801.03018v1 [cs.ce] 9 Jan 2018 1 Center for General Education 2,3 Department of Computer Science

More information

DYNAMIC CONVOLUTIONAL NEURAL NETWORK FOR IMAGE SUPER- RESOLUTION

DYNAMIC CONVOLUTIONAL NEURAL NETWORK FOR IMAGE SUPER- RESOLUTION Journal of Advanced College of Engineering and Management, Vol. 3, 2017 DYNAMIC CONVOLUTIONAL NEURAL NETWORK FOR IMAGE SUPER- RESOLUTION Anil Bhujel 1, Dibakar Raj Pant 2 1 Ministry of Information and

More information

Fully Convolutional Networks for Semantic Segmentation

Fully Convolutional Networks for Semantic Segmentation Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer* Trevor Darrell UC Berkeley Presented by: Gordon Christie 1 Overview Reinterpret standard classification convnets as

More information

Gesture Recognition with Real World Environment using Kinect: A Review

Gesture Recognition with Real World Environment using Kinect: A Review Gesture Recognition with Real World Environment using Kinect: A Review Prakash S. Sawai 1, Prof. V. K. Shandilya 2 P.G. Student, Department of Computer Science & Engineering, Sipna COET, Amravati, Maharashtra,

More information

Lecture 11-1 CNN introduction. Sung Kim

Lecture 11-1 CNN introduction. Sung Kim Lecture 11-1 CNN introduction Sung Kim 'The only limit is your imagination' http://itchyi.squarespace.com/thelatest/2012/5/17/the-only-limit-is-your-imagination.html Lecture 7: Convolutional

More information

Deep Neural Network Architectures for Modulation Classification

Deep Neural Network Architectures for Modulation Classification Deep Neural Network Architectures for Modulation Classification Xiaoyu Liu, Diyu Yang, and Aly El Gamal School of Electrical and Computer Engineering Purdue University Email: {liu1962, yang1467, elgamala}@purdue.edu

More information

Convolutional Neural Network-based Steganalysis on Spatial Domain

Convolutional Neural Network-based Steganalysis on Spatial Domain Convolutional Neural Network-based Steganalysis on Spatial Domain Dong-Hyun Kim, and Hae-Yeoun Lee Abstract Steganalysis has been studied to detect the existence of hidden messages by steganography. However,

More information

Advanced Techniques for Mobile Robotics Location-Based Activity Recognition

Advanced Techniques for Mobile Robotics Location-Based Activity Recognition Advanced Techniques for Mobile Robotics Location-Based Activity Recognition Wolfram Burgard, Cyrill Stachniss, Kai Arras, Maren Bennewitz Activity Recognition Based on L. Liao, D. J. Patterson, D. Fox,

More information

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. ECE 289G: Paper Presentation #3 Philipp Gysel

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. ECE 289G: Paper Presentation #3 Philipp Gysel DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition ECE 289G: Paper Presentation #3 Philipp Gysel Autonomous Car ECE 289G Paper Presentation, Philipp Gysel Slide 2 Source: maps.google.com

More information

What Is And How Will Machine Learning Change Our Lives. Fair Use Agreement

What Is And How Will Machine Learning Change Our Lives. Fair Use Agreement What Is And How Will Machine Learning Change Our Lives Raymond Ptucha, Rochester Institute of Technology 2018 Engineering Symposium April 24, 2018, 9:45am Ptucha 18 1 Fair Use Agreement This agreement

More information

INTAIRACT: Joint Hand Gesture and Fingertip Classification for Touchless Interaction

INTAIRACT: Joint Hand Gesture and Fingertip Classification for Touchless Interaction INTAIRACT: Joint Hand Gesture and Fingertip Classification for Touchless Interaction Xavier Suau 1,MarcelAlcoverro 2, Adolfo Lopez-Mendez 3, Javier Ruiz-Hidalgo 2,andJosepCasas 3 1 Universitat Politécnica

More information

Embedding Artificial Intelligence into Our Lives

Embedding Artificial Intelligence into Our Lives Embedding Artificial Intelligence into Our Lives Michael Thompson, Synopsys D&R IP-SOC DAYS Santa Clara April 2018 1 Agenda Introduction What AI is and is Not Where AI is being used Rapid Advance of AI

More information

Vehicle Color Recognition using Convolutional Neural Network

Vehicle Color Recognition using Convolutional Neural Network Vehicle Color Recognition using Convolutional Neural Network Reza Fuad Rachmadi and I Ketut Eddy Purnama Multimedia and Network Engineering Department, Institut Teknologi Sepuluh Nopember, Keputih Sukolilo,

More information

Mobile Cognitive Indoor Assistive Navigation for the Visually Impaired

Mobile Cognitive Indoor Assistive Navigation for the Visually Impaired 1 Mobile Cognitive Indoor Assistive Navigation for the Visually Impaired Bing Li 1, Manjekar Budhai 2, Bowen Xiao 3, Liang Yang 1, Jizhong Xiao 1 1 Department of Electrical Engineering, The City College,

More information

Lecture 23 Deep Learning: Segmentation

Lecture 23 Deep Learning: Segmentation Lecture 23 Deep Learning: Segmentation COS 429: Computer Vision Thanks: most of these slides shamelessly adapted from Stanford CS231n: Convolutional Neural Networks for Visual Recognition Fei-Fei Li, Andrej

More information

Biologically Inspired Computation

Biologically Inspired Computation Biologically Inspired Computation Deep Learning & Convolutional Neural Networks Joe Marino biologically inspired computation biological intelligence flexible capable of detecting/ executing/reasoning about

More information

Neural Networks The New Moore s Law

Neural Networks The New Moore s Law Neural Networks The New Moore s Law Chris Rowen, PhD, FIEEE CEO Cognite Ventures December 216 Outline Moore s Law Revisited: Efficiency Drives Productivity Embedded Neural Network Product Segments Efficiency

More information

Author(s) Corr, Philip J.; Silvestre, Guenole C.; Bleakley, Christopher J. The Irish Pattern Recognition & Classification Society

Author(s) Corr, Philip J.; Silvestre, Guenole C.; Bleakley, Christopher J. The Irish Pattern Recognition & Classification Society Provided by the author(s) and University College Dublin Library in accordance with publisher policies. Please cite the published version when available. Title Open Source Dataset and Deep Learning Models

More information

Thermal Image Enhancement Using Convolutional Neural Network

Thermal Image Enhancement Using Convolutional Neural Network SEOUL Oct.7, 2016 Thermal Image Enhancement Using Convolutional Neural Network Visual Perception for Autonomous Driving During Day and Night Yukyung Choi Soonmin Hwang Namil Kim Jongchan Park In So Kweon

More information

Robust Hand Gesture Recognition for Robotic Hand Control

Robust Hand Gesture Recognition for Robotic Hand Control Robust Hand Gesture Recognition for Robotic Hand Control Ankit Chaudhary Robust Hand Gesture Recognition for Robotic Hand Control 123 Ankit Chaudhary Department of Computer Science Northwest Missouri State

More information

DEEP LEARNING ON RF DATA. Adam Thompson Senior Solutions Architect March 29, 2018

DEEP LEARNING ON RF DATA. Adam Thompson Senior Solutions Architect March 29, 2018 DEEP LEARNING ON RF DATA Adam Thompson Senior Solutions Architect March 29, 2018 Background Information Signal Processing and Deep Learning Radio Frequency Data Nuances AGENDA Complex Domain Representations

More information

Virtual Grasping Using a Data Glove

Virtual Grasping Using a Data Glove Virtual Grasping Using a Data Glove By: Rachel Smith Supervised By: Dr. Kay Robbins 3/25/2005 University of Texas at San Antonio Motivation Navigation in 3D worlds is awkward using traditional mouse Direct

More information

AI Application Processing Requirements

AI Application Processing Requirements AI Application Processing Requirements 1 Low Medium High Sensor analysis Activity Recognition (motion sensors) Stress Analysis or Attention Analysis Audio & sound Speech Recognition Object detection Computer

More information

Computer Vision in Human-Computer Interaction

Computer Vision in Human-Computer Interaction Invited talk in 2010 Autumn Seminar and Meeting of Pattern Recognition Society of Finland, M/S Baltic Princess, 26.11.2010 Computer Vision in Human-Computer Interaction Matti Pietikäinen Machine Vision

More information

Learned Hand Gesture Classification through Synthetically Generated Training Samples

Learned Hand Gesture Classification through Synthetically Generated Training Samples Learned Hand Gesture Classification through Synthetically Generated Training Samples Kyle Lindgren 1, Niveditha Kalavakonda 1, David E. Caballero 1, Kevin Huang 2, Blake Hannaford 1 Abstract Hand gestures

More information

신경망기반자동번역기술. Konkuk University Computational Intelligence Lab. 김강일

신경망기반자동번역기술. Konkuk University Computational Intelligence Lab.  김강일 신경망기반자동번역기술 Konkuk University Computational Intelligence Lab. http://ci.konkuk.ac.kr kikim01@kunkuk.ac.kr 김강일 Index Issues in AI and Deep Learning Overview of Machine Translation Advanced Techniques in

More information

NU-Net: Deep Residual Wide Field of View Convolutional Neural Network for Semantic Segmentation

NU-Net: Deep Residual Wide Field of View Convolutional Neural Network for Semantic Segmentation NU-Net: Deep Residual Wide Field of View Convolutional Neural Network for Semantic Segmentation Mohamed Samy 1 Karim Amer 1 Kareem Eissa Mahmoud Shaker Mohamed ElHelw Center for Informatics Science Nile

More information

Colorful Image Colorizations Supplementary Material

Colorful Image Colorizations Supplementary Material Colorful Image Colorizations Supplementary Material Richard Zhang, Phillip Isola, Alexei A. Efros {rich.zhang, isola, efros}@eecs.berkeley.edu University of California, Berkeley 1 Overview This document

More information

Creating Intelligence at the Edge

Creating Intelligence at the Edge Creating Intelligence at the Edge Vladimir Stojanović E3S Retreat September 8, 2017 The growing importance of machine learning Page 2 Applications exploding in the cloud Huge interest to move to the edge

More information

ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION

ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION ENGINEERING ENERGY TELECOM TRAVEL AND AVIATION SOFTWARE FINANCIAL SERVICES ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION Sergii Bykov, Technical Lead TECHNOLOGY AUTOMOTIVE Product Vision Road To

More information

LANDMARK recognition is an important feature for

LANDMARK recognition is an important feature for 1 NU-LiteNet: Mobile Landmark Recognition using Convolutional Neural Networks Chakkrit Termritthikun, Surachet Kanprachar, Paisarn Muneesawang arxiv:1810.01074v1 [cs.cv] 2 Oct 2018 Abstract The growth

More information

Artificial Intelligence Machine learning and Deep Learning: Trends and Tools. Dr. Shaona

Artificial Intelligence Machine learning and Deep Learning: Trends and Tools. Dr. Shaona Artificial Intelligence Machine learning and Deep Learning: Trends and Tools Dr. Shaona Ghosh @shaonaghosh What is Machine Learning? Computer algorithms that learn patterns in data automatically from large

More information

Counterfeit Bill Detection Algorithm using Deep Learning

Counterfeit Bill Detection Algorithm using Deep Learning Counterfeit Bill Detection Algorithm using Deep Learning Soo-Hyeon Lee 1 and Hae-Yeoun Lee 2,* 1 Undergraduate Student, 2 Professor 1,2 Department of Computer Software Engineering, Kumoh National Institute

More information

Lesson 08. Convolutional Neural Network. Ing. Marek Hrúz, Ph.D. Katedra Kybernetiky Fakulta aplikovaných věd Západočeská univerzita v Plzni.

Lesson 08. Convolutional Neural Network. Ing. Marek Hrúz, Ph.D. Katedra Kybernetiky Fakulta aplikovaných věd Západočeská univerzita v Plzni. Lesson 08 Convolutional Neural Network Ing. Marek Hrúz, Ph.D. Katedra Kybernetiky Fakulta aplikovaných věd Západočeská univerzita v Plzni Lesson 08 Convolution we will consider 2D convolution the result

More information

Detecting Intra-Room Mobility with Signal Strength Descriptors

Detecting Intra-Room Mobility with Signal Strength Descriptors Detecting Intra-Room Mobility with Signal Strength Descriptors Authors: Konstantinos Kleisouris Bernhard Firner Richard Howard Yanyong Zhang Richard Martin WINLAB Background: Internet of Things (Iot) Attaching

More information

A Vehicular Visual Tracking System Incorporating Global Positioning System

A Vehicular Visual Tracking System Incorporating Global Positioning System A Vehicular Visual Tracking System Incorporating Global Positioning System Hsien-Chou Liao and Yu-Shiang Wang Abstract Surveillance system is widely used in the traffic monitoring. The deployment of cameras

More information

Proposers Day Workshop

Proposers Day Workshop Proposers Day Workshop Monday, January 23, 2017 @srcjump, #JUMPpdw Cognitive Computing Vertical Research Center Mandy Pant Academic Research Director Intel Corporation Center Motivation Today s deep learning

More information

RECENTLY, Human-computer interaction (HCI) has become

RECENTLY, Human-computer interaction (HCI) has become 1 Micro Hand Gesture Recognition System Using Ultrasonic Active Sensing Yu Sang, Laixi Shi, Yimin Liu, Member, IEEE arxiv:1712.00216v1 [eess.sp] 1 Dec 2017 Abstract In this paper, we propose a micro hand

More information

AUDIO TAGGING WITH CONNECTIONIST TEMPORAL CLASSIFICATION MODEL USING SEQUENTIAL LABELLED DATA

AUDIO TAGGING WITH CONNECTIONIST TEMPORAL CLASSIFICATION MODEL USING SEQUENTIAL LABELLED DATA AUDIO TAGGING WITH CONNECTIONIST TEMPORAL CLASSIFICATION MODEL USING SEQUENTIAL LABELLED DATA Yuanbo Hou 1, Qiuqiang Kong 2 and Shengchen Li 1 Abstract. Audio tagging aims to predict one or several labels

More information

An Evaluation of Automatic License Plate Recognition Vikas Kotagyale, Prof.S.D.Joshi

An Evaluation of Automatic License Plate Recognition Vikas Kotagyale, Prof.S.D.Joshi An Evaluation of Automatic License Plate Recognition Vikas Kotagyale, Prof.S.D.Joshi Department of E&TC Engineering,PVPIT,Bavdhan,Pune ABSTRACT: In the last decades vehicle license plate recognition systems

More information

Deep learning for INTELLIGENT machines

Deep learning for INTELLIGENT machines Deep learning for INTELLIGENT machines GAMING DESIGN ENTERPRISE VIRTUALIZATION HPC & CLOUD SERVICE PROVIDERS INTELLIGENT MACHINES THE WORLD LEADER IN VISUAL COMPUTING 2 3 APPLICATIONS OF DEEP LEARNING

More information

Image Manipulation Detection using Convolutional Neural Network

Image Manipulation Detection using Convolutional Neural Network Image Manipulation Detection using Convolutional Neural Network Dong-Hyun Kim 1 and Hae-Yeoun Lee 2,* 1 Graduate Student, 2 PhD, Professor 1,2 Department of Computer Software Engineering, Kumoh National

More information

Understanding Head and Hand Activities and Coordination in Naturalistic Driving Videos

Understanding Head and Hand Activities and Coordination in Naturalistic Driving Videos 214 IEEE Intelligent Vehicles Symposium (IV) June 8-11, 214. Dearborn, Michigan, USA Understanding Head and Hand Activities and Coordination in Naturalistic Driving Videos Sujitha Martin 1, Eshed Ohn-Bar

More information

Deep Learning for Human Activity Recognition: A Resource Efficient Implementation on Low-Power Devices

Deep Learning for Human Activity Recognition: A Resource Efficient Implementation on Low-Power Devices Deep Learning for Human Activity Recognition: A Resource Efficient Implementation on Low-Power Devices Daniele Ravì, Charence Wong, Benny Lo and Guang-Zhong Yang To appear in the proceedings of the IEEE

More information

Artificial Intelligence and Deep Learning

Artificial Intelligence and Deep Learning Artificial Intelligence and Deep Learning Cars are now driving themselves (far from perfectly, though) Speaking to a Bot is No Longer Unusual March 2016: World Go Champion Beaten by Machine AI: The Upcoming

More information

CUDA-Accelerated Satellite Communication Demodulation

CUDA-Accelerated Satellite Communication Demodulation CUDA-Accelerated Satellite Communication Demodulation Renliang Zhao, Ying Liu, Liheng Jian, Zhongya Wang School of Computer and Control University of Chinese Academy of Sciences Outline Motivation Related

More information

GESTURE RECOGNITION SOLUTION FOR PRESENTATION CONTROL

GESTURE RECOGNITION SOLUTION FOR PRESENTATION CONTROL GESTURE RECOGNITION SOLUTION FOR PRESENTATION CONTROL Darko Martinovikj Nevena Ackovska Faculty of Computer Science and Engineering Skopje, R. Macedonia ABSTRACT Despite the fact that there are different

More information

A NEW DATASET FOR HAND GESTURE ESTIMATION

A NEW DATASET FOR HAND GESTURE ESTIMATION A NEW DATASET FOR HAND GESTURE ESTIMATION Biyao Shao 1, Yifeng Xie 1, Hongnan Yang 1, Yatong Jiang 1, Chenggang Yan 1, Hongtao Xie 2, and Yangang Wang 3 1 Hangzhou Dianzi University, Hangzhou, China 2

More information

The Hand Gesture Recognition System Using Depth Camera

The Hand Gesture Recognition System Using Depth Camera The Hand Gesture Recognition System Using Depth Camera Ahn,Yang-Keun VR/AR Research Center Korea Electronics Technology Institute Seoul, Republic of Korea e-mail: ykahn@keti.re.kr Park,Young-Choong VR/AR

More information

CS 7643: Deep Learning

CS 7643: Deep Learning CS 7643: Deep Learning Topics: Toeplitz matrices and convolutions = matrix-mult Dilated/a-trous convolutions Backprop in conv layers Transposed convolutions Dhruv Batra Georgia Tech HW1 extension 09/22

More information

Going Deeper into First-Person Activity Recognition

Going Deeper into First-Person Activity Recognition Going Deeper into First-Person Activity Recognition Minghuang Ma, Haoqi Fan and Kris M. Kitani Carnegie Mellon University Pittsburgh, PA 15213, USA minghuam@andrew.cmu.edu haoqif@andrew.cmu.edu kkitani@cs.cmu.edu

More information

A Step Forward in Virtual Reality. Department of Electrical and Computer Engineering

A Step Forward in Virtual Reality. Department of Electrical and Computer Engineering A Step Forward in Virtual Reality Team Step Ryan Daly Electrical Engineer Jared Ricci Electrical Engineer Joseph Roberts Electrical Engineer Steven So Electrical Engineer 2 Motivation Current Virtual Reality

More information

A Fuller Understanding of Fully Convolutional Networks. Evan Shelhamer* Jonathan Long* Trevor Darrell UC Berkeley in CVPR'15, PAMI'16

A Fuller Understanding of Fully Convolutional Networks. Evan Shelhamer* Jonathan Long* Trevor Darrell UC Berkeley in CVPR'15, PAMI'16 A Fuller Understanding of Fully Convolutional Networks Evan Shelhamer* Jonathan Long* Trevor Darrell UC Berkeley in CVPR'15, PAMI'16 1 pixels in, pixels out colorization Zhang et al.2016 monocular depth

More information

KINECT HANDS-FREE. Rituj Beniwal. Department of Electrical Engineering Indian Institute of Technology, Kanpur. Pranjal Giri

KINECT HANDS-FREE. Rituj Beniwal. Department of Electrical Engineering Indian Institute of Technology, Kanpur. Pranjal Giri KINECT HANDS-FREE Rituj Beniwal Pranjal Giri Agrim Bari Raman Pratap Singh Akash Jain Department of Aerospace Engineering Indian Institute of Technology, Kanpur Atharva Mulmuley Department of Chemical

More information

Volume 3, Issue 5, May 2015 International Journal of Advance Research in Computer Science and Management Studies

Volume 3, Issue 5, May 2015 International Journal of Advance Research in Computer Science and Management Studies Volume 3, Issue 5, May 2015 International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online at: www.ijarcsms.com A Survey

More information

arxiv: v1 [cs.lg] 2 Jan 2018

arxiv: v1 [cs.lg] 2 Jan 2018 Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing arxiv:1801.00723v1 [cs.lg] 2 Jan 2018 Pegah Karimi pkarimi@uncc.edu Kazjon Grace The University of Sydney Sydney, NSW 2006

More information

SmartCanvas: A Gesture-Driven Intelligent Drawing Desk System

SmartCanvas: A Gesture-Driven Intelligent Drawing Desk System SmartCanvas: A Gesture-Driven Intelligent Drawing Desk System Zhenyao Mo +1 213 740 4250 zmo@graphics.usc.edu J. P. Lewis +1 213 740 9619 zilla@computer.org Ulrich Neumann +1 213 740 0877 uneumann@usc.edu

More information

A STUDY ON DYNAMIC HAND GESTURE RECOGNITION FOR FINGER DISABILITY USING MULTI-LAYER NEURAL NETWORK

A STUDY ON DYNAMIC HAND GESTURE RECOGNITION FOR FINGER DISABILITY USING MULTI-LAYER NEURAL NETWORK A STUDY ON DYNAMIC HAND GESTURE RECOGNITION FOR FINGER DISABILITY USING MULTI-LAYER NEURAL NETWORK 1 ROMI FADILLAH RAHMAT, 2 SARAH PURNAMAWATI, 3 EKA PRATIWI GOENFI, 4 OPIM SALIM SITOMPUL, 5 MUHAMMAD FERMI

More information

Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising

Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising Peng Liu University of Florida pliu1@ufl.edu Ruogu Fang University of Florida ruogu.fang@bme.ufl.edu arxiv:177.9135v1 [cs.cv]

More information

11/13/18. Introduction to RNNs for NLP. About Me. Overview SHANG GAO

11/13/18. Introduction to RNNs for NLP. About Me. Overview SHANG GAO Introduction to RNNs for NLP SHANG GAO About Me PhD student in the Data Science and Engineering program Took Deep Learning last year Work in the Biomedical Sciences, Engineering, and Computing group at

More information

A Vehicular Visual Tracking System Incorporating Global Positioning System

A Vehicular Visual Tracking System Incorporating Global Positioning System A Vehicular Visual Tracking System Incorporating Global Positioning System Hsien-Chou Liao and Yu-Shiang Wang Abstract Surveillance system is widely used in the traffic monitoring. The deployment of cameras

More information

A Vehicular Visual Tracking System Incorporating Global Positioning System

A Vehicular Visual Tracking System Incorporating Global Positioning System Vol:5, :6, 20 A Vehicular Visual Tracking System Incorporating Global Positioning System Hsien-Chou Liao and Yu-Shiang Wang International Science Index, Computer and Information Engineering Vol:5, :6,

More information

Compact Deep Convolutional Neural Networks for Image Classification

Compact Deep Convolutional Neural Networks for Image Classification 1 Compact Deep Convolutional Neural Networks for Image Classification Zejia Zheng, Zhu Li, Abhishek Nagar 1 and Woosung Kang 2 Abstract Convolutional Neural Network is efficient in learning hierarchical

More information

Deep learning architectures for music audio classification: a personal (re)view

Deep learning architectures for music audio classification: a personal (re)view Deep learning architectures for music audio classification: a personal (re)view Jordi Pons jordipons.me @jordiponsdotme Music Technology Group Universitat Pompeu Fabra, Barcelona Acronyms MLP: multi layer

More information

Chess Recognition Using Computer Vision

Chess Recognition Using Computer Vision Chess Recognition Using Computer Vision May 30, 2017 Ramani Varun (U6004067, contribution 50%) Sukrit Gupta (U5900600, contribution 50%) College of Engineering & Computer Science he Australian National

More information

What was the first gestural interface?

What was the first gestural interface? stanford hci group / cs247 Human-Computer Interaction Design Studio What was the first gestural interface? 15 January 2013 http://cs247.stanford.edu Theremin Myron Krueger 1 Myron Krueger There were things

More information

Introduction to Mobile Robotics Welcome

Introduction to Mobile Robotics Welcome Introduction to Mobile Robotics Welcome Wolfram Burgard, Michael Ruhnke, Bastian Steder 1 Today This course Robotics in the past and today 2 Organization Wed 14:00 16:00 Fr 14:00 15:00 lectures, discussions

More information

Can you tell a face from a HEVC bitstream?

Can you tell a face from a HEVC bitstream? Can you tell a face from a HEVC bitstream? Saeed Ranjbar Alvar, Hyomin Choi and Ivan V. Bajić School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada Email: {saeedr,chyomin, ibajic}@sfu.ca

More information

Generating an appropriate sound for a video using WaveNet.

Generating an appropriate sound for a video using WaveNet. Australian National University College of Engineering and Computer Science Master of Computing Generating an appropriate sound for a video using WaveNet. COMP 8715 Individual Computing Project Taku Ueki

More information

CONVOLUTIONAL NEURAL NETWORKS: MOTIVATION, CONVOLUTION OPERATION, ALEXNET

CONVOLUTIONAL NEURAL NETWORKS: MOTIVATION, CONVOLUTION OPERATION, ALEXNET CONVOLUTIONAL NEURAL NETWORKS: MOTIVATION, CONVOLUTION OPERATION, ALEXNET MOTIVATION Fully connected neural network Example 1000x1000 image 1M hidden units 10 12 (= 10 6 10 6 ) parameters! Observation

More information

Driver Assistance for "Keeping Hands on the Wheel and Eyes on the Road"

Driver Assistance for Keeping Hands on the Wheel and Eyes on the Road ICVES 2009 Driver Assistance for "Keeping Hands on the Wheel and Eyes on the Road" Cuong Tran and Mohan Manubhai Trivedi Laboratory for Intelligent and Safe Automobiles (LISA) University of California

More information

CROSS-LAYER FEATURES IN CONVOLUTIONAL NEURAL NETWORKS FOR GENERIC CLASSIFICATION TASKS. Kuan-Chuan Peng and Tsuhan Chen

CROSS-LAYER FEATURES IN CONVOLUTIONAL NEURAL NETWORKS FOR GENERIC CLASSIFICATION TASKS. Kuan-Chuan Peng and Tsuhan Chen CROSS-LAYER FEATURES IN CONVOLUTIONAL NEURAL NETWORKS FOR GENERIC CLASSIFICATION TASKS Kuan-Chuan Peng and Tsuhan Chen Cornell University School of Electrical and Computer Engineering Ithaca, NY 14850

More information

Synthetic View Generation for Absolute Pose Regression and Image Synthesis: Supplementary material

Synthetic View Generation for Absolute Pose Regression and Image Synthesis: Supplementary material Synthetic View Generation for Absolute Pose Regression and Image Synthesis: Supplementary material Pulak Purkait 1 pulak.cv@gmail.com Cheng Zhao 2 irobotcheng@gmail.com Christopher Zach 1 christopher.m.zach@gmail.com

More information

Convolutional neural networks

Convolutional neural networks Convolutional neural networks Themes Curriculum: Ch 9.1, 9.2 and http://cs231n.github.io/convolutionalnetworks/ The simple motivation and idea How it s done Receptive field Pooling Dilated convolutions

More information

Learning to Predict Indoor Illumination from a Single Image. Chih-Hui Ho

Learning to Predict Indoor Illumination from a Single Image. Chih-Hui Ho Learning to Predict Indoor Illumination from a Single Image Chih-Hui Ho 1 Outline Introduction Method Overview LDR Panorama Light Source Detection Panorama Recentering Warp Learning From LDR Panoramas

More information

Hand Gesture Recognition by Means of Region- Based Convolutional Neural Networks

Hand Gesture Recognition by Means of Region- Based Convolutional Neural Networks Contemporary Engineering Sciences, Vol. 10, 2017, no. 27, 1329-1342 HIKARI Ltd, www.m-hikari.com https://doi.org/10.12988/ces.2017.710154 Hand Gesture Recognition by Means of Region- Based Convolutional

More information

CS6700: The Emergence of Intelligent Machines. Prof. Carla Gomes Prof. Bart Selman Cornell University

CS6700: The Emergence of Intelligent Machines. Prof. Carla Gomes Prof. Bart Selman Cornell University EMERGENCE OF INTELLIGENT MACHINES: CHALLENGES AND OPPORTUNITIES CS6700: The Emergence of Intelligent Machines Prof. Carla Gomes Prof. Bart Selman Cornell University Artificial Intelligence After a distinguished

More information