ECE 253A: Digital Processing: Course Related Class Website: https://sites.google.com/a/eng.ucsd.edu/ece253fall2017/ Course Graduate Assistants: Nachiket Deo Borhan Vasili Kirill Pirozenko Piazza Grading: Assignments/Programming: 35% (Total 4) (last assignment/project: Due day of the final) Quizzes (in-class): 25% (Total 4, last Dec 1 st 2017) Exam (in-class, November 22, 2017): 40% ECE 253A: Digital Processing: Introduction Week 1-2 (October 2, 2017) : Course Introduction and Announcements What is Digital Processing? How is it related to other disciplines? Examples and Utility Explorations in Visual Perception Reading: Chapter 1 and 2 (Gonzalez, Woods book) Articles on the Class web Digital Processing What is Digital Processing? Digital Processing: Introduction Digital Processing and mother discipline: Relationship with other disciplines: 1) Digital Signal Processing, 2) Pattern Recognition 3) Computer Vision 4) Computer Graphics 5) Computational Photography Signal Signal Processing Applications: Signal Vision -- disciplines Enhancement Filtering Coding Compression 1
Digital Processing: Introduction Digital Processing and mother discipline: Signal Signal Processing Signal Digital signal processing (DSP) is the numerical manipulation of signals, usually with the intention to measure, filter, produce or compress continuous analog signals. It is characterized by the use of digital signals to represent these signals as discrete time, discrete frequency, or other discrete domain signals in the form of a sequence of numbers or symbols to permit the digital processing of these signals. https://en.wikipedia.org/wiki/digital_signal_processing Digital Signal Processing Represent signals by a sequence of numbers Sampling or analog-to-digital conversions Perform processing on these numbers with a digital processor Digital signal processing Reconstruct analog signal from processed numbers Reconstruction or digital-to-analog conversion analog signal digital signal A/D DSP D/A Analog input analog output Digital recording of music Analog input digital output Touch tone phone dialing Digital input analog output Text to speech Digital input digital output Compression of a file on computer digital signal analog signal Copyright (C) 2005 Güner Arslan 351M Digital Signal Processing (Spring 2005) 15 Pros and Cons of Digital Signal Processing Pros Accuracy can be controlled by choosing word length Repeatable Sensitivity to electrical noise is minimal Dynamic range can be controlled using floating point numbers Flexibility can be achieved with software implementations Non-linear and time-varying operations are easier to implement Digital storage is cheap Digital information can be encrypted for security Price/performance and reduced time-to-market Cons Sampling causes loss of information A/D and D/A requires mixed-signal hardware Limited speed of processors Quantization and round-off errors Speech Signal: Acoustic Features Signal Digital Processing: Introduction Signal Processing Preprocessing 1. Pitch 2. Energy 3. Speaking rate... Acoustic Feature Extraction: A typical block diagram Windowing Black Box Acoustic signal 351M Digital Signal Processing (Spring 2005) Copyright (C) 2005 Güner Arslan 16 2
Digital Signal Processing: Example Digital Signal Processing + Analysis + Understanding Speech Signal Real World Environment Clean Speech What about questions like: Who is the speaker? Male or Female? Noisy Speech What is the language spoken? What is he/she is asking? Angry? Happy? Noise type In-car application Freeway (FWY) Parking lot (PRK) City street (CST) White Noise (WGN) How to achieve clean signal? Ashish Tawari and Mohan M. Trivedi, "Speech Emotion Analysis in Noisy Real-World Environment, International Conference on Pattern Recognition (ICPR), 2010. Ashish Tawari and Mohan M. Trivedi, "Speech Emotion Analysis in Noisy Real-World Environment, International Conference on Pattern Recognition (ICPR), 2010. Digital Signal Processing + Analysis + Understanding Speech Emotion Recognition Higher semantic Analysis Digital Processing: Introduction Digital Processing and sister disciplines: Semantic Analysis: A typical block diagram Processing Signal Signal Processing Model Desired semantics Applications: Enhancement Noise Filtering Compression Ashish Tawari and Mohan M. Trivedi, "Speech Emotion Analysis in Noisy Real-World Environment, International Conference on Pattern Recognition (ICPR), 2010. 3
Digital Processing Digital Processing: Introduction Computer Vision and sister disciplines: Digital Processing and sister disciplines: Processing Processing Processing Example Processing Example 4
Pattern Recognition Pattern Recognition: Features and Feature Space Computer Vision and sister disciplines: Measurement Vector Pattern Recognition Classified Vectors Not necessarily an Can be calculated from Properties http://www.ph.tn.tudelft.nl/prinfo/index.html Pattern Recognition: Ticklish Teddy Bear Computer Vision Types: various kinds of tickling in the class tickles, i.e. tickle static soft, tickle moving soft, tickle static hard, tickle moving hard Classes: the collection of various types in one class Reponses: A higher level of classification than classes where affective content is labeled, i.e. teasing 1. Derived Measurements 2. Models : Prior knowledge about Imaging, Application Domain, and other useful information Computer Vision Examples: Object Recognition Face recognition Lane detection Activity analysis Recognition of Objects and Events embedded in s and Video ( Semantic level Classification) http://courses.media.mit.edu/2004fall/mas622j/04.projects/students/stiehl/stiehl_affective_touch.pdf 5
Vision for Automobiles Active Safety: as Context Visual Context Aware Capture Spaces and Televiewing Televiewing, Interfaces and Virtual Environments Face-Off Challenge s from 23 videos, 20 vehicles, varying occlusions from hands, wearable, etc. LISA Project 2001-2004 56 [1] Sujitha Martin, Ashish Tawari, Erik Murphy-Chutorian, Shinko Y. Cheng, Mohan Trivedi, "On the Design and Evaluation of Robust Head Pose for Visual User Interfaces: Algorithms, Databases, and Comparisons, Automotive User Interfaces and Interactive Vehicular Applications (AUTO-UI) Conference, Oct. 2012. [2] Ashish Tawari, Sujitha Martin, and Mohan M. Trivedi, Continuous head movement estimator for driver assistance: Issues, algorithms, and on-road evaluations, IEEE Transactions on Intelligent Transportation Systems, 2014. Looking at Faces: Levels of Analysis Computer Vision: Introduction Computer Vision and sister disciplines: Mathematical Model of Objects and Events Computer Graphics s ( synthesized ) Examples: Driving Simulation Virtual Tours Video Games Animated Models for Education Face Detection, Landmark Localization, Occlusion & Pose Estimation with Deep CNN Approach 65 6
Computer Vision and sister disciplines: Computer Computer Vision: Graphics Introduction Computer Vision and sister disciplines: Computer Graphics and Processing Examples: Driving Simulation Virtual Tours Video Games Animated Models for Education Mohan M. Trivedi, Tarak L. Gandhi, Kohsia S. Huang, Distributed Interactive Video Arrays for Event Capture and Enhanced Situational Awareness, IEEE Intelligent Systems, Special Issue on AI in Homeland Security,2005 Computer Vision: Introduction Computer Graphics and Processing Basic Computational Hierarchy Object/Scene Understanding Models PATTERN RECOGNITION AND LEARNING COMPUTER GRAPHICS Computer Graphics COMPUTER VISION Analysis IMAGE PROCESSING Processing Capture ( passive or active ) 7
ECE 253A: Digital Processing: Introduction Week 1-2 (October 2, 2017) : Course Introduction and Announcements What is Digital Processing? How is it related to other disciplines? Examples and Utility Explorations in Visual Perception Reading: Digital Processing: Sister Disciplines Marriage of Digital Processing Digital Analysis Machine (Computer) Vision Computer Graphics and Animation Chapter 1 and 2 (Gonzalez, Woods book) Articles on the Class web Doug Fidaleo, Ulrich Neumann. "Analysis of co articulation regions for performance driven facial animation." Computer Animation & Virtual Worlds 2004. Vision and Graphics Fusion - Expression Driven Facial Animation Digital Processing Relationship with other disciplines: ü 1) Digital Signal Processing, ü 2) Pattern Recognition ü 3) Computer Vision ü 4) Computer Graphics 5) Computational Photography Useful Link (course at UC Berkeley): CS294-26: Manipulation and Computational Photography https://inst.eecs.berkeley.edu/~cs194-26/fa14/ D. Fidaleo, U. Neumann. "Analysis of co articulation regions for performance driven facial animation." Computer Animation and Virtual Worlds 2004. 8
Digital Processing: Introduction Digital Processing and sister disciplines: Digital Processing: Introduction Digital Processing and sister disciplines: s, Reflectance, Range, etc. Computational Photography Photograph Computational photography refers to computational image capture + processing + manipulation techniques that enhance or extend the capabilities of digital photography, typically through the use of multiple pictures of the same subject matter, such, as, for example, by using differently exposed pictures of the same scene to extend dynamic range beyond even that of analog filmbased media. [1] https://en.wikipedia.org/wiki/computational_photography s, Reflectance, Range, etc. Computational Photography Photograph Computational photography refers to computational image capture + processing + manipulation techniques that enhance or extend the capabilities of digital photography, Other examples of computational photography include processing and merging differently illuminated images of the same subject matter ("lightspace") and differently focused pictures of the same subject matter. [2] The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. https://en.wikipedia.org/wiki/computational_photography Digital Processing: Computational Photo (FrankenCam-Stanford) Digital Processing s, Reflectance, Range, etc. Computational Photography Photograph s, Reflectance, Range, etc. Computational Photography Photograph 9
Digital Processing: Computational Photo (High Dynamic Rage) Digital Processing: Computational Photo ( Selective-focus ) s, Reflectance, Range, etc. Computational Photography Photograph Digital Processing: Computational Photo ( Panorama ) Digital Processing: Computational Photo ( Panorama ) s, Reflectance, Range, etc. Computational Photography Photograph s, Reflectance, Range, etc. Computational Photography Photograph R. Szeliski, alignment and stitching: a tutorial, Foundations and Trends in Computer Graphics and Vision, 2006 A. Ramirez, E. Ohn-Bar, M. Trivedi, "Panoramic Stitching for Driver Assistance and Applications to Motion Saliency-based Risk Analysis, IEEE Intelligent Transportation Systems Conference, 2013. 10
Digital Processing: Computational Photo ( Stabilization ) Digital Processing: Computational Photo ( Stabilization ) (a) Videos captured with a cell-phone camera tend to be shaky due to the device s size and weight. (b) The rolling shutter used by sensors in these cameras also produces warping in the output frames (we have exagerrated the effect for illustrative purposes). (c) We use gyroscopes to measure the camera s rotations during video capture. (d) We use the measured camera motion to stabilize the video and to rectify the rolling shutter. (Golden Gate photo courtesy of Salim Virji.) A. Karpenko, D. Jacobs, J. Baek, M. Levoy, Digital video stabilization and rolling shutter correction using gyroscopes, Stanford Tech Report, CTSR 2011-03. A. Karpenko, D. Jacobs, J. Baek, M. Levoy, Digital video stabilization and rolling shutter correction using gyroscopes, Stanford Tech Report, CTSR 2011-03. Human Visual System: Capture Human Visual System: Capture 11
Human Visual System: Rods and Cones Capture: Pinhole Camera Neuroscience and Vision Vision: Perceptual Psychology What do you see? Is it really in the image or it is in your mind? http://ruccs.rutgers.edu/~blaser/lecture10.html 12
Human Visual System: Capture Human Visual System: Perceptual Effects Mach Bands Adaptation level: order of 10 10 Subjective brightness: logarithmic function of incident light intensity Brightness adaptation: Dynamic sensitivity Scotopic: dim light vision, with rods Photopic: bright light vision, with cones Human Visual System: Perceptual Effects - Simultaneous Contrast Human Visual System: Perceptual Effects - Simultaneous Contrast 13
Human Visual System: Perceptual Effects 14
Spatial Frequency Human Vision: Spatial Frequency, Contrast Sensitivity, Acuity Spatial Frequency Spatial Frequency 15