CSE Tue 10/09. Nadir Weibel

Transcription

1 CSE Tue 10/09 Nadir Weibel

2 Today Admin Teams Assignments, grading, submissions Mini Quiz on Week 1 (readings and class material) Low-Fidelity Prototyping 1st Project Assignment Computer Vision, Kinect, and Depth Cameras

3 Who is NOT in a team yet?

4 Assignments: Google Classroom

5 Go to classroom.google.com

6 Make sure that you are using the right Google account

7 If everything went well, you should see your classes (118 or 218)

8 If you still do not see CSE 218 or CSE 118 there, you can join the class using a code

9 If you still do not see CSE 218 or CSE 118 there, you can join the class using a code

10 Code for CSE 118

11 If you are an extension student, send us your to

12 MiniQuiz 5min Go to this web site

13 Low-Fidelity and Paper Prototyping

14 A low-fidelity prototype is a prototype that is sketchy and incomplete, that has some characteristics of the target product but is otherwise simple, usually in order to quickly produce the prototype and test broad concepts.

15 From Requirements to Specification Storyboarding Paper prototypes Low-Fidelity Prototypes Digital mock-ups

16 It is not a design pre-view.

17 It will be generic or even ugly, but that is OK.

18 Rough Sketch

19 Rapid Prototyping at Google X

20 Low-Fidelity: Paper Prototype

21 Lo-Fi Prototyping for AR

22 Lo-Fi Prototyping for Mobile

23 More Prototyping

24 Project Assignment As a team: submit 3x Low Fidelity Prototypes by Sunday 10/14/2018, 11.59pm 3 short videos (max 1min) 3 short descriptions (max 1/2 page) Work as a team On Google Classroom

25 Computer Vision

26 Computer Vision Make computers understand images and video. What kind of scene? Where are the cars? How far is the building?

27 What is Computer Vision? Computer vision is the science and technology of machines that see. Concerned with the theory for building artificial systems that obtain information from images. The image data can take many forms, such as a video sequence, depth images, views from multiple cameras, or multi-dimensional data from a medical scanner

28 Why computer vision matters? Safety Health Security Comfort Fun Access

29 A little story about Computer Vision In 1966, Marvin Minsky at MIT asked his undergraduate student Gerald Jay Sussman to spend the summer linking a camera to a computer and getting the computer to describe what it saw. We now know that the problem is slightly more difficult than that. (Szeliski 2009, Computer Vision)

30 Brief history of computer vision 1966: Minsky assigns computer vision as an undergrad summer project 1960 s: interpretation of synthetic worlds Guzman s: some progress on interpreting selected images 1980 s: ANNs come and go; shift toward geometry and increased mathematical rigor 1990 s: face recognition; statistical analysis in vogue 2000 s: broader recognition; large annotated datasets available; video processing starts; vision & graphics; vision for HCI; internet vision, etc. Ohta Kanade 78 Turk and Pentland 91

31 Examples of Computer Vision use Optical Character Recognition Face Detection Object Recognition Smile Detection Biometrics Smart Cars

32 Kinect and Depth Cameras

33 Depth Cameras Stereo Camera Time-of-Flight Camera Kinect

34 Time of Flight Cameras

35 Kinect Kinect for Windows Commercial version Near Mode No Xbox Kinect v2 Kinect for Xbox Accessory for Xbox Input Sensor

36 History Original technology developed in 2005 Announced in 2009 as codenamed Natal (after the Brazilian city and because of its relation to being of or related to birth ) Released Kinect for Windows Beta on June 16, 2011 On February 1st, released commercial version (which you now have access to for this class) On May, released SDK 1.5

37 Why Kinect? It is the first practical natural interface for machines communicating with people It works in normal rooms It is combining acoustical and visual sense It is recognizing full body movements, even complicated ones It is recognizing persons

38 Sensors and Components IR Emitter Color Sensor IR Depth Sensor Tilt Motor Microphone Array

39 Inside

40 Vision Sensors RGB Camera IR Camera IR Laser Projector

41 Lasers Everywhere!!! Depth map from: 1. analysis of reflection intensity 2. speckle pattern using structured light technique. Ghosts???

42 IR Pattern

43 Depth Map

44 Kinect Depth Image Camera s computer vision image (stream). Bunch of blobs. Represents reflection intensity well.

45 Kinect Bonus: People! Real-Time Human Pose Recognition in Parts from Single Depth Images, Jamie Shotton Andrew Fitzgibbon Mat Cook Toby Sharp Mark Finocchio, Richard Moore Alex Kipman Andrew Blake, CVPR 2011

46 Algorithm design Shotton et al. proposed two main steps: 1. Find body parts 2. Compute joint positions.

47 Synthetic data

48 Synthetic training/testing

49 Common Uses Motion Capture Produces a skeleton made up of joints to capture a person s movement

50 RGB-D sensors Microsoft Kinect V2 Intel RealSense Series Leap Motion structure.io

51

52 Kinect v2

53 Kinect Features $199.99

54 Developement Kinect for Windows Developer Toolkt Microsoft VS 10 or above Main Functionalities RGB map Depth map Skeletal map and tracking Audio Facial recognition

55 Fundamental Data Skeletal Data: Joints in 3D space Provided in meters Depth Data: Near and Far modes. Depth of each pixel. Image Data: Different resolutions/frame rates Audio Array: Speech Recognition SDK

56 Intel Real Sense

57 Leap Motion

58 structure.io

59 Phone Depth Cameras? Apple iphone XS

60 Applications - Games

61 Exergames A game that motivates people to exercise Example: Dance Central Featuring full-body tracking, the magic of Kinect for Xbox 360 allows you to experience real dance with your friends and family. dance-central/ xbox

62 Educational Games Research on how to use the Kinect for education Example: helping students who have learning disabilities The [Kinect] games work on the basis of kinesthetic learning -- the learning style by which students adopt new ideas more easily when they touch or are physically involved in what they are studying. kinems

63 Applications - Health

64 Diagnoses Research on how to use the Kinect to make diagnoses Example: using the Kinect to evaluate whether a person has scoliosis The proposed system is capable to identify the posture of the human body and to monitor the correctness of the realized exercises to correct the scoliosis. The system constructed from the Kinect sensor which has the role to collect the spatial data about the patient. [sic] arnumber=

65 Monitoring Research in monitoring those with health problems Example: monitoring the health of seniors in case of falls We have presented the senior health monitoring system using the Kinect device to collect data and performed posture recognition (sitting versus standing) and gait recognition (normal walking versus abnormal walking). arnumber=

66 Tangible User Interfaces Research to explore a Kinect-based interface using real-life objects Example: creating 3D animations easily with the Kinect During a performance, the puppeteer physically manipulates these puppets in front of a Kinect depth sensor. Our system uses a combination of image-feature matching and 3D shape matching to identify and track the physical puppets. doi= &rep=rep1&type=pdf

67 Natural User Interfaces (NUI) Research done on how to use Kinect as a more natural interface Example: exploring new ways for users to interact with different kinds of maps Hands-free gesture and speech recognition, e.g., as offered by the Kinect sensor, are expected to change our human-computer interaction with online and desktop interfaces.

68 Kinect v2 Demo Janet Johnson

69 Next Steps Read/Annotate all assigned papers Morrison et. al and Rettig Submit your reading-summary and annotations by Thursday 12.30pm Annotate paper #1 (Morrison et. al) Submit reading summary for paper #1 (Morrison et al.) Submissions will be on Google Classroom

70 Next Steps Thursday s Discussion M. Weiser, "The Computer for the 21st Century". Scientific American, pp , September 1991 V. Bush, "As We May Think", Atlantic Monthly, July 1945 Morrison C., et al. "Assessing Multiple Sclerosis With Kinect: Designing Computer Vision Systems for Real- World Use." Human Computer Interaction, (2016): Week 1 and Week 2 Lectures

71 Thursday Discussion CSE : Thursdays 12.30pm-1.50pm Rooms/Groups: Group A: EBU3B (CSE Building) 2154 Group B: EBU3B (CSE Building) 2109 Group C: EBU3B (CSE Building) 2217 Group D: EBU3B (CSE Building) 3217 Group E: EBU3B (CSE Building) 4217 Groups members released tomorrow: Web page and Piazza

72 Thursday Lab Thursday , CSE 2154 Kinect Lab Bring your Computer if you have a Windows machine Hand on experience with coding for Kinect

73 Next Week Augmented, Virtual and Mixed Reality Prototyping with PrototipAR

74 Thanks