Suggested projects for EL-GY 6123 Image and Video Processing (Spring 2018) 360 Degree Video View Prediction (contact: Chenge Li,

Transcription

1 Suggested projects for EL-GY 6123 Image and Video Processing (Spring 2018) Updated 2/6/ Degree Video View Prediction (contact: Chenge Li, Pan, Junting, et al. "Shallow and deep convolutional networks for saliency prediction." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Wu, Chenglei, et al. "A Dataset for Exploring User Behaviors in VR Spherical Video Streaming." Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, Lo, Wen-Chih, et al. "360 Video Viewing Dataset in Head-Mounted Virtual Reality." Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, Bao, Yanan, et al. "Viewing 360 degree videos: Motion prediction and bandwidth optimization." Network Protocols (ICNP), 2016 IEEE 24th International Conference on. IEEE, Bao, Yanan, et al. "Shooting a moving target: Motion-prediction-based transmission for 360-degree videos." Big Data (Big Data), 2016 IEEE International Conference on. IEEE, machine learning and preferably deep learning 360 degree or panoramic video or image stitching (contact: Yixiang Mao ym1496@nyu.edu) Brown, Matthew, and David G. Lowe. "Automatic panoramic image stitching using invariant features." International journal of computer vision 74.1 (2007): Szeliski, Richard. "Image alignment and stitching: A tutorial." Foundations and Trends in Computer Graphics and Vision2.1 (2006): Shum, Heung-Yeung, and Richard Szeliski. "Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment." International Journal of Computer Vision 36.2 (2000): Image registration and warping Vehicle detection and tracking results visualization on geo-maps in near real-time (Contact: Chenge Li,cl2840@nyu.edu) Fetch data from DoT's public online traffic cameras. Use trained network to detect vehicles. Map the result visualization onto the geography map of New York City. Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE international conference on computer vision Ren, Shaoqing, et al. "Faster R-CNN: Towards real-time object detection with region proposal networks." Advances in neural information processing systems

2 1. Related literature study object detection. 2. Deep learning based facial feature representation knowledge strongly preferred. (Don t worry, you will learn along the way.) 3. Coding in Python and OpenCV, Tensorflow or CAFFE preferred. Coloring black and white images (Contact: Chenge Li,cl2840@nyu.edu) Deep Colorization [pdf], 2015 Colorful Image Colorization [pdf] (website), 2016 Learning Representations for Automatic Colorization [pdf] (website), 2016 Image Colorization with Deep Convolutional Neural Networks [pdf], Basic knowledge about image processing. 2. Coding in Python and OpenCV, Tensorflow or CAFFE preferred. People detection and tracking in video using deep learning method (Contact:?) Faster R-CNN Background: basics of deep learning People detection and tracking in multiview videos (need to associate same person appeared in multiple views) (Contact: Fanyi Duanmu, fanyi.duanmu@nyu.edu) Background desired: Deep learning, and graph matching Image/video debluring or super-resolution with or without knowing the blurring kernels (Contact: Amirhossein Khalilian akg404@nyu.edu ) S. Farsiu, M. D. Robinson, M. Elad and P. Milanfar, "Fast and robust multiframe super resolution," in IEEE Transactions on Image Processing, vol. 13, no. 10, pp , Oct Linear Algebra background is required. Additionally, you need to know some basic topics of convex optimization. Image denoising/debluring using deep learning (Contact: Chuanmin Jia Xu, Li, et al. "Deep convolutional neural network for image deconvolution." Advances in Neural Information Processing Systems Xie, et al. "Image denoising and inpainting with deep neural networks.", NIPS, D Eigen, D Krishnan, and R Fergus. "Restoring an image taken through a window covered with dirt or rain.", ICCV, 2013.

3 Dong C, Deng Y, Change Loy C, and Xiaoou Tang. Compression artifacts reduction by a deep convolutional network., ICCV, Zhang, K., Zuo, W., Chen, Y., Meng, D. and Zhang, L. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising., IEEE Transactions on Image Processing Real or Fake face ID images? (Telling apart real people and faces appeared in painting. May or may not use stereo cameras) (Contact Chenge Li cl2840@nyu.edu) Ranjan, Rajeev, Vishal M. Patel, and Rama Chellappa. "Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition." IEEE Transactions on Pattern Analysis and Machine Intelligence (2017). Yang, Shuo, et al. "Faceness-Net: Face Detection through Deep Facial Part Responses." arxiv preprint arxiv: (2017). 1. Related literature study in face detection and recognition algorithms. 2. Deep learning based facial feature representation knowledge strongly preferred. (Don t worry, you will learn along the way.) 3. Coding in Python and OpenCV, Tensorflow or CAFFE preferred. Medical image segmentation using classical techniques (Contact: Amirhossein Khalilian akg404@nyu.edu ) Level-set-based method: Chan, Tony F., and Luminita A. Vese. "Active contours without edges." IEEE Transactions on image processing 10.2 (2001): Graph cuts-based method: Y. Boykov, O. Veksler, and R. Zabih, Fast approximate energy min- imization via graph cuts, IEEE Transactions on pattern analysis and machine intelligence, vol. 23, no. 11, pp , Background: Linear Algebra and good mathematical background. Medical image segmentation using deep learning (Contact:?) Ref. Long, et al. "Fully convolutional networks for semantic segmentation." CVPR, 2015 Medical Image registration (cross modality, or within the same modality) (Contact: Amirhossein Khalilian akg404@nyu.edu) Image or video compression using deep learning (Contact: Chuanmin Jia,

4 Background: good understanding with compression and deep learning Refs.: Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." arxiv preprint arxiv: (2016). Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." arxiv preprint arxiv: (2016). Toderici, George, et al. "Full Resolution Image Compression with Recurrent Neural Networks." arxiv preprint arxiv: (2016). Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." arxiv preprint arxiv: (2016). Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." arxiv preprint arxiv: (2016). Painting with deep artificial neural networks (Contact Yuan Wang yw1225@nyu.edu) We even have a publication from our previous students in the class! Python, deep learning, Tensorflow, and likely cloud computing platform. 1. Jing, Yongcheng, et al. "Neural style transfer: A review." arxiv preprint arxiv: (2017). 2. Hardy, Megan, and Sumanto Pal. "Split Consideration for Foreground and Background Painting Using Artificial Neural Networks." Proceedings of the 2017 ACM on Multimedia Conference. ACM, (Paper from previous students :) Haze removel (Contact Yuan Wang yw1225@nyu.edu) Programming skill in Python or matlab. Ref. 1. He, Kaiming, Jian Sun, and Xiaoou Tang. "Single image haze removal using dark channel prior." IEEE transactions on pattern analysis and machine intelligence (2011): He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." IEEE transactions on pattern analysis and machine intelligence 35.6 (2013): Multi-person pose estimation (Contact: Yuan Wang yw1225@nyu.edu ) Goal:

5 Deep neural network obtains the state-of-the-art person pose estimation from videos, namely, finding out the skeletons of persons. However, the neural network is applied on each frame independently which leads to shaky artifacts in the estimation results. We aim to use temporal correlation to attenuate the problem. The processing can be put into three steps: 1. Run a pre-trained deep neural network to find initial results. 2. find find body parts corresponding relationship between adjacent frames with bipartite matching 3. Smooth skeletons across time. Python, deep learning, Tensorflow, and likely cloud computing platform. You only need to run pretrain deep learning model. 1. Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." CVPR. Vol. 1. No Kinect captured joints data analysis for exercise guidance or game (Andy Chiang, atc327@nyu.edu ) Basic knowledge of computer vision, image process. Programming skill in Python Xiaodong Yang, YingLi Tian, Effective 3d action recognition using eigenjoints, Journal of Visual Communication and Image Representation, Liuyang Zhou, Zhiguang Liu, Howard Leung and Hubert P. H. Shum, "Posture Reconstruction Using Kinect with a Probabilistic Model," in VRST '14. Feature descriptor compression for mobile visual search (Contact: Chuanmin Jia, Z. Zhang, L. Li, Z. Li, and H. Li, Visual query compression with locality preserving projection on grassmann manifold, IEEE International Conference on Image Processing (ICIP), Z. Zhang, L. Li, and Z. Li, Visual query compression with embedded transforms on grassmann manifold, IEEE International Conference on Multimedia and Expo (ICME), Duan L Y, Chandrasekhar V, Chen J, et al. Overview of the MPEG-CDVS Standard. IEEE Transactions on Image Processing, 2016, 25(1): Duan L Y, Chandrasekhar V, Wang S, et al, Compact Descriptors for Video Analysis: the Emerging MPEG Standard, arxiv preprint arxiv: , 2017.