Exposing Photo Manipulation with Geometric Inconsistencies

Exposing Photo Manipulation with Geometric Inconsistencies James F. O Brien U.C. Berkeley Collaborators Hany Farid Eric Kee Valentina Conotter Stephen Bailey 1 image-forensics-pg14.key - October 9, 2014

Communication by Images 2-1 image-forensics-pg14.key - October 9, 2014

Communication by Images 2-2 image-forensics-pg14.key - October 9, 2014

Image Manipulation Iranian missile test, 2008 3-1 image-forensics-pg14.key - October 9, 2014

Image Manipulation Iranian missile test, 2008 3-2 image-forensics-pg14.key - October 9, 2014

Image Manipulation Iranian stealth fighter, 2013 4-1 image-forensics-pg14.key - October 9, 2014

Image Manipulation Iranian stealth fighter, 2013 4-2 image-forensics-pg14.key - October 9, 2014

Image Manipulation Economist manipulates image of Obama, 2010 5-1 image-forensics-pg14.key - October 9, 2014

Image Manipulation Economist manipulates image of Obama, 2010 5-2 image-forensics-pg14.key - October 9, 2014

Image Manipulation Fabricated image of John Kerry and Jane Fonda, 2004 6-1 image-forensics-pg14.key - October 9, 2014

Image Manipulation Fabricated image of John Kerry and Jane Fonda, 2004 6-2 image-forensics-pg14.key - October 9, 2014

Video Manipulation Flying Birdman Hoax, 2012 7-1 image-forensics-pg14.key - October 9, 2014

Video Manipulation Flying Birdman Hoax, 2012 7-2 image-forensics-pg14.key - October 9, 2014

Historical Image Manipulation Image manipulation as old as photography Primitive techniques work surprisingly well Library of Congress archive photo of Abraham Lincoln 1826 8-1 image-forensics-pg14.key - October 9, 2014

Historical Image Manipulation 9-1 image-forensics-pg14.key - October 9, 2014

Historical Image Manipulation 9-2 image-forensics-pg14.key - October 9, 2014

Image Forensics Detect forgeries Detect signs of manipulation Prove image was modified in some way Cannot prove an image unmodified! Suite of detection tools Individual methods can be countered by informed attacker Individual tools may not apply in all cases Each additional method makes forgery harder 10 image-forensics-pg14.key - October 9, 2014

Advantage: Forgers People: Good at understanding scene content Poor at noticing many types of inconsistencies Simple manipulation methods work well New manipulation methods being developed 11 image-forensics-pg14.key - October 9, 2014

Example Inconsistency Selected as correct: 62.1% Selected as correct: 50.1% N = 20; RT = 7.6s Farid and Bravo 2010 12 image-forensics-pg14.key - October 9, 2014

Things we don t see 13 image-forensics-pg14.key - October 9, 2014

Things we don t see 14 image-forensics-pg14.key - October 9, 2014

Image Forensics Format Methods EXIF meta data Quantization tables Coding decisions Signatures or watermarks Pixel Methods Linear dependance Bayer pattern artifacts Chromatic aberration Compression artifacts Not tied to scene content Easy to apply Easy to fool (informed attacker) Not robust to common operations 16 image-forensics-pg14.key - October 9, 2014

Image Forensics Geometric methods Content inconsistencies Require human annotation Computer analysis Examples: Shadows Lighting Reflections 17 image-forensics-pg14.key - October 9, 2014

Geometric Image Forensics Not same as Computer Vision Possibly user involved in loop Only looking for inconsistencies only Don t need to fully extract scene content 18 image-forensics-pg14.key - October 9, 2014

19 image-forensics-pg14.key - October 9, 2014

20 image-forensics-pg14.key - October 9, 2014

21 image-forensics-pg14.key - October 9, 2014

22 image-forensics-pg14.key - October 9, 2014

23 image-forensics-pg14.key - October 9, 2014

24 image-forensics-pg14.key - October 9, 2014

25 image-forensics-pg14.key - October 9, 2014

26 image-forensics-pg14.key - October 9, 2014

27 image-forensics-pg14.key - October 9, 2014

28 image-forensics-pg14.key - October 9, 2014

29 image-forensics-pg14.key - October 9, 2014

30 image-forensics-pg14.key - October 9, 2014

31 image-forensics-pg14.key - October 9, 2014

32 image-forensics-pg14.key - October 9, 2014

33 image-forensics-pg14.key - October 9, 2014

34 image-forensics-pg14.key - October 9, 2014

35 image-forensics-pg14.key - October 9, 2014

Light in front of camera Light behind camera 36 image-forensics-pg14.key - October 9, 2014

37 image-forensics-pg14.key - October 9, 2014

38 image-forensics-pg14.key - October 9, 2014

39 image-forensics-pg14.key - October 9, 2014

40 image-forensics-pg14.key - October 9, 2014

Shading Constraints (b) (c) 41 image-forensics-pg14.key - October 9, 2014

Shading Constraints 1 a b 2 c 3 4 6 5 d 7 1 2 3 5 6 7 4 a b c d 42 image-forensics-pg14.key - October 9, 2014

Shading Constraints a 1 c b 2 3 6 5 d 8 7 4 8 a b c d 43 image-forensics-pg14.key - October 9, 2014

Shading Constraints 2 1 2 1 3 a 4 c 4 b 3 a b c 44 image-forensics-pg14.key - October 9, 2014

Motion in Video 45-1 image-forensics-pg14.key - October 9, 2014

Motion in Video 45-2 image-forensics-pg14.key - October 9, 2014

Parabolic Motion in World (Still Camera) p = p 0 + t v 0 + 1 2 ( t )2 g p = c + (q c) 2 1..n p Solve for: v 0 g q c 46 image-forensics-pg14.key - October 9, 2014

Matching observed motion y x z 47 image-forensics-pg14.key - October 9, 2014

http:// www.youtube.com/ watch?v=wbah52ji3so 48-1 image-forensics-pg14.key - October 9, 2014

http:// www.youtube.com/ watch?v=wbah52ji3so 48-2 image-forensics-pg14.key - October 9, 2014

49-1 image-forensics-pg14.key - October 9, 2014

49-2 image-forensics-pg14.key - October 9, 2014

y x z 50-1 image-forensics-pg14.key - October 9, 2014

y x z 50-2 image-forensics-pg14.key - October 9, 2014

Parabolic Motion in World (Moving Camera) p = p 0 + t v 0 + 1 2 ( t )2 g p = c + (q c) 2 1..n p Solve for: v 0 g q Track camera motion c 51 image-forensics-pg14.key - October 9, 2014

52-1 image-forensics-pg14.key - October 9, 2014

52-2 image-forensics-pg14.key - October 9, 2014

y z x 53 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry Linear perspective image Mirror Image View Object Reflection of object 54 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry 55-1 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry 55-2 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry Mirror-Parallel View Object Point Reflected Point n COP Mirror 56 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry Mirror-Parallel View Object Reflection Mirror 57 image-forensics-pg14.key - October 9, 2014

Basic Mirror Geometry Bundle of parallel lines Mirror-Parallel View Object Reflection In original image they must converge to a common vanishing point.! (Possibly at infinity) Mirror 58 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Real Photograph 59 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Real Photograph v 60 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Altered Photograph 61 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Altered Photograph 62 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Altered Photograph 63 image-forensics-pg14.key - October 9, 2014

Reflection Vanishing Point Altered Photograph 64 image-forensics-pg14.key - October 9, 2014

Examples 65 image-forensics-pg14.key - October 9, 2014

Examples 66-1 image-forensics-pg14.key - October 9, 2014

Examples 66-2 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points 71-1 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points v 1 v 2 v 3 71-2 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points v 1 v 2 C v 1 Image Plane v 3 v 2 72 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points v 1 v 2 C v 1 Image Plane v 3 v 2 73-1 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points v 1 v 2 v 1 C Image Plane v 2 v 3 (C V 1 ) (C V 2 )=0 73-2 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points v 1 v 2 v 1 C Image Plane v 2 v 3 (C V 1 ) (C V 2 )=0 (C V 2 ) (C V 3 )=0 (C V 3 ) (C V 1 )=0 74 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points V 1 V 2 C v 1 Image Plane v 2 C V 3 (C V 1 ) (C V 2 )=0 (C V 2 ) (C V 3 )=0 (C V 3 ) (C V 1 )=0 75 image-forensics-pg14.key - October 9, 2014

Center of Projection COP determined by 3 orthogonal vanishing points System of quadratic equations (C V 1 ) (C V 2 )=0 (C V 2 ) (C V 3 )=0 (C V 3 ) (C V 1 )=0 Easy to solve by change of variables 76 image-forensics-pg14.key - October 9, 2014

Center of Projection Building and other structures Reflectors with rectangular frames!! Frames: two orthogonal vanishing points Reflected features: third vanishing point Compare COP from separate elements in the image 77 image-forensics-pg14.key - October 9, 2014

Center of Projection Computation is unstable Step 1: intersect [nearly parallel] lines Step 2: intersect spheres 78-1 image-forensics-pg14.key - October 9, 2014

Center of Projection Computation is unstable Step 1: intersect [nearly parallel] lines Step 2: intersect spheres 78-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Computation is unstable Step 1: intersect [nearly parallel] lines Step 2: intersect spheres 79-1 image-forensics-pg14.key - October 9, 2014

Center of Projection Computation is unstable Step 1: intersect [nearly parallel] lines Step 2: intersect spheres 79-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Computation is unstable Step 1: intersect [nearly parallel] lines Step 2: intersect spheres Instability squared 79-3 image-forensics-pg14.key - October 9, 2014

Center of Projection Error sources: Image resolution User pointing accuracy Features from different perspectives COP calculation magnifies error Structure in instability 80-1 image-forensics-pg14.key - October 9, 2014

Center of Projection Error sources: Image resolution User pointing accuracy Features from different perspectives COP calculation magnifies error Structure in instability Specify regions, not points 80-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Real Photograph 81-1 image-forensics-pg14.key - October 9, 2014

Center of Projection Real Photograph 81-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Real Photograph 81-3 image-forensics-pg14.key - October 9, 2014

Center of Projection Altered Photograph 82-1 image-forensics-pg14.key - October 9, 2014

Center of Projection Altered Photograph 82-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Altered Photograph 82-3 image-forensics-pg14.key - October 9, 2014

Center of Projection 83-1 image-forensics-pg14.key - October 9, 2014

Center of Projection 83-2 image-forensics-pg14.key - October 9, 2014

Center of Projection Real Photograph Altered Photograph 83-3 image-forensics-pg14.key - October 9, 2014

CoP from Faces Work in progress 84 image-forensics-pg14.key - October 9, 2014

CoP from Faces Work in progress 85 image-forensics-pg14.key - October 9, 2014

CoP from Faces Work in progress 86 image-forensics-pg14.key - October 9, 2014

CoP from Faces Work in progress 87 image-forensics-pg14.key - October 9, 2014

Summary Geometric Image Forensics Human annotation Computer analysis Part of analysis toolbox Not always applicable Together make forgery more difficult Constrain image content 88 image-forensics-pg14.key - October 9, 2014

Relevant Papers Eric Kee, James F. O'Brien, and Hany Farid. Exposing Photo Manipulation from Shadows and Shading. ACM Transactions on Graphics, too appear. Presented at SIGGRAPH 2014. http://graphics.berkeley.edu/papers/kee-epm-2014-xx! Eric Kee, James F. O'Brien, and Hany Farid. Exposing Photo Manipulation with Inconsistent Shadows. ACM Transactions on Graphics, 32(4):28:1 12, September 2013. Presented at SIGGRAPH 2013. http://graphics.berkeley.edu/papers/kee-epm-2013-09! Valentina Conotter, James F. O'Brien, and Hany Farid. Exposing Digital Forgeries in Ballistic Motion. IEEE Transactions on Information Forensics and Security, 7(1):283 296, February 2012. http://graphics.berkeley.edu/papers/conotter-edf-2012-02! James F. O'Brien and Hany Farid. Exposing Photo Manipulation with Inconsistent Reflections. ACM Transactions on Graphics, 31(1):4:1 11, January 2012. Presented at SIGGRAPH 2012. http://graphics.berkeley.edu/papers/obrien-epm-2012-01 89 image-forensics-pg14.key - October 9, 2014

Thank You 90 image-forensics-pg14.key - October 9, 2014