The Threshold Between Human and Computational Creativity. Pindar Van Arman

Size: px

Start display at page:

Download "The Threshold Between Human and Computational Creativity. Pindar Van Arman"

Nathaniel Clarke
6 years ago
Views:

1 The Threshold Between Human and Computational Creativity Pindar Van Arman

2 One of Them is Human #1 Photo by Maiji Tammi that was recently shortlisted for the Taylor Wessing Prize. Notable in that prize requires sitting to be with a living subject - Erica, is a Japanese android.

3 What is it to be Human? Perhaps the best way to think of what makes a human being human is to look at something that seems almost human and subtract the difference. Nigel Warburton The Guardian

4 What is it to be Creative? Perhaps the best way to think of what makes a human being creative is to look at something that seems almost creative and subtract the difference. Nigel Warburton The Guardian

5 Answer... I realized early on that painting robots could help explore this question.

6 Average about 1 robot system every 2 years, though that is ramping up...

7 12 Years of trying to discover what makes me creative, by teaching as much creativity as I can to my robots.

8 Low Creative Spectrum High????? Technical Random Imitation Creation Art Creative Spectrum

9 Low Creative Spectrum High????? Technical Random Imitation Creation Art Creative Spectrum Connect the Dots Paint by Numbers Stroke Techniques

10 Low Creative Spectrum High????? Technical Random Imitation Creation Art Creative Spectrum Fill in Compositions Abstract Backgrounds Error Minimization with Feedback Loops

11 Low High????? Technical Random Imitation Creation Art Creative Spectrum Collaboration Image Enhancement Feedback Loops Combined Approach

12 Low Creative Spectrum High????? Technical Random Imitation Creation Art Creative Spectrum Artistic Direction Make all Aesthetic Decisions Meaningful Abstraction

13 Low Creative Spectrum High????? Technical Random Imitation Creation Art Creative Spectrum Is human creativity and art what is left?

14 My Current Process Starts with Style Painting 1. Take and Select Photograph 2. Analyze & Optimize Photograph 3. Style Transfer 4. Analyze & Optimize Style Transfer 5. Model Strokes from Historic DB 6. Paint with Feedback Loops

15 CloudPainter Process Content & Style Deep Learning Style Transfer Pastiche Brushstroke Modelling Completed Painting Database with Brushstroke Geometry Feedback Loops

16 CloudPainter Process Content & Style Deep Learning Style Transfer Pastiche Brushstroke Modelling 1: What to Paint 2: How to Paint Completed Painting Database with Brushstroke Geometry Feedback Loops

17 1: What to Paint + = 1. Get collection of images 2. Select most aesthetically pleasing 3. Balance contrast and color levels 4. Crop balanced composition 5. Create pastiche with Style Transfer

18 1: How to Paint Find lines in pastiche 2. Model lines against strokes in art database 3. Paint subset of strokes 4. Compare Painting to Style Transfer 5. Adjust and return to Step 2 6. Stop painting when image stops changing

19 1: How to Paint Model 2. Paint 3. Compare 4. Adjust

20 Problems - Lots of Them but Biggest are... Creativity is not a Two Phased Linear Process Deep Learning & Style Transfer Needs Context Machines are Emotionally Clueless

21 CloudPainter s creative process is pretty good but... 1: What to Paint 2: How to Paint

22 ...a better creative process would have feedback loops at every decision point, and ability to change direction of style, influence, and content mid-painting

Deep Learning & Style Transfer A Neural Algorithm of Artistic Style by Gatys, Eckers, and Bethge 2015.

23 Deep Learning & Style Transfer A Neural Algorithm of Artistic Style by Gatys, Eckers, and Bethge Style Transfer takes a source image and uses Convolutional Neural Nets to find patterns in it and then applies the patterns to a content image. The resulting image is sometimes called a pastiche.

24 Deep Learning & Style Transfer Style Transfer works great at capturing texture and color, but...

25 Deep Learning & Style Transfer it is completely unaware of context. In a Style Transfer of these two compositionally similar source and content image, the resulting pastiche appears to have had the texture and color applied indiscriminately. Context was ignored.

26 Deep Learning & Style Transfer Region-based Painting Style Transfer Shugo Yamaguchi, Takuya Kato, Tsukasa Fukusato, Chie Furusawa, and Shigeo Morishima of Waseda Universtity, 2015

Style Transfer to regions, the results become dramatically more

27 Deep Learning & Region-based Style Transfer Source Image Content Image Style Transfer Once you begin considering context and applying Style Transfer to regions, the results become dramatically more meaningful. Region Correspondence Maps (Context Maps) Region-based Style Transfer

28 Getting Context? Analysis of famous works is probably up to artists, but getting context from photos can be done either by Sliding over Image with OpenCV or with Convolutional Neural Nets, a similar deep learning technique to the one used in Style Transfer.? OpenCV vs. Deep Convolutional Neural Networks Weird thing is they work similarly...

29 Getting Context? OpenCV (Brute Force) Slide across overlapping tiles of image at multiple magnifications until clean face is detected.

30 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution 3: Downsample (Max Pooling) 4: Fully Connected Layer (Multilayer Perceptron) 5: Classification Max Pooling CIFAR Tiger Wolf Bridge Castle House Road Skyscraper Cloud Forest Mountain Plain Sea Camel Cattle...

31 Getting Context? Convolutional Neural Network 1: Break Image Up

32 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution

33 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution 3: Downsample (Max Pooling)

34 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution 3: Downsample (Max Pooling) Max Pooling

35 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution 3: Downsample (Max Pooling) 4: Fully Connected Layer (Multilayer Perceptron) Max Pooling

36 Getting Context? Convolutional Neural Network 1: Break Image Up 2: Convolution 3: Downsample (Max Pooling) 4: Fully Connected Layer (Multilayer Perceptron) 5: Classification Max Pooling CIFAR Tiger Wolf Bridge Castle House Road Skyscraper Cloud Forest Mountain Plain Sea Camel Cattle...

37 CIFAR-100 Bridge Mountain Sea Boy

38 1: Complex Feedback Loops & 2: Regional Style Transfer 3:... Nothing about anything described to this point is impossible for a machine. Which brings us to the fact that...

39 1: Complex Feedback Loops & 2: Regional Style Transfer 3: Machines are Clueless It goes without saying that machines don t have emotions, but why does this matter?

40 ?? Beyond Experimentation Machines can experiment and create random things, but it is unclear if they can evaluate their experimentation to determine if it produced meaningful art with emotional relevance...

41 ?? Beyond Experimentation Study after Velázquez's Portrait of Pope Self Portrait with Bandaged Ear, LLick and Lather, Janine Antoni, 1993 Innocent X, Francis Bacon, 1953 Vincent Van Gogh, 1889

Bengio, 2014 Study after Velázquez's Portrait of Pope Innocent X, Francis Bacon, 1953 CAN: Creative Adversarial

42 ?? Beyond Experimentation GANs and CANs Generative Adversarial Networks Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014 Study after Velázquez's Portrait of Pope Innocent X, Francis Bacon, 1953 CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norm Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, Marian Mazzone, 2017 r

43 ?? Generative Adversarial Network Generator Adjust Face Not Discriminator Training Data

44 ?? Generative Adversarial Network Generator Art Not Discriminator Training Data

45 ?? Creative Adversarial Network Generator Art Not Style Ambiguity Discriminator Training Data CANs are similar to GANs except they attempt to emphasize new styles.

46 ?? Creative Adversarial Network Generator? Art Not Discriminator Style Ambiguity Training Data Study after Velázquez's Portrait of Pope Innocent X, Francis Bacon, 1953

47 ?? Creative Adversarial Network Generator Study after Velázquez's Portrait of Pope Innocent X, Francis Bacon, 1953 Portrait of Pope Innocent X, Velázquez, 1650 Art Not Discriminator Style Ambiguity Training Data

48 ?? Could a robot create these? Study after Velázquez's Portrait of Pope Self Portrait with Bandaged Ear, LLick and Lather, Janine Antoni, 1993 Innocent X, Francis Bacon, 1953 Vincent Van Gogh, 1889

Artistic Image Colorization with Visual Generative Networks

Artistic Image Colorization with Visual Generative Networks Final report Yuting Sun ytsun@stanford.edu Yue Zhang zoezhang@stanford.edu Qingyang Liu qnliu@stanford.edu 1 Motivation Visual generative models,