DSC160 Data Science and the Arts - Final Project - Generative Arts - Spring 2020
Project Team Members:
- Yicen Ma, [email protected]
- Xingyu Jiang, [email protected]
- Chang Yuan, [email protected]
- Michael Kusnadi, [email protected]
- Kaixin Huang, [email protected]
For the final project, we will find if there exists any differences or similarities between computer generated Van Gogh paintings and his actual work by using deep photo transfer and style transfer. This means that we will find some real images which has similar objects as some Van Gogh's paintings. Then we will put those real images into models and find the similarties or differences between computer generated painting and his actual work.
We will divide 2 groups of real image in this project. One group we will put those real images into both deep photo transfer and style transfer. Another group we will just put real images into the style transfer. The reason for us to do that is we find out some real images that has pretty similar object and environment with Van Gogh's real painting. But their main color is different. So we use some strong color image as style and change their color by using deep photo transfer. Then putting them into style transfer and compare the difference and similarities between generated paintings and his actual work. Another group we will just put those real images into style transfer. And compare the differences and similarities between generated paintings and his actual work.
For the result, I hope the deep photo transfer, it can successfully change the color from the style real image into the content real image. For the style transfer, it can accurately get Van Gogh's painting skills and changing our real images by using those painting skills. By making some images “van Gogh” like, we can better understand van Gogh’s artistic styles and how he would draw the objects that he had never tried to draw. And we hope our generated results from both groups can be some paintings that will let people think they are looking actual paintings from Van Gogh.
Reference:
-
https://www.theverge.com/2020/4/2/21204498/art-transfer-google-artists-style-photos. This is talking about using the art transfer tech by Google to let you apply famous artists’ styles to your own works
-
https://medium.com/tensorflow/neural-style-transfer-creating-art-with-deep-learning-using-tf-keras-and-eager-execution-7d541ac31398 This gives us some idea on Neural Style Transfer: Creating Art with Deep Learning using tf.keras and eager execution
-
http://headforart.com/2016/12/16/how-artists-use-colour/ This is talking about the importance of color for the artists. And this gives us the idea that we can change the color of those artists’ artworks and then put them into our Van-Gogh model for transferring to Van Gogh art style. And we want to see the influence on changing color on the original artworks during our process of transferring to Van Gogh style. (see whether it will generate different artworks after transfering)
-
https://github.com/simulacre7/tensorflow-IPythonNotebook/blob/master/neural-style/neural_style.ipynb This is a github project for combing one paint style into another artwork. We believe we will do the similar steps as this project.
Image Style Transfer Using Convolutional Neural Networks (CNN). This method inverts the image representation based on CNN and by using a texture model, it transfers the style of an image to another with an adjustable weight ratio of the two inputs that can affect the representation of the result.
Image Style Transfer Using Convolutional Neural Networks: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf
Deep Photo Style Transfer, a method to apply the style of an image onto a content image but in realistic settings. With the use of segmentation masks, deep photo style transfer can apply style onto the specified objects instead of on random objects to make the result look realistic.
Code: https://github.com/LouieYang/deep-photo-styletransfer-tf
Deep Photo Style Transfer - https://arxiv.org/pdf/1703.07511.pdf
Artist Info
Van Gogh was a Dutch Post-Impressionist artist in the 1900 century. He was famed for his bold, dramatic brush strokes which expressed emotion and added a feeling of movement to his works.His emerging style saw him emotionally reacting to subjects through his use of color and brush work. He deliberately used colors to capture mood by using the complementary color contrasts and a bolder composition in his advanced years.
Images and oil painting: We selected all the images and oil paintings of landscapes.
Irises
The oil painting Irises was painted in 1889 inSaint Remy de Provence (France) and it showed the beauty of irises from a special point of view. Van Gogh used a high concentration of green and blue in this painting. It shows the full of softness and lightness. Thoses irises are full of life without tragedy. Similar Composition image: real image with irises flower.
Data Source:https://www.wikiart.org/en/vincent-van-gogh/irises-1889
https://longislandnatives.com/products/iris-versicolor-blue-flag-iris
Road with Cypress
The oil painting Road with Cypress was painted in 1890 in Saint Remy de Provence (France) and it showed a tall cypress tree in a country side. The sky in this painting shares the similar sky in one of the famous paintings The Starry Night. Also, the cypress was always presented in Van Gogh’s paintings in the advanced years. The Cypress dominated the painting and dwarfed elements around it. Similar Composition image: it also has a road with a cypress. However, the land was covered by snow, because we want to use the deep photo transfer to add the color on it. Deep photo transfer image: it is a landscape of a wheatfield in yellow and cloudy in the sky.
Data Source:https://www.wikiart.org/en/vincent-van-gogh/road-with-cypresses-1890
https://commons.wikimedia.org/wiki/File:Wheatfield_in_Ottawa.jpg
Summer Evening, Wheatfield with Setting sun
Van Gogh painted the oil painting Summer Evening, Wheatfield with Setting sun in 1888 in France. There were vast tracts of wheatfield in the painting with the village and sunset as the background. He used different linewidth to represent each wheat from near to far. Similar Composition image: it is a landscape of a wheatfield in yellow and cloudy in the sky. There are also three buildings in shadow in the distance.
Data Source: https://www.wikiart.org/en/vincent-van-gogh/summer-evening-wheatfield-with-setting-sun-1888
https://commons.wikimedia.org/wiki/File:Wheatfield_in_Ottawa.jpg
Still Life - Vase with Fifteen Sunflowers
The oil painting Still Life - Vase with Fifteen Sunflowers was painted in 1888 in France. Van Gogh painted a sunflower series and there were only some minor differences between each oil painting. We selected the all yellow color in the background, vase and sunflower itself with green in stem. But he used different brightness of yellow to distinguish the edges. Similar Composition image: a landscape of polar forest fulfilled with fallen leaves in orange color.
Data Source: https://www.wikiart.org/en/vincent-van-gogh/still-life-vase-with-fifteen-sunflowers-1888-1
https://www.52112.com/pic/143913.html
The Church at Auver
When Van Gogh was at Auver, he saw a church and painted in 1890 at Auver in France. It reminded him back to the landscape of his childhood with small houses with thatched roofs. This time, he didn't paint the sky as dart blue color rather than the style in The Starry Night. It brought a peaceful emotion from this painting. Similar Composition image: This is the church prototype from Van Gogh’s painting at Auver. In 1890, Van Gogh saw the church and created The Church at Auver.
Data Source: https://www.wikiart.org/en/vincent-van-gogh/the-church-at-auvers-1890
https://www.wikiwand.com/en/The_Church_at_Auvers
Description of deep photo style transfer: The execution of the deep photo style transfer method is a little different than usual, in that to run this code, you have to run the specified command line in the console. The .py files necessary to run the command line are listed above, and the exact command lines are given in the Deep_Photos_Style_Transfer.ipynb.
-
This notebook contains the necessary commands needed to run the deep photo style transfer method onto the specified content image and the style image.
-
This py file uses the Matting Laplacian to constrain the transformation from the input to the output to be locally affine in colorspace.
-
This py file contains all the argument input options such as the training optimizer, the weight regularization, and whether to apply the smooth local affine. Once all the necessary options are specified, this deep_photostyle py file will execute photo_style.py, closed_form_matting.py, and smooth_local_affine.py if specified.
-
This py file calls the loss function that tries to minimize the content loss and style loss. This enables the style transfer affect, which is overlaying the style of the style image onto the content image. In addition, the photorealism effect can be augmented by incorporating segmentation masks by calling the stylized function. This helps to specify which style certain objects should contain instead of randomly over-laced with random colors.
-
This py file enables the reconstructed image to be represented by locally affine color transformations of the input to prevent distortions.
Description of neural style transfer: This method transfers the style of images using Convolutional Neural Networks (CNN). The notebook from the link above contains two parts: data scraping and preprocessing, and execution of style transfer. For the first part of the notebook, we scraped the images we need from wikiart. Then, along with the content images we had locally, we scaled and cropped them so that the content images and style images are in the same size for style transfer (we did this for each set of content image and style image, and for two sets of them we did the style transfer on the images we got from deep photo transfer). We used gpu to make the images have higher resolutions. After transforming them to pytorch tensors, we are done with the first part. For the second part, we used the pretrained vgg19 model to make the style transfer while tracking the style losses and content losses. And by trying different combinations of style weights and content weights, we choosed the best resulting images as our final results.
The purpose of this project is to use deep photo style transfer and image style transfer using convolutional neural networks to apply Van Gogh’s art style to images or photographs of similar composition. The results show that it is important to choose an appropriate picture or canvas to do the style transfer on, this is because the style transfer we are implementing above only allows the replication of colors and grain, and not the details. The style transfer that we are implementing above does not have the ability to change details such as the line directions as well as the overall shape of the image. Although this is so, when given an appropriate image, the style transfer works well and it does replicate Van Gogh’s style to the given image.
This project is culturally innovative in the view that we are combining aspects of reality and paintings as one. The style transfer from paintings to photography is in itself a culturally innovative action as it combines two forms of artwork which is photography and painting. Furthermore we are able to see how Van Gogh’s artwork could translate in this day and age, this becomes an interesting view to see because we get to see what his artworks would look like in today’s environment and put it into today’s culture which makes this project innovative.
Compared to traditional paintings that are done or even photography, this generative computational approach of style transfer has to have a basis from which it takes its style. So therefore compared to traditional methods where we are creating originally, this approach is more of synthesizing what already exists and renewing it. This is an approach that is debatable and there are also sides which say that it might not be original. With traditional production we also use a more organic way of measuring and creating by using our own senses and feelings to decide. Our approach uses a very calculative method and has no feelings that are used in the process, everything is numerical except for the fact that we can tweak our results and parameters to produce what we want, we also choose our canvas and this is where the originality comes from.
Style and art go hand in hand with each other, and every art piece has its own style. Each artist and place has their own specific art style that they usually gravitate towards. Because of this exploring style transfer is something that is socially and culturally significant. From this we can further pinpoint certain features that come from a certain artstyle. With this style transfer, culturally, we are able to intermingle many different settings in many different geographical locations that are culturally meaningful and see it in a new style such as Van Gogh’s painting we have done above. Socially, this idea of style transfer will bring about new methods of art and new art pieces will arise from it. By taking reference to an artist’s style though it might be seen as plagiarism and could be ethically problematic as many argue that art should be original, but if we are able to solve these problems, style transfer would be able to open up many possibilities in both the world of art and data analytics.
This project could further be explored by combining or synthesizing two or more styles to create possibly a new art style; this can also work with combining artists within the same art genre. For example, it would be possible for us to create an art style that combines realism and impressionism and by combining them into one style and then transferring it as we have done in our project above, we will see the production of something culturally innovative as well as something that is original and interesting. In terms of extension, we could also find out how to change details so that the style could be applied to not just similar images but to any image, and this would make this project more adaptable and useful as well.
Chang Yuan: any code related work of neural style transfer and its results
Xingyu Jiang: idea thinking, data finding, proposal and first draft result part
Yicen Ma: idea generator, idea thinking, data finding, data part, proposal and final writing of result part
Michael Kusnadi: discussion part
Kaixin Huang: any code related work of deep photo style transfer and its results
Neural Style Transfer: We mainly used pytorch for this part of the code. And we used the pretrained vgg19 model as the Convolutional Neural Networks (CNN) for our work. Other than that, we used some other standard libraries such as numpy, matplotlib, beautifulsoup, etc. The code can be run on datahub and no other software is required.
For rapid iterative development which is crucial for deep photo style transfer, PYCUDA is recommended for installation. This enables NVIDIA GPU accelerated computing with Python.
https://www.theverge.com/2020/4/2/21204498/art-transfer-google-artists-style-photos.
http://headforart.com/2016/12/16/how-artists-use-colour/
https://github.com/LouieYang/deep-photo-styletransfer-tf