{"id":26410,"date":"2021-06-07T06:00:00","date_gmt":"2021-06-07T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=26410"},"modified":"2021-06-08T12:07:57","modified_gmt":"2021-06-08T19:07:57","slug":"applying-gans-to-image-generation-tasks","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2021\/06\/07\/applying-gans-to-image-generation-tasks\/","title":{"rendered":"Applying GANs to Image Generation Tasks"},"content":{"rendered":"\n

This article overviews StyleGAN2 application to image generation task and is based on MobiDev\u2019s logotype synthesis research<\/a>. <\/p>\n\n\n\n

When it comes to powerful generative models for image synthesis, the most commonly mentioned are StyleGAN<\/a> and its updated version StyleGAN2<\/a>. These models created by Nvidia Labs are able to solve image generation tasks and produce remarkably high fidelity images of non-existent people, animals, landscapes, and other objects given an appropriate training dataset<\/strong>.<\/p>\n\n\n\n

StyleGAN, just like the other GAN architectures, features two sub-networks: Discriminator and Generator. During the training, the Generator is tasked with producing synthetic images while the Discriminator is trained to differentiate between the fakes from Generator and the real images.<\/p>\n\n\n\n

The first iteration of StyleGAN appeared in 2019. It was applied to produce fake faces with high detailization and natural appearance with resolutions up 1024×1024, not previously achieved by other similar models. However, some AI-generated faces had artifacts, so Nvidia Labs decided to improve the model and presented StyleGAN2. One of the main issues of  the original StyleGAN were blob-like artifacts that looked like drops of water. <\/p>\n\n\n\n

\"\"
Figure 1. A blob-like artifact example in the image generated by StyleGan. <\/strong>Source<\/a><\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Figure 2.<\/strong> AdaIN based Style Transfer Network Source<\/a> <\/figcaption><\/figure><\/div>\n\n\n\n

According to the StyleGAN2 paper, this problem is related to the instance normalization operation applied in AdaIN layers. The basic purpose of AdaIN is to fuse together two images: one containing style (style is a property that is present throughout all the image) and one containing the content of the image. Part of the crucial information of the inputs is being lost during this process. Why is this happening?           <\/p>\n\n\n\n

AdaIN works separately with the variance and mean of individual feature maps. The feature map is an intermediate representation of an image within the neural network. Feature maps carry the information regarding the image that is being generated and normalize their values. <\/p>\n\n\n\n

As each feature map is normalized individually, information about the relative feature map values in regard to each other is lost in this process. As a result, the generator network produces the blob artifact with a strong signal within the image, effectively bypassing the AdaIN in an attempt to preserve the relative information that the normalization destroys.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n
\"\"
Figure 3. StyleGAN synthesis network architecture redesign. Source<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

The development of StyleGAN2 was related not only to addressing the issue with AdaIN, there were other improvements in comparison with StyleGAN:<\/p>\n\n\n\n