At the GTC 2019 conference, Nvidia showed the capabilities of the GauGAN neural network (the name is consonant with Gauguin), which is able to transform custom scribbles into photorealistic masterpieces. So far, however, the neural network is able to work only with landscapes, but the developers promise to expand its capabilities.
Working with GauGAN is easy. Using a couple of three simple tools, draw a circle, for example, indicating that there should be water here. The neural network immediately turns the schematic drawing into a lake image. How it works, you can see in the video:
To train GauGAN, neural networks fed about a million landscape photos from Flickr. In this case, the developers emphasize that the system does not just insert pieces of rocks, fields, sky, and so on, but generates absolutely unique images. This takes into account all sorts of little things. For example, if there is a pond in front of a tree, then the tree will be reflected in it. It is enough to change the season, and the bushes will be left without leaves.
The results so far can not be called perfect. As noted in TechCrunch, on closer inspection, flaws are visible, especially on the “gluing” between objects. However, Nvidia promises to improve the algorithm.