This is part three in the series about writing good VQGAN+CLIP (VQC) prompts. In the first I talked about style and media modifiers, in the second I covered artist and genre modifiers, and in this one I’m going to talk about environment modifiers – materials, colour and lighting – and how they effect the outcome.
As always, the techniques here are ones I’ve found that work for me. Other artists will do things differently and as you develop your own style you’ll no doubt do things differently too.
Prompts and modifiers
A surprising ability of VQC and other text-to-image AI models is that it’s only possible to specify what to generate but also how the image should look. This includes the media used, e.g. watercolour paints or Unreal Engine, the artistic style, such as Cubism or Jackson Pollock, and also the materials, colour palette and lighting. These last three I call environment modifiers, for want of a better word.
Environment modifiers are useful as they give much more fine-grained control over the content of your image, for example by changing the materials or lighting. It’s yet another tool for gaining creative control over the generation process and steering it in the direction you want.
Examples of environmental modifiers
This is by no means a definitive list and I'm sure you'll be able to think of more, but these are ones I find helpful and use regularly.
This is what the elements in your image are made of. Examples could be glass, plastic, concrete, wool etc. You can also use adjectives to describe the materials, such as polished wood, glowing glass, or textured stone.
When selecting materials, think about the effect you’re looking to achieve and choose combinations that might look interesting. It’s fun to specify contradictory materials and textures to see what happens, such as brutalist skyscrapers made of jelly, or leaves made of felt.
It’s also possible to specify a few different materials in one prompt. It’ll be fairly random as to what ends up going where, but you can control it to a certain extent through the wording of your prompt and the initial image you use.
In this example the combination of two contrasting materials has resulted in a much better image, even though nothing else has been changed.
In this example I’ve used a black and white picture of the Southbank Centre as the init image. I didn’t specify where to use the two materials, but it’s automatically made the windows out of glass and the walls concrete as Clip knows that’s how buildings are made.
By switching round the materials you can see that areas which used to be concrete are now glass and vice versa, although the effect is not always consistent.
It came as a surprise to me that it was possible to control the colour palette quite so granularly, but it works, and can make a big difference to your image. All you need to do is specify the colour of something and it’ll do its best to accommodate
It’s harder to make a completely monochrome image as the system really wants to add colour, but there are ways. The simplest is to add ‘black and white’ or ‘noir’, but other ways might be to add a black and white film such as ‘Ilford HP-5’ or use a negative weight, such as ‘colour: -1’ to suppress it.
It’s possible to control the light sources in your image by describing what you want to see in the prompt. This works both for the global environmental lighting and individual sources of light, for example by lighting something individually or making an element glow.
Clip understands instructions such as ‘surrounded by’, ‘in’, so you can use them and other similar ones to alter the background and theme of your image without affecting the main elements in your scene too much.
I hope you find the techniques I’ve pulled together over these articles useful and that they might be a good starting point for your own creations. I’m barely scratching the surface of what’s possible here – there’s so much more to discover.