Image-to-text AI as Creative Technique for AI Artists
Feb 2023·byAndrei Kovalev
If you read my previous guides, you know my prompting strategy is highly minimalist. I keep my prompts as simple as possible. And when going for a complex one, I will comprise it of smaller parts that I know work by themselves. It allows control and repeatability of the outcome.
"I am always doing that which I cannot do, in order that I may learn how to do it."
— Pablo Picasso
The idea is to check that Midjourney can interpret specific parts of the final complex prompt on their own in a given style.
However, in this study, the prompts used to achieve the final results are the exact opposite of my approach. They are wordy, seem half-random, and some parts don't make sense—even as stand-alone prompts.
The method we will dive into today can be such a fantastic creative tool that I, for once, will step away from my cozy and controlled prompt strategy. (≧∀≦)
Image to Text to Image
Original photograph
A cover photo I made for RBC magazine some years ago. The previous image is its image-to-text interpretation made by CLIP Interrogator 2
It's difficult to believe, but despite the inflated and often absurd prompts, their interpretation is stunningly close to the original. It seems like an AI knows how to talk to another AI... |｡_｡|
Original generation
Interpretation
CLIP Interrogator's primary goal is Stable Diffusion prompts. However, the experience of using its prompts in Midjourney is simply fantastic! It gives you a whole new perspective on creating art in MJ and launches a creative journey to chaos, full of unexpected insights and mind-blowing visuals.
Original generation
Interpretation
Original generation
Interpretation
CLIP Interrogator has a very straightforward interface, works fast (all samples in this study were interpreted in between 30 to 120 seconds each), and has only four settings.
Different modes render different results, and each is worth experimenting with. For this study, however, I primarily used the default Best mode.
Original generation
Best mode at 16 flavors
Classic mode
Fast mode
Original generation
Interpretation
Original generation
Interpretation
New Perspective
I started the previous chapter with an example from my photographic portfolio. Of course, it's not the only one. :) For visual artists with a portfolio of past works, CLIP Interrogator is an excellent instrument to re-imagine their work and look at it with fresh eyes.
Of course, I couldn't help but start by offering CLIP Interrogator the portrait of Francis D. for interpretation.
Original photograph
Francis D. (2017)
Interpretation
Original photograph
Patrice Piveteau from Cognac Frapin (2018)
Interpretation
Original photo
Danish performance artist Christian Falsnaes for Baku magazine (2016)
Interpretation
Original photograph
Polar Stalker. A photograph from my photography course (2018)
Interpretation
Original photograph
Key visual for DiveBar Tbilisi promotion (2018)
Interpretation
Apart from looking at your own work through different optics, I easily imagine text-to-image helping artists overcome creative blocks. How about not just getting a prompt from an image but developing a whole story from one picture?
Image to Text to Story to Image
Undoubtedly, you've heard about ChatGPT . So I'll skip the introduction and cut to the chase: I used it to generate a sensible short story from the CLIP Interrogator's results. Here is what I asked:
Please, rewrite the following gibberish into a sensible text. Use all the words from the original.
And this is what happened:
Original generation
Clip Interrogator interpretation
ChatGPT re-interpretation
Original generation
Clip Interrogator interpretation
ChatGPT re-interpretation
But how about we remove the limitation and let ChatGPT run wild(er)?
Please, rewrite the following gibberish into a short story (think of it as a movie still). Use all the words from the original.
Original genertaion
Clip Interrogator interpretation
ChatGPT re-interpretation
Conclusion
Image-to-text AI tools may be great for analyzing other peoples' generations to figure out their prompts, overcoming a creative block, or finding a new perspective on your past work. And for sure—for having pure artistic fun!
Original photo
Five generation of Georgian Women. Personal project (2016)
Interpretation
Happy midjourneys! — Andrei
About the author
Andrei KovalevLibrarian
I am a visual artist with 17 years of experience in portrait photography, film, mask design, and education. As Midjourney AI and its analogs are revolutionizing the world of art and our creativity itself, I am overwhelmingly excited to explore the limits of these fantastic new tools and share my findings with the community.
