VQGAN+CLIP — How does it work?

Early stages of training on the prompt “A high-tech outer circle with a low-tech inner filling trending on art station”
  1. What is VQGAN+CLIP
  2. Who made VQGAN+CLIP
  3. How does it work technically
  4. What is VQGAN
  5. What is CLIP
  6. How do VQGAN and CLIP work together
  7. What about the training data?
  8. Further reading and cool links

1. What is VQGAN+CLIP?

2. Who made VQGAN+CLIP

3. How does it work technically?

4. What is VQGAN?

  • a type of neural network architecture
  • VQGAN = Vector Quantized Generative Adversarial Network
  • was first proposed in the paper “Taming Transformers” by University Heidelberg (2020)
  • it combines convolutional neural networks (traditionally used for images) with Transformers (traditionally used for language)
  • it’s great for high-resolution images

5. What is CLIP?

  • a model trained to determine which caption from a set of captions best fits with a given image
  • CLIP = Contrastive Language–Image Pre-training
  • it also uses Transformers
  • proposed by OpenAI in Januar 2021
  • Paper: “Learning transferable visual models from natural language supervision”
  • Git Repository: https://github.com/openai/CLIP

6. How do VQGAN and CLIP work together

7. What about the training data?

8. Further reading and cool links

--

--

--

A mix of Frontend Development, Machine Learning, Musings about Creative AI and more

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Going through 10,000 pictures in 30 seconds

Flood Detection in Images using KERAS (A Tutorial on Transfer Learning/Fine-Tuning)

Better weakly supervised object detection by using absolutely wrong data with NDI-WSOD

What is Text-to-Speech (TTS): Initial Speech Synthesis Explained

Choosing A Classification Algorithm

Anomaly Detection

Start-off your ML journey with K-Nearest Neighbors!

Transfer Learning- An unorthodox explanation

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alexa Steinbrück

Alexa Steinbrück

A mix of Frontend Development, Machine Learning, Musings about Creative AI and more

More from Medium

Quantum Computing Spend to Reach $8.6 Billion

25 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧𝐬 𝐟𝐨𝐫 𝐓𝐡𝐞 𝐍𝐞𝐱𝐭 25 𝐘𝐞𝐚𝐫𝐬: 𝐓𝐞𝐜𝐡 & 𝐒𝐨𝐜𝐢𝐞𝐭𝐲

<The Plan Less Visited> Chi Keung Tang

These Bored Apes Do Not Exist

A GIF showing slight tweaks to four generated Bored Apes.