MAS.S66 F’22 | Computer Visions: Generative Methods for Creative Applications


Machine learning is transforming our reality. Today’s models deeply learn about their input domains and reach beyond-human capabilities on a growing list of tasks. While learning to predict, many ML models construct a view of their input worlds that allow them to also generate new input data – they are generative models. In this fast-paced class students will use generative ML models to “paint” and “sketch”, “write” a poem, transfer visual artistic style, hallucinate structure out of noise and “compose” music. The class will focus on hands-on practical programmatic methods to implement computer visions in Python. We will host REAL artists / researchers that will share REAL methods, we will not dive into high-level academic overview. Students will learn how machine learning models are used to generate information in multiple media (text, sound, picture), and by the end of the course will be able to apply these tools to their own domain of interest. The course is designed for persons without prior experience in deep learning, but it can benefit advanced students looking for an overview of the latest generative methods.

Location: E15-359 (some sessions will be remote)
Time: Tuesdays, 1-3pm
Instructor: Dr. Roy Shilkrot
Units: 0-6-0

Syllabus | Topics Covered

  • Students with use online materials to get familiar with the mechanics and concepts in Machine Learning and Deep Learning (model, classification, regression, supervision, train-test, loss, overfitting, regularization, linear models, neural nets, back-propagation optimization, convolution, recurrence, attention, transformers). While it will be mentioned in class, we will not cover them in depth.
  • Working environments for generative media, e.g. Google Colab, Jupyter Notebooks, Gradio APIs, Hugginface Spaces, etc.
  • Significant ML models and architectures of interest: GANs (pix2pixHD, Style), Latent Diffusion Models (Stable, Disco, etc.), DALL-E, Transformers, GLIDE, CLIP, GPT, and more.


We will host a number of speakers from the world of generative media, all active artists, researchers and leaders that will share the actual methods they use for their work (including links to code and examples).

David Ha

Research Scientist @ Google

Rinon Gal

PhD Researcher @ TAU and Nvidia

Doron Adler

Generative AI Artist, Software engineer

Emad Mostaque

Founder @

Nora Wixom

AI Artist, Research Engineer @ Apple

Jonathan Whitaker

Generative AI Artist, Data Scientist

Tom Clive

Generative AI Artist

Matty Mariansky

Generative AI Artist

Yuval Gur

Generative AI Composer and Film Maker

Dan Novy

Professor of Emerging Media Arts @ UNL

Apolinario Passos

ML Art Engineer @ Hugging Face 🤗


Date Topic Assignment Links
9/13 Introduction and expositions HW1: Hello generative world!  
9/20 Speaker: Matti Mariansky
The world of Generative Art, NFTs, curation and outlets.
9/27 Speaker: Jonathan Whitaker
Exploring the Generative Landscape.
HW2: Loss functions  
10/4 Speaker: Emad Mostaque
Building Stability.AI and Stable Diffusion.

In-class: Assignment, GANs, CLIP
10/11 No class! HW3: GANs  
10/18 Speaker: Doron Adler
Practicing AI art, contemporary methods.
10/25 Speaker: Prof. Dav Novy
Generative storytelling.

In-class: assignment, Diffusion models
HW4: Diffusers  
11/1 Speaker: Nora Wixom
Illusory Truth: Bias in CV/ML and Generative Systems
11/8 Speaker: TomLikesRobots
An Artistic approach to Generative AI
HW5: StyleGAN Playground  
11/15 Speaker: Yuval Gur
Generative music algorithms.

In-class: Assignment, text generation models
11/22 Speaker: Rinon Gal
StyleGAN to the max, domain adaptation, object insertion.
Final Projects  

Speaker: David Ha (meeting 9am)

Contemporary work on Generative Media at Stability.AI

Final projects  
12/6 Speaker: Apolinário Passos from HuggingFace Final projects  
12/13 Class and final projects review Final projects  

Class schedule is subject to change throughout the semester.


This class will have bi-weekly take-home work, and a final project. The final grade will consist of 50% take-home work grades and 50% final project grade.


Cover image created by Stable Diffusion, prompt: “robotic sheep in pasture, by Rob Gonsalves”