MAS.S66 F’22 | Computer Visions: Generative Methods for Creative Applications

About

Machine learning is transforming our reality. Today’s models deeply learn about their input domains and reach beyond-human capabilities on a growing list of tasks. While learning to predict, many ML models construct a view of their input worlds that allow them to also generate new input data – they are generative models. In this fast-paced class students will use generative ML models to “paint” and “sketch”, “write” a poem, transfer visual artistic style, hallucinate structure out of noise and “compose” music. The class will focus on hands-on practical programmatic methods to implement computer visions in Python. We will host REAL artists / researchers that will share REAL methods, we will not dive into high-level academic overview. Students will learn how machine learning models are used to generate information in multiple media (text, sound, picture), and by the end of the course will be able to apply these tools to their own domain of interest. The course is designed for persons without prior experience in deep learning, but it can benefit advanced students looking for an overview of the latest generative methods.

Location: E15-359 (some sessions will be remote)
Time: Tuesdays, 1-3pm
Instructor: Dr. Roy Shilkrot
Units: 0-6-0

Syllabus | Topics Covered

• Students with use online materials to get familiar with the mechanics and concepts in Machine Learning and Deep Learning (model, classification, regression, supervision, train-test, loss, overfitting, regularization, linear models, neural nets, back-propagation optimization, convolution, recurrence, attention, transformers). While it will be mentioned in class, we will not cover them in depth.
• Working environments for generative media, e.g. Google Colab, Jupyter Notebooks, Gradio APIs, Hugginface Spaces, etc.
• Significant ML models and architectures of interest: GANs (pix2pixHD, Style), Latent Diffusion Models (Stable, Disco, etc.), DALL-E, Transformers, GLIDE, CLIP, GPT, and more.

Speakers

We will host a number of speakers from the world of generative media, all active artists, researchers and leaders that will share the actual methods they use for their work (including links to code and examples).

Schedule

Date	Topic	Assignment	Links
9/13	Introduction and expositions	HW1: Hello generative world!
9/20	Speaker: Matti Mariansky The world of Generative Art, NFTs, curation and outlets.
9/27	Speaker: Jonathan Whitaker Exploring the Generative Landscape.	HW2: Loss functions
10/4	Speaker: Emad Mostaque Building Stability.AI and Stable Diffusion. In-class: Assignment, GANs, CLIP
10/11	No class!	HW3: GANs
10/18	Speaker: Doron Adler Practicing AI art, contemporary methods.
10/25	Speaker: Prof. Dav Novy Generative storytelling. In-class: assignment, Diffusion models	HW4: Diffusers
11/1	Speaker: Nora Wixom Illusory Truth: Bias in CV/ML and Generative Systems
11/8	Speaker: TomLikesRobots An Artistic approach to Generative AI	HW5: StyleGAN Playground
11/15	Speaker: Yuval Gur Generative music algorithms. In-class: Assignment, text generation models
11/22	Speaker: Rinon Gal StyleGAN to the max, domain adaptation, object insertion.	Final Projects
11/29	Speaker: David Ha (meeting 9am) Contemporary work on Generative Media at Stability.AI	Final projects
12/6	Speaker: Apolinário Passos from HuggingFace	Final projects
12/13	Class and final projects review	Final projects

Class schedule is subject to change throughout the semester.

Grading

This class will have bi-weekly take-home work, and a final project. The final grade will consist of 50% take-home work grades and 50% final project grade.

Resources

• https://pharmapsychotic.com/tools.html
• https://web.stanford.edu/class/cs25/
• https://www.kdnuggets.com/2022/08/free-ai-beginners-course.html
• https://d2l.ai/index.html
• https://ml4a.net/
• https://ali-design.github.io/deepcreativity/
• Andrej Karpathy’s Tutorials
• http://introtodeeplearning.com/
• https://huggingface.co/course/chapter1/1

Cover image created by Stable Diffusion, prompt: “robotic sheep in pasture, by Rob Gonsalves”