MAS.S66 F’22 | Computer Visions: Generative Methods for Creative Applications
Machine learning is transforming our reality. Today’s models deeply learn about their input domains and reach beyond-human capabilities on a growing list of tasks. While learning to predict, many ML models construct a view of their input worlds that allow them to also generate new input data – they are generative models. In this fast-paced class students will use generative ML models to “paint” and “sketch”, “write” a poem, transfer visual artistic style, hallucinate structure out of noise and “compose” music. The class will focus on hands-on practical programmatic methods to implement computer visions in Python. We will host REAL artists / researchers that will share REAL methods, we will not dive into high-level academic overview. Students will learn how machine learning models are used to generate information in multiple media (text, sound, picture), and by the end of the course will be able to apply these tools to their own domain of interest. The course is designed for persons without prior experience in deep learning, but it can benefit advanced students looking for an overview of the latest generative methods.
Location: E15-359 (some sessions will be remote)
Time: Tuesdays, 1-3pm
Instructor: Dr. Roy Shilkrot
Syllabus | Topics Covered
- Students with use online materials to get familiar with the mechanics and concepts in Machine Learning and Deep Learning (model, classification, regression, supervision, train-test, loss, overfitting, regularization, linear models, neural nets, back-propagation optimization, convolution, recurrence, attention, transformers). While it will be mentioned in class, we will not cover them in depth.
- Working environments for generative media, e.g. Google Colab, Jupyter Notebooks, Gradio APIs, Hugginface Spaces, etc.
- Significant ML models and architectures of interest: GANs (pix2pixHD, Style), Latent Diffusion Models (Stable, Disco, etc.), DALL-E, Transformers, GLIDE, CLIP, GPT, and more.
We will host a number of speakers from the world of generative media, all active artists, researchers and leaders that will share the actual methods they use for their work (including links to code and examples).
Research Scientist @ Google
PhD Researcher @ TAU and Nvidia
Generative AI Artist, Software engineer
Founder @ Stability.ai
AI Artist, Research Engineer @ Apple
Generative AI Artist, Data Scientist
Generative AI Artist
|9/13||Introduction and expositions||HW1: Hello generative world!|
|9/20||Speaker: Matti Mariansky|
The world of Generative Art, NFTs, curation and outlets.
|9/27||Speaker: Jonathan Whitaker|
Exploring the Generative Landscape.
|HW2: Loss functions|
|10/4||Speaker: Emad Mostaque|
Building Stability.AI and Stable Diffusion.
In-class: Assignment, GANs, CLIP
|10/11||No class!||HW3: GANs|
|10/18||Speaker: Doron Adler|
Practicing AI art, contemporary methods.
|10/25||Speaker: Prof. Dav Novy|
In-class: assignment, Diffusion models
|11/1||Speaker: Nora Wixom|
Illusory Truth: Bias in CV/ML and Generative Systems
An Artistic approach to Generative AI
|HW5: StyleGAN Playground|
|11/15||Speaker: Yuval Gur|
Generative music algorithms.
In-class: Assignment, text generation models
|11/22||Speaker: Rinon Gal|
StyleGAN to the max, domain adaptation, object insertion.
|HW6: Music generators|
Speaker: David Ha (meeting 9am)
|12/6||In-class review, final projects workshop||Final projects|
|12/13||Final projects review||Final projects|
Class schedule is subject to change throughout the semester.
This class will have bi-weekly take-home work, and a final project. The final grade will consist of 50% take-home work grades and 50% final project grade.
- Andrej Karpathy’s Tutorials
Cover image created by Stable Diffusion, prompt: “robotic sheep in pasture, by Rob Gonsalves”