MAS.S66 F’22 | Computer Visions: Generative Methods for Creative Applications

About

Machine learning is transforming our reality. Today’s models deeply learn about their input domains and reach beyond-human capabilities on a growing list of tasks. While learning to predict, many ML models construct a view of their input worlds that allow them to also generate new input data – they are generative models. In this fast-paced class students will use generative ML models to “paint” and “sketch”, “write” a poem, transfer visual artistic style, hallucinate structure out of noise and “compose” music. The class will focus on hands-on practical programmatic methods to implement computer visions in Python. We will host REAL artists / researchers that will share REAL methods, we will not dive into high-level academic overview. Students will learn how machine learning models are used to generate information in multiple media (text, sound, picture), and by the end of the course will be able to apply these tools to their own domain of interest. The course is designed for persons without prior experience in deep learning, but it can benefit advanced students looking for an overview of the latest generative methods.

Location: E15-359 (some sessions will be remote)
Time: Tuesdays, 1-3pm
Instructor: Dr. Roy Shilkrot
Units: 0-6-0

Syllabus | Topics Covered

  • Students with use online materials to get familiar with the mechanics and concepts in Machine Learning and Deep Learning (model, classification, regression, supervision, train-test, loss, overfitting, regularization, linear models, neural nets, back-propagation optimization, convolution, recurrence, attention, transformers). While it will be mentioned in class, we will not cover them in depth.
  • Working environments for generative media, e.g. Google Colab, Jupyter Notebooks, Gradio APIs, Hugginface Spaces, etc.
  • Significant ML models and architectures of interest: GANs (pix2pixHD, Style), Latent Diffusion Models (Stable, Disco, etc.), DALL-E, Transformers, GLIDE, CLIP, GPT, and more.

Speakers

We will host a number of speakers from the world of generative media, all active artists, researchers and leaders that will share the actual methods they use for their work (including links to code and examples).

David Ha

Research Scientist @ Google

Rinon Gal

PhD Researcher @ TAU and Nvidia

Doron Adler

Generative AI Artist, Software engineer

Emad Mostaque

Founder @ Stability.ai

Nora Wixom

AI Artist, Research Engineer @ Apple

Jonathan Whitaker

Generative AI Artist, Data Scientist

Tom Clive

Generative AI Artist

Matty Mariansky

Generative AI Artist

Yuval Gur

Generative AI Composer and Film Maker

Dan Novy

Professor of Emerging Media Arts @ UNL

Schedule

DateTopicAssignmentLinks
9/13Introduction and expositionsHW1: Hello generative world! 
9/20Speaker: Matti Mariansky
The world of Generative Art, NFTs, curation and outlets.
  
9/27Speaker: Jonathan Whitaker
Exploring the Generative Landscape.
HW2: Loss functions 
10/4Speaker: Emad Mostaque
Building Stability.AI and Stable Diffusion.

In-class: Assignment, GANs, CLIP
  
10/11No class!HW3: GANs 
10/18Speaker: Doron Adler
Practicing AI art, contemporary methods.
  
10/25Speaker: Prof. Dav Novy
Generative storytelling.

In-class: assignment, Diffusion models
HW4: Diffusers 
11/1Speaker: Nora Wixom
Illusory Truth: Bias in CV/ML and Generative Systems
  
11/8Speaker: TomLikesRobots
An Artistic approach to Generative AI
HW5: StyleGAN Playground 
11/15Speaker: Yuval Gur
Generative music algorithms.

In-class: Assignment, text generation models
  
11/22Speaker: Rinon Gal
StyleGAN to the max, domain adaptation, object insertion.
HW6: Music generators 
11/29

Speaker: David Ha (meeting 9am)

Contemporary work on Generative Media at Stability.AI

Final projects 
12/6In-class review, final projects workshopFinal projects 
12/13Final projects reviewFinal projects 

Class schedule is subject to change throughout the semester.

Grading

This class will have bi-weekly take-home work, and a final project. The final grade will consist of 50% take-home work grades and 50% final project grade.

Resources

Cover image created by Stable Diffusion, prompt: “robotic sheep in pasture, by Rob Gonsalves”