Class Materials (Slides, Recordings, Homeworks)
😀 Public Session
https://mit.zoom.us/j/96620651005
🖥 Technical Session
https://mit.zoom.us/j/97016907993?pwd=RkwxQlVtRDhZWW5ZY000U2g1N1lVUT09
Password : awesome!
💬 Q&A
https://github.com/mitmedialab/MAS.S60.Fall2020/issues
📋 Homework
http://deepfakes.media.mit.edu/class-materials#homework
🗂 Student Repository
(for uploading homework)
🤚 Prerequisites
To be able to complete the homeworks and follow the technical matrial in class, please make sure you run through the below list of tutorials:
- Python – you don’t need to be an expert python programmer, but you do need to know the basics. If you don’t, the official Python tutorial is a good place to start (as well as the below Python and Numpy Tutorial).
- Scientific Python – We will be using a few popular python libraries, in particular NumPy, matplotlib and pandas. If you are not familiar with these libraries, you should probably start by going through:
- Math – We will also use some notions of Linear Algebra, Calculus, Statistics and Probability theory. You should be able to follow along if you learned these in the past as it won’t be very advanced. However, please run through the following notebooks as an introduction or refresher:
🦾 Introduction to Machine Learning
Below are recorded sessions (about 25min / ea) of introduction to machine learning as was presented in class, and beyond it. Please review them. (x1.5 speed is recommended)
- Part 1 – Visual recognition tasks and ML intuitions
- Part 2 – Simple models, measuring performance and naive Bayes
- Part 3 – Linear regression
- Part 4 – Logistic regression and multi-logistic regression (softmax)
- Part 5 – Support Vector Machines, Neural Networks
- Part 6 – Learning and training, convex optimization and gradient descent
- Part 7 – Learning matters: data, curves and overfitting
- Part 8 – Backpropagation
- Part 9 – Convolutional neural networks
sss VViVid📋 Homework
We will be posting homework notebooks on our github repo for this class: https://github.com/mitmedialab/MAS.S60.Fall2020
If you run into any problems – post an “issue” on the github repository, and we will address your questions.
Each notebook in GitHub will have a “Run in Colab” link which will take you right into a Colab session with that notebook (Note: you may need to let the Colab execute the code by approving the unauthorized Github origin).
If you are new to Google Colab, please check out this tutorial : https://www.youtube.com/watch?v=inN8seMm7UI&ab_channel=TensorFlow
Your completed notebooks (e.g. all cells executed with outputs) should be uploaded to Google Drive in the appropriate folder with your name: https://drive.google.com/drive/folders/1yme96GbPKtwPm_GVLLCLWM2kq0OxEP0P?usp=sharing . It’s possible to upload the completed notebook right from Colab
0️⃣ Homework 0: Warmup
https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW0%20-%20Warmup.ipynb
This exercise is designed to get us all up and running executing code on Colab notebooks. The notebook is self explanatory, but there are many opportunities for you to tweak parameters and experiment.
The due date for HW0 is Thursday October 1st.
1️⃣ Homework 1: Generative Models
This assignment will take us through utilizing some generative models (in particular GANs) that were pre-trained and published to the world by Google, NVidia, et al., to generate new never-seen-before realistic images from “noise”.
The due date for HW1 is Monday October 12 at 11:59pm.
2️⃣ Homework 2: Video Deepfakes & First Order Model
https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW2.ipynb
In this assignment, we will create a generative talking head video using First Order Model and other image animation.
The due date for HW2 is Monday October 26 at 11:59pm.
3️⃣ Homework 3: Voice Cloning
https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW3_Voice_Cloning.ipynb
In this assignment, we will cover two methods for deepfaking voices. One is “Voice-Cloning” based on a neural network-based system for text-to-speech synthesis (TTS). Another is expressive voice synthesis, where you can make a voice emote and sing (rhythm and pitch).
The due date for HW3 is Thursday, November 12th at 11:59 pm.
Class #1 | 9/3
Class Introduction & Overview
Dr. Roy Shilkrot, Pat Pataranutaporn, Joanne Leong, Professor Pattie Maes
Class #2 | 9/10
Deepfakes Production
👤Invited Speaker: Dr. Omer Ben-Ami, Canny AI
🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning
Class #3 | 9/17
History of Media Manipulation & Faking
👤Invited Speaker: Dr. Judith Donath, Harvard’s Berkman Center
🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning
Class References
Extensive law journal article on dangers of deep fakes
- Citron, D. K. and R. Chesney (2018). “Deep Fakes: A Looming Crisis for National Security, Democracy and Privacy?” Lawfare.
How to video so that the result can be relied up
- WITNESS Video as evidence : https://vae.witness.org/video-as-evidence-field-guide/
- How the courts look at video as evidence – and how they ought to:
- Silbey, J. “EVIDENCE VERITÉ AND THE LAW OF FILM.” CARDOZO LAW REVIEW 31: 4.
- Silbey, J. M. (2003). “Judges as Film Critics: New Approaches to Filmic Evidence.” University of Michigan Journal of Law Reform(2): 493-572.
- Making fakes makes people more resistant to them:
- Roozenbeek, J. and S. van der Linden (2019). “Fake news game confers psychological resistance against online misinformation.” Palgrave Communications 5(1): 65
- Ricky Leacock was a founding Media Lab professor. I don’t quite agree with his assessment of cinema verite as a way of conveying truth, but it is interesting to read:
- Blue, J. (1965). “THOUGHTS ON CINÉMA VÉRITÉ AND A DISCUSSION WITH THE MAYSLES BROTHERS.” Film Comment 2(4): 22-30.
- Leacock, R. and J. Blue (1965). “ONE MAN’S TRUTH: An Interview with Richard Leacock The Second in A Series on Cinéma Vérité.” Film Comment 3(2): 15-23.
History photo fakery
- Blakemore, E. (2019). “How Photos Became a Weapon in Stalin’s Great Purge.” History Stories. Retrieved November 15, 2019, from https://www.history.com/news/josef-stalin-great-purge-photo-retouching.
- The power of photography to reshape our memory of events:
- Henkel, L. A. (2011). “Photograph-induced memory errors: When photographs make people claim they have done things they have not.” Applied Cognitive Psychology 25(1): 78-86.
- The power of photography to shape self-image
- Kleemans, M., S. Daalmans, et al. (2018). “Picture perfect: The direct effect of manipulated Instagram photos on body image in adolescent girls.” Media Psychology 21(1): 93-110.
- Lee, M. and H.-H. Lee (2019). “Can virtual makeovers using photo editing applications moderate negative media influences on SNS users’ body satisfaction?” Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement.
- Mitchell, W. J. (1994). The reconfigured eye: Visual truth in the post-photographic era, MIT Press.
Class #4 | 9/24
Image-to-Image Translation for Deepfakes
👤Invited speaker: Dr. Ohad Fried, IDC/ Stanford University
👤Invited speaker: Dr. Phillip Isola, MIT CSAIL
Class #5 | 10/1
Pushing the Boundaries of Synthetic Media & Deepfakes
👤Invited Speaker : Aliaksandr Siarohin, University of Trento
Aliaksandr Siarohin is a Ph.D. candidate in Information and Communication Technology at the University of Trento. He has published several works on GANs, motion transfer, and human image generation at conferences such as CVPR and NEURIPs. One of his well-known works has been the “First Order Model for Image Animation.”
👤Invited Speaker : Sam Kriegman, University of Vermont
Sam Kreigman is a postdoc at the University of Vermont interested in evolution, robots, and computer-designed organisms. He has worked on creating ”living and self-healing robots,’ in which living cells are repurposed and assembled into new life-forms.
Class #6 | 10/8
Generative Algorithm for Art & Design
👤Invited Speaker : Ali Jahanian, MIT CSAIL
Ali Jahanian is currently a postdoc at MIT CSAIL, interested in the intersection between AI, creativity, aesthetics, and design. Part of his research is to explore how to translate designers’ intuitions into machine representations, with the goal to develop technologies that augment and democratize human creativity. He has published several papers at conferences such as CHI, IUI, and CVPR. One of his recent papers investigated the “steerability” of GANs.
👤Invited Speaker : Harshit Agrawal, Adobe
Harshit Agrawal is an alumnus of the Fluid Interfaces group and is currently a Design Researcher at the Adobe Design Lab. He explores the “human-machine creativity continuum” that spans the areas of AI and HCI. He has published several research papers at HCI conferences including SIGGRAPH, UIST, IUI, and Ubicomp. Additionally, many of his works have been exhibited at art shows around the world. One of his works is included in the permanent exhibition at the HeinzNixdofs Museum Forum in Paderborn, Germany.
Class #7 |10/15
Generative Algorithm for Art & Design
👤Invited Speaker : Carter Huffman, Modulate.ai
Class #8 | 10/22
Special Conversation : Deepfakes, Science-Fiction and the Future ✨
👤Invited Speaker : Jonathan Nolan, creators/director/writer of Westworld, Interstellar, Batman Begins, and more
Jonathan Nolan is a British-American screenwriter, television producer, director and author. He is the creative genius behind numerous thrillers, superhero films, as well as science fiction series and blockbusters. In collaboration with his brother, Christopher Nolan, he has co-written Batman: The Dark Knight (2008), Batman: The Dark Knight Rises (2012), and Interstellar (2014). His short story Memento Mori was the basis for Memento (2000). He is the creator of the CBS science fiction series Person of Interest (2011-2016). Alongside his wife Lisa Joy, he is a co-creator of the smash hit HBO science fiction western series, Westworld (2016-present).
👤Moderator: Neo Mohsenvand, MIT Media Lab
Class #9 |10/29
Deepfakes & Governance
👤Invited Speaker : John Bowers, Harvard’s Berkman Klein Center
Class #10 | 11/5
Detecting Deepfakes
👤Invited Speaker : Matt Groh is a PhD student in the Affective Computing group. His research work lies at the intersection of machine learning, human behavior, and decision making. One of his most recent publications is a paper titled “Human detection of machine manipulated media,” which reports on a study that examined 15,000 individuals’ ability to discern machine-manipulated media and investigated factors that can impact their capacity to do so.
👤Invited Speaker : Yossef Daar is a co-founder and CPO at Cyabra, a company that is tackling the issue of disinformation with deepfakes. Cyabra’s mission is to solve the threat of fake news for brands and the public sector by providing software to detect fictitious accounts on social media platforms, discover trends, understand narratives, and connect with one’s audience.
Class #11 |11/12
Deepfakes, Documentary, and Digital Literacy
👤Invited Speaker :Francesca Panetta is the XR Creative Director in the MIT Center for Advanced Virtuality. She is an immersive artist and journalist, who has a wealth of experience in using emerging technologies to create innovative new forms of storytelling that have a social impact. She co-led the project “In Event of Moon Disaster,” which has helped to spark critical awareness of deepfake technologies among the public.
👤Invited Speaker :Dr. Joshua Glick is an Assistant Professor of English, Film and Media Studies at Hendrix College and a Fellow at the MIT Open Documentary Lab. His research and teaching explore global documentary, critical race studies, emerging media, and Hollywood as an evolving form of industrial and artistic production. He is actively working on designing a media literacy curriculum to teach students about the threat of disinformation in light of deepfakes, as well as the civic uses of synthetic media.
Class #12 | 11/19
Deepfakes & Human Physiology
👤Invited Speaker : Dr. Daniel McDuff is a Principal Researcher at Microsoft. His research intersects psychology and computer science. He is designing hardware and algorithms for sensing human behavior at scale in order to build technologies that make life better. Applications for his research include understanding mental health, improving online learning, and designing new connected devices (IoT) and mixed reality experiences. He attained his Ph.D. from the MIT Media Lab’s Affective Computing Group.
Class #14 |12/3
Final Presentation
# |
Date |
Topics |
| 1 | 9/3 | Class Introduction & Overview |
| 2 | 9/10 |
👤Invited Speaker: Dr. Omer Ben-Ami, Canny AI 🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning |
| 3 | 9/17 |
👤Invited Speaker: Dr. Judith Donath, Harvard’s Berkman Center 🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning (con) |
| 4 | 9/24 |
👤Invited speaker: Dr. Phillip Isola, MIT CSAIL 👤Invited speaker: Dr. Ohad Fried, IDC/ Stanford University |
| 5 | 10/1 |
👤Invited Speaker : Aliaksandr Siarohin, University of Trento 👤Invited Speaker : Sam Kriegman, University of Vermont 🖐Pre-recorded tutorial : pix2pix 🖐Pre-recorded tutorial : Face Cloning & Swapping 📋Homework 1 : pix2pix & Homework 2 : first order model |
| 6 | 10/8 |
👤Invited Speaker : Harshit Agrawal, Adobe 👤Invited Speaker : Ali Jahanian, MIT CSAIL 📋Introduce Homework 3 : Voice Cloning |
| 7 | 10/15 |
👤Invited Speaker : Carter Huffman, Modulate.ai 📋Homework 1,2, 3 Review |
| 8 | 10/22 | ✨Special Conversation : Deepfakes, Science-Fiction and the Future by Jonathan Nolan, creators/director/writer of Westworld, Interstellar, Batman Begins, and more and Neo Mohsenvand, MIT Media Lab💡 |
| 9 | 10/29 |
👤Invited speaker : John Bowers, Harvard’s Berkman Klein Center 📋Homework 3 Review 💡Final Project Brainstorming |
| 10 | 11/5 |
👤Invited speaker : Matt Groh, MIT Media Lab 👤Invited Speaker : Yossef Daar,Co-founder and CPO at Cyabra |
| 11 | 11/12 |
👥Invited Speaker : Fran Panetta, MIT Center for Advanced Virtuality and Dr. Josh Glick, Hendrix College & MIT Open Documentary Lab 💡Final Project Check-in |
| 12 | 11/19 |
👥Invited Speaker : Dr. Daniel McDuff and Dr. Javier Hernandez, Microsoft Research 👤Invited Speaker : Dr. Tal Hassner, Facebook AI & Open University of Israel |
| Work on Final Project | ||
| 13 | 11/26 | 😀Thanksgiving Vacation |
| 14 | 12/3 | 👏Final Projects presentations |