Class Materials (Slides, Recordings, Homeworks)

🗂 Student Repository

(for uploading homework)

🤚 Prerequisites

To be able to complete the homeworks and follow the technical matrial in class, please make sure you run through the below list of tutorials:

Python – you don’t need to be an expert python programmer, but you do need to know the basics. If you don’t, the official Python tutorial is a good place to start (as well as the below Python and Numpy Tutorial).
Scientific Python – We will be using a few popular python libraries, in particular NumPy, matplotlib and pandas. If you are not familiar with these libraries, you should probably start by going through:
- Python and Numpy Tutorial
- Maplotlib Tutorial
Math – We will also use some notions of Linear Algebra, Calculus, Statistics and Probability theory. You should be able to follow along if you learned these in the past as it won’t be very advanced. However, please run through the following notebooks as an introduction or refresher:
- Linear Algebra in Python
- Multivariate Differential Calculus in Python

🦾 Introduction to Machine Learning

Below are recorded sessions (about 25min / ea) of introduction to machine learning as was presented in class, and beyond it. Please review them. (x1.5 speed is recommended)

sss VViVid📋 Homework

We will be posting homework notebooks on our github repo for this class: https://github.com/mitmedialab/MAS.S60.Fall2020

If you run into any problems – post an “issue” on the github repository, and we will address your questions.

Each notebook in GitHub will have a “Run in Colab” link which will take you right into a Colab session with that notebook (Note: you may need to let the Colab execute the code by approving the unauthorized Github origin).

If you are new to Google Colab, please check out this tutorial : https://www.youtube.com/watch?v=inN8seMm7UI&ab_channel=TensorFlow

Your completed notebooks (e.g. all cells executed with outputs) should be uploaded to Google Drive in the appropriate folder with your name: https://drive.google.com/drive/folders/1yme96GbPKtwPm_GVLLCLWM2kq0OxEP0P?usp=sharing . It’s possible to upload the completed notebook right from Colab

0️⃣ Homework 0: Warmup

https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW0%20-%20Warmup.ipynb

This exercise is designed to get us all up and running executing code on Colab notebooks. The notebook is self explanatory, but there are many opportunities for you to tweak parameters and experiment.

The due date for HW0 is Thursday October 1st.

1️⃣ Homework 1: Generative Models

https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW1%20-%20Generative%20Models.ipynb

This assignment will take us through utilizing some generative models (in particular GANs) that were pre-trained and published to the world by Google, NVidia, et al., to generate new never-seen-before realistic images from “noise”.

The due date for HW1 is Monday October 12 at 11:59pm.

2️⃣ Homework 2: Video Deepfakes & First Order Model

https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW2.ipynb

In this assignment, we will create a generative talking head video using First Order Model and other image animation.

The due date for HW2 is Monday October 26 at 11:59pm.

3️⃣ Homework 3: Voice Cloning

https://github.com/mitmedialab/MAS.S60.Fall2020/blob/master/homework/HW3_Voice_Cloning.ipynb

In this assignment, we will cover two methods for deepfaking voices. One is “Voice-Cloning” based on a neural network-based system for text-to-speech synthesis (TTS). Another is expressive voice synthesis, where you can make a voice emote and sing (rhythm and pitch).

The due date for HW3 is Thursday, November 12th at 11:59 pm.

Class #1 | 9/3

Class Introduction & Overview

Dr. Roy Shilkrot, Pat Pataranutaporn, Joanne Leong, Professor Pattie Maes

🎬 Class Recording

Class #2 | 9/10

Deepfakes Production

👤Invited Speaker: Dr. Omer Ben-Ami, Canny AI

🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning

🎬 Class Recording

💻 Technical Session

Class #3 | 9/17

History of Media Manipulation & Faking

👤Invited Speaker: Dr. Judith Donath, Harvard’s Berkman Center

🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning

🎬 Class Recording

💻 Technical Session

💻 Introduction to Machine Learning Series

📋 Homework 0

Class References

Extensive law journal article on dangers of deep fakes

Citron, D. K. and R. Chesney (2018). “Deep Fakes: A Looming Crisis for National Security, Democracy and Privacy?” Lawfare.

How to video so that the result can be relied up

WITNESS Video as evidence : https://vae.witness.org/video-as-evidence-field-guide/
How the courts look at video as evidence – and how they ought to:
Silbey, J. “EVIDENCE VERITÉ AND THE LAW OF FILM.” CARDOZO LAW REVIEW 31: 4.
Silbey, J. M. (2003). “Judges as Film Critics: New Approaches to Filmic Evidence.” University of Michigan Journal of Law Reform(2): 493-572.
Making fakes makes people more resistant to them:
Roozenbeek, J. and S. van der Linden (2019). “Fake news game confers psychological resistance against online misinformation.” Palgrave Communications 5(1): 65
Ricky Leacock was a founding Media Lab professor. I don’t quite agree with his assessment of cinema verite as a way of conveying truth, but it is interesting to read:
Blue, J. (1965). “THOUGHTS ON CINÉMA VÉRITÉ AND A DISCUSSION WITH THE MAYSLES BROTHERS.” Film Comment 2(4): 22-30.
Leacock, R. and J. Blue (1965). “ONE MAN’S TRUTH: An Interview with Richard Leacock The Second in A Series on Cinéma Vérité.” Film Comment 3(2): 15-23.

History photo fakery

Blakemore, E. (2019). “How Photos Became a Weapon in Stalin’s Great Purge.” History Stories. Retrieved November 15, 2019, from https://www.history.com/news/josef-stalin-great-purge-photo-retouching.
The power of photography to reshape our memory of events:
Henkel, L. A. (2011). “Photograph-induced memory errors: When photographs make people claim they have done things they have not.” Applied Cognitive Psychology 25(1): 78-86.
The power of photography to shape self-image
Kleemans, M., S. Daalmans, et al. (2018). “Picture perfect: The direct effect of manipulated Instagram photos on body image in adolescent girls.” Media Psychology 21(1): 93-110.
Lee, M. and H.-H. Lee (2019). “Can virtual makeovers using photo editing applications moderate negative media influences on SNS users’ body satisfaction?” Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement.
Mitchell, W. J. (1994). The reconfigured eye: Visual truth in the post-photographic era, MIT Press.

Class #4 | 9/24

Image-to-Image Translation for Deepfakes

👤Invited speaker: Dr. Ohad Fried, IDC/ Stanford University

👤Invited speaker: Dr. Phillip Isola, MIT CSAIL

💻 Talk by Dr. Ohad Fried

💻 Talk by Dr. Phillip Isola

Class #5 | 10/1

Pushing the Boundaries of Synthetic Media & Deepfakes

👤Invited Speaker : Aliaksandr Siarohin, University of Trento

Aliaksandr Siarohin is a Ph.D. candidate in Information and Communication Technology at the University of Trento. He has published several works on GANs, motion transfer, and human image generation at conferences such as CVPR and NEURIPs. One of his well-known works has been the “First Order Model for Image Animation.”

👤Invited Speaker : Sam Kriegman, University of Vermont

Sam Kreigman is a postdoc at the University of Vermont interested in evolution, robots, and computer-designed organisms. He has worked on creating ”living and self-healing robots,’ in which living cells are repurposed and assembled into new life-forms.

💻 Talk by Aliaksandr & Sam

Class #6 | 10/8

Generative Algorithm for Art & Design

👤Invited Speaker : Ali Jahanian, MIT CSAIL

Ali Jahanian is currently a postdoc at MIT CSAIL, interested in the intersection between AI, creativity, aesthetics, and design. Part of his research is to explore how to translate designers’ intuitions into machine representations, with the goal to develop technologies that augment and democratize human creativity. He has published several papers at conferences such as CHI, IUI, and CVPR. One of his recent papers investigated the “steerability” of GANs.

👤Invited Speaker : Harshit Agrawal, Adobe

Harshit Agrawal is an alumnus of the Fluid Interfaces group and is currently a Design Researcher at the Adobe Design Lab. He explores the “human-machine creativity continuum” that spans the areas of AI and HCI. He has published several research papers at HCI conferences including SIGGRAPH, UIST, IUI, and Ubicomp. Additionally, many of his works have been exhibited at art shows around the world. One of his works is included in the permanent exhibition at the HeinzNixdofs Museum Forum in Paderborn, Germany.

💻 Talk by Ali & Harshit

Class #7 |10/15

Generative Algorithm for Art & Design

👤Invited Speaker : Carter Huffman, Modulate.ai

Carter Huffman is the CTO and co-founder of Modulate.ai, a Boston-based startup empowering the future of identity, online safety, and communication through voice. Modulate.ai introduces the world’s first “voice skins,” with which users can speak with the voice of a chosen character in real-time. Their technology is capable of making these voices sound “real,” while preserving the intent and emotion of the original user’s speech. Prior to founding his company, he graduated from MIT with a BS in Physics in 2014, served as a technologist at the Machine Learning and Instrument Autonomy group at JPL, NASA’s Jet Propulsion Laboratory, and worked as a software engineer at Dimensional Insight.

💻 Talk by Carter

💻 Class Presentation

Class #8 | 10/22

Special Conversation : Deepfakes, Science-Fiction and the Future ✨

👤Invited Speaker : Jonathan Nolan, creators/director/writer of Westworld, Interstellar, Batman Begins, and more

Jonathan Nolan is a British-American screenwriter, television producer, director and author. He is the creative genius behind numerous thrillers, superhero films, as well as science fiction series and blockbusters. In collaboration with his brother, Christopher Nolan, he has co-written Batman: The Dark Knight (2008), Batman: The Dark Knight Rises (2012), and Interstellar (2014). His short story Memento Mori was the basis for Memento (2000). He is the creator of the CBS science fiction series Person of Interest (2011-2016). Alongside his wife Lisa Joy, he is a co-creator of the smash hit HBO science fiction western series, Westworld (2016-present).

👤Moderator: Neo Mohsenvand, MIT Media Lab

💻 Nolan's talk

Class #9 |10/29

Deepfakes & Governance

👤Invited Speaker : John Bowers, Harvard’s Berkman Klein Center

John Bowers is a Research Associate at Harvard’s Berkman Klein Center, investigating security vulnerabilities in autonomous systems, cryptographic methods for securing distributed archives, the regulation of social media platforms, and the quantitative analysis of legal texts. He co-authored a report published in Science, titled “Adversarial attacks on medical machine learning.” He has also penned several articles for WIRED, the Wall Street Journal, and more on the topics of ethics, governance, internet services, AI, and disinformation. This includes a recent article, “What should newsrooms do about deepfakes? These three things, for starters” for the Nieman Journalism Lab.

💻 Talk by John

Class #10 | 11/5

Detecting Deepfakes

👤Invited Speaker : Matt Groh is a PhD student in the Affective Computing group. His research work lies at the intersection of machine learning, human behavior, and decision making. One of his most recent publications is a paper titled “Human detection of machine manipulated media,” which reports on a study that examined 15,000 individuals’ ability to discern machine-manipulated media and investigated factors that can impact their capacity to do so.

👤Invited Speaker : Yossef Daar is a co-founder and CPO at Cyabra, a company that is tackling the issue of disinformation with deepfakes. Cyabra’s mission is to solve the threat of fake news for brands and the public sector by providing software to detect fictitious accounts on social media platforms, discover trends, understand narratives, and connect with one’s audience.

💻 Talk by Matt & Yossef

Class #11 |11/12

Deepfakes, Documentary, and Digital Literacy

👤Invited Speaker :Francesca Panetta is the XR Creative Director in the MIT Center for Advanced Virtuality. She is an immersive artist and journalist, who has a wealth of experience in using emerging technologies to create innovative new forms of storytelling that have a social impact. She co-led the project “In Event of Moon Disaster,” which has helped to spark critical awareness of deepfake technologies among the public.

👤Invited Speaker :Dr. Joshua Glick is an Assistant Professor of English, Film and Media Studies at Hendrix College and a Fellow at the MIT Open Documentary Lab. His research and teaching explore global documentary, critical race studies, emerging media, and Hollywood as an evolving form of industrial and artistic production. He is actively working on designing a media literacy curriculum to teach students about the threat of disinformation in light of deepfakes, as well as the civic uses of synthetic media.

💻 Talk by Francesca & Joshua

Class #12 | 11/19

Deepfakes & Human Physiology

👤Invited Speaker : Dr. Daniel McDuff is a Principal Researcher at Microsoft. His research intersects psychology and computer science. He is designing hardware and algorithms for sensing human behavior at scale in order to build technologies that make life better. Applications for his research include understanding mental health, improving online learning, and designing new connected devices (IoT) and mixed reality experiences. He attained his Ph.D. from the MIT Media Lab’s Affective Computing Group.

👤Invited Speaker : Dr. Javier Hernandez is from Microsoft Research. He is focused on the development of emotionally intelligent tools to further the understanding of humans while fostering greater health and quality of life. Much of his research works has been on the measurement, understanding, and management of daily life stress. His work has received a multitude of recognitions and awards, and has also been featured by several media outlets including National Geographic, The Economist, and The Times. He graduated with a Ph.D. from the MIT Media Lab’s Affective Computing Group.

👤Invited Speaker : Dr. Tal Hassner is an Applied Research Lead at Facebook AI. He was formerly a Principal Scientist at Amazon AWS, and is affiliated with The Open University of Israel, Department of Mathematics and Computer Science where he was an associate professor until 2018. He has also served as associate editor and co-chair for several IEEE journals and conferences. His research intersects Machine Learning (Deep Learning), Statistical Pattern Recognition, Computer Vision, and Computer Graphics. Many of his works have focused on face recognition, attribute prediction, face alignment, and 3D reconstruction of face shapes in images and videos.

💻 Talk by Daniel, Javier, and Tal

Class #14 |12/3

Final Presentation

💻 Final Presentation

#	Date	Topics
1	9/3	Class Introduction & Overview
2	9/10	👤Invited Speaker: Dr. Omer Ben-Ami, Canny AI 🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning
3	9/17	👤Invited Speaker: Dr. Judith Donath, Harvard’s Berkman Center 🖐In-class tutorial: Introduction to Python, Machine, and Deep Learning (con) 📋Introduce Homework 0 (Due October 1, 2020)
4	9/24	👤Invited speaker: Dr. Phillip Isola, MIT CSAIL 👤Invited speaker: Dr. Ohad Fried, IDC/ Stanford University
5	10/1	👤Invited Speaker : Aliaksandr Siarohin, University of Trento 👤Invited Speaker : Sam Kriegman, University of Vermont 🖐Pre-recorded tutorial : pix2pix 🖐Pre-recorded tutorial : Face Cloning & Swapping 📋Homework 1 : pix2pix & Homework 2 : first order model
6	10/8	👤Invited Speaker : Harshit Agrawal, Adobe 👤Invited Speaker : Ali Jahanian, MIT CSAIL 📋Introduce Homework 3 : Voice Cloning
7	10/15	👤Invited Speaker : Carter Huffman, Modulate.ai 📋Homework 1,2, 3 Review
8	10/22	✨Special Conversation : Deepfakes, Science-Fiction and the Future by Jonathan Nolan, creators/director/writer of Westworld, Interstellar, Batman Begins, and more and Neo Mohsenvand, MIT Media Lab💡
9	10/29	👤Invited speaker : John Bowers, Harvard’s Berkman Klein Center 📋Homework 3 Review 💡Final Project Brainstorming
10	11/5	👤Invited speaker : Matt Groh, MIT Media Lab 👤Invited Speaker : Yossef Daar,Co-founder and CPO at Cyabra
11	11/12	👥Invited Speaker : Fran Panetta, MIT Center for Advanced Virtuality and Dr. Josh Glick, Hendrix College & MIT Open Documentary Lab 💡Final Project Check-in
12	11/19	👥Invited Speaker : Dr. Daniel McDuff and Dr. Javier Hernandez, Microsoft Research 👤Invited Speaker : Dr. Tal Hassner , Facebook AI & Open University of Israel
		Work on Final Project
13	11/26	😀Thanksgiving Vacation
14	12/3	👏Final Projects presentations

Class Materials (Slides, Recordings, Homeworks)

😀 Public Session

🖥 Technical Session

💬 Q&A

📋 Homework

🗂 Student Repository

(for uploading homework)

🤚 Prerequisites

🦾 Introduction to Machine Learning

sss VViVid📋 Homework

1️⃣ Homework 1: Generative Models

2️⃣ Homework 2: Video Deepfakes & First Order Model

3️⃣ Homework 3: Voice Cloning

Class #1 | 9/3

Class Introduction & Overview

Class #2 | 9/10

Deepfakes Production

Class #3 | 9/17

History of Media Manipulation & Faking

Class References

Class #4 | 9/24

Image-to-Image Translation for Deepfakes

Class #5 | 10/1

Pushing the Boundaries of Synthetic Media & Deepfakes

Class #6 | 10/8

Generative Algorithm for Art & Design

Class #7 |10/15

Generative Algorithm for Art & Design

Class #8 | 10/22

Special Conversation : Deepfakes, Science-Fiction and the Future ✨

Class #9 |10/29

Deepfakes & Governance

Class #10 | 11/5

Detecting Deepfakes

Class #11 |11/12

Deepfakes, Documentary, and Digital Literacy

Class #12 | 11/19

Deepfakes & Human Physiology

Class #14 |12/3

Final Presentation

#

Date

Topics