**FALL 2010 Class Information:**

Lectures: Tuesdays and Thursdays 9:30-11:00, 8-205

Recitations: Fridays 10-11, 8-205

Textbook: *Pattern Classification* by Duda, Hart, and Stork, with
other readings

Staff | Announcements | Assignments| Syllabus | Policies

**Instructor**:

Prof. Rosalind W. Picard

Office: E14-374g

Office hours: I'm available Tuesdays 11-12 pm, weekly, and for additional hours during project planning season: Nov 5, Nov 9 and Nov 16. Outside of posted hours you can meet with me by scheduling an appointment with Alex at email below. I'm also usually available right after class walking back to the lab & always happy to try to answer your questions in person (I usually don't have enough hours to answer by email).

Phone: 617-253-0611

picard ( @ media dot mit dot edu)

**Teaching Assistants**:

Daniel McDuff

Office: E14-374a

Office hours: Monday 5-6pm

Phone: 617-254-2136

djmcduff ( @ media dot mit dot edu)
**
**

**Javier Hernandez Rivera
**

Office: E14-374a

Office hours: Wednesday 10-11am

Phone: 617-253-8628

javierhr ( @ mit dot edu)

**Support Staff**:

**Lillian Lai**

Office: E14-374h

Phone: 617-253-0369

lillian ( @ media dot mit dot edu)

Staff | Announcements | Assignments| Syllabus | Policies

**R 9-9** First day of class. (Visitors welcome, although class
size is limited to 20.)

The first recitation will be held Friday Sep 10, providing an
overview for those who would like help with MATLAB, probability, or
the first problem set (PS). This class has traditionally made heavy use of
MATLAB. Here is
a MATLAB
Tutorial you may find helpful. You might also enjoy this short
insightful (optional) piece
on
nuances in the use of probability by Media Lab graduate Tom Minka,
Ph.D.

There is a useful website for the text, which includes not only ppt slides for the chapters, but also errata for each edition of the text. Please correct the errors in your text now to save you time later when you are reading and trying to understand.

The IAPR Pattern Recognition Education Resources web site
was initiated by the Internation Association for
Pattern Recognition (http://www.iapr.org/).
The goal was a web site that can support students, researchers and staff.
Of course, advances in pattern recognition and its subfields means that
developing the site will be a never-ending process.
What resources does the IAPR Education web site have?
The most important resources are for students, researchers and educators.
These include lists with URLs to:
- Tutorials and surveys
- Explanatory text
- Online demos
- Datasets
- Book lists
- Free code
- Course notes
- Lecture slides
- Course reading lists
- Coursework/homework
- A list of course web pages at many universities

**OLD EXAMPLES OF PROJECTS** Here are the 2002
Class Projects Page, the 2004
Class Projects Page, the 2006
Class Projects Page, and the 2008
Class Projects Page.

Staff | Announcements | Assignments| Syllabus | Policies

Note: Future assignments below, beyond what has been stated in class each week, are predictions. They are subject to revision, with each revision a more accurate prediction than the one before.

* "If you must predict,
predict often." -- Prof. Paul Samuelson, 1970 Nobel Prize in Economics
*

R 9-9 ** Lecture 1 ** Introduction, Reading: DHS Chap 1, A.1-A.2
**
**

**PS 1 +
Dataset (500x2 array, 500 points in 2-dim space, each column row a sample point (x,y))
**

F 9-10 ** Recitation 1 ** Matlab Introduction [file + tutorial]

T 9-14 ** Lecture 2 ** Reading: DHS Chap A.2-A.5.

Recommended readings on probability:

- Alvin W. Drake. Fundamentals of applied probability theory.

- William Feller. An introduction to probability theory and its applications.

- Richard W. Hamming. The art of probability for scientis and engineers.

- Andrew's Moore's tutorials: [ probabilistic analytics,
probability densities, and
gaussians ]

R 9-16 ** Lecture 3 ** Reading: DHS Chap 2.1-2.4 (can skip
2.3.1, 2.3.2) Notes (courtesy by Rob Speer)

F 9-17 ** Recitation 2 **

M 9-20 First submission PS 1

T 9-21 ** Lecture 4 ** Reading: DHS Chap 2.5-2.7 Notes

R 9-23 ** Lecture 5 ** Reading: DHS Chap 2.8.3, 2.9, 3.1-3.2

Final submission PS 1

F 9-24 ** Recitation 3 **

T 9-28 ** Lecture 6 ** DHS 3.1-3.5.1 Slides

R 9-30 ** Lecture 7 ** Guest lecture from Javier Hernandez Rivera: Dimensionality Reduction
DHS 3.7.1-.3, 3.8 Slides

F 10-1 ** Recitation 4 **

PS 3 + data prob 1, data prob 4

M 10-4 Final submission PS 2

T 10-5 ** Lecture 8 ** HMMs, reading: A Tutorial On Hidden Markov Models and Selected Applications in Speech Recognition, L.R. Rabiner, Proceedings of the IEEE, Vol 77 No 2, Feb. 1989.; optional extra reading: DHS 3.10 (beware, as DHS uses non-standard notation in this section).

R 10-7 ** Lecture 9 ** HMMs, same reading. Notes

F 10-8 ** Recitation 5 **

T 10-12 ** Lecture 10 ** DHS 2.10 and 3.9, Missing Data and Expectation Maximization, brief intro to Bayes Nets, 2.11
Notes

First submission PS 3 (prob. 1, 2 and 3)

R 10-14 ** Lecture 11 **(Picard away co-directing TTT meeting)
Inference on Bayes Nets and Dynamic Bayes Nets, guest lecture from
Dr. Rana el Kaliouby, Reading: Introduction
to Graphical Models and Bayesian Networks, Kevin Murphy,
1998. (DBN Lecture PDF slides)

First submission PS 3 (prob. 4 and 5) + solution

F 10-15 ** Recitation 6 **(during 25th anniversary sponsor/alumni event)

M 10-18 Final submission PS 3

T 10-19 ** Lecture 12 ** DHS 10.2-10.4.3 Mixture densities, K-means clustering, Quiz review

R 10-21 ** Lecture 13 ** DHS 10.4.3, 10.4.4, 10.6-10.10 Clustering

F 10-22 ** Recitation 7** (Midterm 2008)

M 10-25 First submission PS 4 + (solution + code)

T 10-26 ** Lecture 14 ** DHS 4.5.1, 4.5.4, K-nn classifier, DHS 5.1-5.3, 5.8.1 Linear Discriminants. Slides

W 10-27 Final submission PS 4

R 10-28 ** MIDTERM QUIZ, Covers Lectures 1-13 (Nov 17, 2010 = DROP DATE)**

F 10-29 ** Recitation 8**

PS 5 (paper and datasets: prob. 1 and prob. 3)

T 11-2 ** Lecture 15 ** (Picard in UK), Provided Data for Projects Ready
Presentations on project data available, Optional Readings (short and
very interesting articles to discuss with your friends, given Election
Day)

"Election Selection: Are we using the worst voting procedure?" Science
News, Nov 2 2002.

Range voting: Best way to select a leader?

Slides on Class Project

R 11-4 ** Lecture 16 ** Guest lecture on regression by Sophia Yuditskaya Slides

Project Plan Due if you're using your own data

F 11-5 ** Recitation 9**

M 11-8 First submission PS 5 + solution

T 11-9 ** Lecture 17 ** DHS 6.1-6.6, 6.8 Multilayer Neural Nets

Project Plan Due if you're using class-provided data

R 11-11 ** Veteran's Day Holiday - No Class **

F 11-12 ** Recitation 10**

Final submission PS 5

T 11-16 ** Lecture 18 ** Feature Selection, webpage shown in class: http://ro.utia.cz/fs/fs_guideline.html DHS reading: Entropy/Mutual information A.7, Decision Trees, 8.1-8.4

R 11-18 ** Lecture 19 ** Project Progress Presentations/ Critique Day/ Attendance counts toward grade today

F 11-19 ** Recitation 11**

M 11-22 First submission PS 6 + (solution + code)

T 11-23 ** Lecture 20 ** Project Progress Presentations/ Critique Day/ Attendance counts toward grade today

W 11-24 Final submission PS 6

R 11-25 ** Thanksgiving Holiday - No Class **

F 11-26 * No Recitation, Thanksgiving Vacation*

T 11-30 ** Lecture 21 ** 5.11 SVM, 9.1-9.2.1 No free lunch, 9.2.3-9.2.5 MDL, Occam; 9.3 Bias and Variance, 9.5 Bagging, Boosting, Active Learning, 9.6 Estimating and Comparing Classifiers, 9.7 Classifier Combination

R 12-2 ** Lecture 22 ** Guest lecture from Dan McDuff: Gaussian Processes. Slides

F 12-3 ** Recitation 12** Project help session - your staff has lots of experience - get our input and help

T 12-7 **Final Project Presentations: ** All students required
to attend: Attendance counts significantly for grade today.

R 12-9 **Final Project Presentations, Last Day of Class:** All students required to attend; attendance counts significantly for grade today.

Staff | Announcements | Assignments| Syllabus | Policies

**Fall 2010 Syllabus: (subject to adjustment)**

Intro to pattern recognition, feature detection, classification

Review of probability theory, conditional probability and Bayes rule

Random vectors, expectation, correlation, covariance

Review of linear algebra, linear transformations

Decision theory, ROC curves, Likelihood ratio test

Linear and quadratic discriminants

Sufficient statistics, coping with missing or noisy features

Template-based recognition, feature extraction

Eigenvector and Fisher linear discriminant analysis

Independent component analysis

Training methods, Maximum likelihood and Bayesian parameter estimation

Linear discriminant/Perceptron learning, optimization by gradient descent

Support Vector Machines

K-Nearest-Neighbor classification

Non-parametric classification, density estimation, Parzen estimation

Unsupervised learning, clustering, vector quantization, K-means

Mixture modeling, Expectation-Maximization

Hidden Markov models, Viterbi algorithm, Baum-Welch algorithm

Bayesian networks

Bagging, boosting

Decision trees, Multi-layer Perceptrons

Optional other topics toward end of term

Staff | Announcements | Assignments | Syllabus | Policies

30% Homework/Mini-projects, due every 1-2 weeks up until 3 weeks before the end of the term. These will involve both programming (Matlab) and non-programming assignments.

New homework submission and grading policy:

It is NOT ALLOWED to look at old homeworks before handing in your first pass at your homework. If we catch you doing this you will get a zero on the homework. This year we want to help you get as far as possible on your own or working with the other students collaboratively, and then after you hand in that work, we will hand out the solutions and let you fix your homework and improve your grade. This policy has multiple goals: (1) it forces you to learn by figuring things out, which develops your abilities better, (2) you are allowed to collaborate, which also facilitates discussion-based learning and meeting other people in the class (although you should not copy anyone's answers - and don't let them copy yours - write your own please, (3) it levels the playing field and is fair to those who don't have access to old homeworks, and (4) if many of you don't do well, it shows us we need to teach better, giving us important feedback so we can make the course better. Finally, because looking at solutions can raise your grade and be educational, we will hand out the solutions in a fair way to everyone after you hand your problem set in, and if you fix your answers by the grading deadline, we will regrade your homework and give you the average of the two grades, before and after the solutions.

Here is how it works: Homework will be due by 5pm on the due date. On the day when the homework is due, submit the best efforts you have made and photocopy or otherwise make a copy for yourself of what you handed in. We will hand out the homework solution and you will have a chance to refine your work given the solution. Submit your revised homework by the date and time specified (please do not be late - it is not fair to our graders and TAs). Do not copy your revised answers directly from the solution sheet: directly copying answers from the solution sheet will not be counted. Our goal is to help you learn and understand the ideas in the course materials. Optimizing your grades in grad school is not as important as getting depth of knowledge and developing your learning skills. Feel free to add comments to your solution to make the graders aware of specific things you have learnt from the 'second pass' - this will improve your chance of achieving the maximum bonus.

30% Project with approximate due dates:

- T 11-2 All course-provided data available by today . R 11-4 One page plan is due today if using your own data.
- T 11-9 One page plan is due today if using course-provided data.
- R 11-18 Project progress presentation/critique day
- T 11-23 Project progress presentation/critique day
- T 12-7 Project Final Presentations. We probably need this extra day for presentations, and if so, all students are expected to attend today.
- R 12-9 Project Final Presentations. All students are expected to attend today.

25% Midterm Quiz: R 10-28 (Drop Date is W 11-17)

15% Your presence and interaction in lectures (especially your
presence during the two days of project critiques and two days of
final project presentations, which is 10%), in recitation, and with
the staff outside the classroom.

The midterm will be closed-book, but we will allow a cheat sheet.