The first step, of course, was to go through all the training data by hand and classify it. The training data was all captured from a video tape of people sitting in front of the camera performing a task, so the expressions were genuine and not on purpose. I classified it into two groups, one of obvious smiles and one of everything else. I had some trouble doing this because a few of the subjects did not change expression much over the course of their interaction, so I was forced to put all their data in one category or another, or leave it out. For the first round I left out data that I couldn't decide where it belonged.
Once the training set was ready, I began some simple testing. I first computed the first 100 PCA coefficients using the entire training set to determine the basis; then I used Fisher to find the basis that maximally discriminated the 2 groups. The 2 groups were fairly well discriminated using this method, so ran a test to get a sense for recognition accuracy. For a testing set I drew randomly from my training samples, and removed those samples from the training pool. They were used in the computation of the PCA basis, but not in determining the basis using Fisher. The Results were good, even using simply a linear decision boundary (the average of the two training classes sample means):
Yellow: Smiles -- Green:NonSmiles
Red:Smiles Test Set -- Blue:NonSmiles Test Set
The average accuracy of classification with this method was 92%.
Since this is supposed to be used in an online application, it is impractical to regenerate the PCA basis and recalculate the best basis with Fisher for every classification, so I tried again but this time removing the test data right at the beginning, so it is not used in the calculation of the PCA basis. I expected a slight drop in performance, but the average classification stayed right at 92%.
Next I wanted to test how well this method would work for people not in the training set. To do this, I removed all the training data of one of the subjects right from the start, and calculated the PCA basis and then the Fisher basis using the other subject. The results here varied greatly, which I believe reflects the data I was training with; since I only included images that I could classify by hand, some subjects had many images in the training set while others had just a handful. Also, with only 4 subjects in the training set it is easy to imagine that it is hard for the classifier to generalize to new subject. The average recognition of a novel subject's smiles was 70%, but the values ranged widely, one as low as 36%. Given the size of the training data I had, I decided to stick to the problem of recognizing smiles of those people in the training set.