\underline{Problem Set 3}

MAS 622J/1.126J: Pattern Recognition and Analysis

Due: 5:00 p.m. on October 12

[Note: All instructions to plot data or write a program should be carried out using Matlab. In order to maintain a reasonable level of consistency and simplicity we ask that you do not use other software tools.]

If you collaborated with other members of the class, please write their names at the end of the assignment. Moreover, you will need to write and sign the following statement: “In preparing my solutions, I did not look at any old homeworks, copy anybody’s answers or let them copy mine.”

Problem 1: Discriminability and ROC [20 points]

Please download the datasets (dataset1.txt, dataset2.txt, dataset3.txt, dataset4.txt) from the course website. Each dataset includes 1-D data samples from two classes w_{1} and w_{2}. The first column and the second column of each dataset specify 1000 samples from class w_{1} and 1000 samples from class w_{2}, respectively.

a. For each dataset, compute the discriminability d′ = \begin{matrix}| & μ_{2}−μ_{1} & | \\ √ & \overline{σ_{1}^{2}+σ_{2}^{2}} \end{matrix} where μ_{1} and σ_{1} are the mean and

standard deviation of the distribution of class w_{1}, and μ_{2} and σ_{2} are the mean and standard deviation of the distribution of class w_{2}.

Solution: Please refer to the Matlab code for (b). Approximately, For dataset1: μ_{1} ≃ 10, μ_{2} ≃ 12, σ_{1} ≃ 1, σ_{2} ≃ 1, d′ ≃ 1.39 For dataset2: μ_{1} ≃ 10, μ_{2} ≃ 12, σ_{1} ≃ 4, σ_{2} ≃ 1, d′ ≃ 0.46 For dataset3: μ_{1} ≃ 10, μ_{2} ≃ 12, σ_{1} ≃ 1, σ_{2} ≃ 4, d′ ≃ 0.45 For dataset4: μ_{1} ≃ 10, μ_{2} ≃ 12, σ_{1} ≃ 4, σ_{2} ≃ 4, d′ ≃ 0.31

b. Now we compute the ROC curve for each dataset. Please plot four ROC curves in the same figure. To do this, we approximate P_{TP} = P(x > x^{∗}|x ∈ w_{2}) by N(x > x^{∗}|x ∈ w_{2})/1000, and P_{FP} = P(x > x^{∗}|x ∈ w_{1}) by N(x > x^{∗}|x ∈ w_{1})/1000. Here, for i =1 or 2, N(x > x^{∗}|x ∈ w_{i}) is denotes the number of samples in class w_{i} whose value is greater than x^{∗}. Note that N doesn’t denote a normal distribution!

Solution: Changing the classifier value (x^{∗}) from −∞ to ∞, we can plot the ROC curve for each dataset. Note that when x^{∗} = −∞, P_{TP} = 1 and P_{FP} = 1 (corresponding to the uppermost/rightmost position in the ROC curve), and when x^{∗} = ∞, P_{TP} = 0 and P_{FP} = 0 (corresponding to the lowermost/leftmost position in the ROC curve).

clear all

num_samples = 1000;  % number of test samples from each class
sprintf('dataset1:')
dd = load('dataset1.txt');
% test samples from class 1
x1 = dd(:,1);
% test samples from class 2
x2 = dd(:,2);
Pd_save = [];
Pf_save = [];
% x_c is a classifier
for x_c = -10:0.5:30
    % number of TP (True Positive) samples
    nTP = length(find(x2 > x_c));
    % approximate probability of hit, P(x > x_c | x in w=2)
    Pd = nTP / num_samples;
    % number of FP (False Positive) samples
    nFP = length(find(x1 > x_c));
    % approximate probability of false alarm, P(x > x_c | x in w=1)
    Pf = nFP / num_samples;
    Pd_save = [Pd_save; Pd];
    Pf_save = [Pf_save; Pf];
end

mean1 = mean(x1)
mean2 = mean(x2)
sig1 = std(x1)
sig2 = std(x2)
sig = sqrt(sig1^2 + sig2^2)
discrim = abs(mean1 - mean2)/sig % d'
figure
hold on
plot(Pf_save, Pd_save, 'r')

sprintf('dataset2:')
dd = load('dataset2.txt');
% test samples from class 1
x1 = dd(:,1);
% test samples from class 2
x2 = dd(:,2);

Pd_save = [];
Pf_save = [];
% x_c is a classifier
for x_c =  -10:0.5:30
    % number of TP (True Positive) samples
    nTP = length(find(x2 > x_c));
    % approximate probability of hit, P(x > x_c | x in w=2)
    Pd = nTP / num_samples;
    % number of FP (False Positive) samples
    nFP = length(find(x1 > x_c));
    % approximate probability of false alarm, P(x > x_c | x in w=1)
    Pf = nFP / num_samples;
    Pd_save = [Pd_save; Pd];
    Pf_save = [Pf_save; Pf];
end
mean1 = mean(x1)
mean2 = mean(x2)
sig1 = std(x1)
sig2 = std(x2)
sig = sqrt(sig1^2 + sig2^2)
discrim = abs(mean1 - mean2)/sig % d'
plot(Pf_save, Pd_save, 'b')
sprintf('dataset3:')
dd = load('dataset3.txt');
% test samples from class 1
x1 = dd(:,1);

% test samples from class 2
x2 = dd(:,2);
Pd_save = [];
Pf_save = [];
% x_c is a classifier
for x_c = -10:0.5:30
    % number of TP (True Positive) samples
    nTP = length(find(x2 > x_c));
    % approximate probability of hit, P(x > x_c | x in w=2)
    Pd = nTP / num_samples;
    % number of FP (False Positive) samples
    nFP = length(find(x1 > x_c));
    % approximate probability of false alarm, P(x > x_c | x in w=1)
    Pf = nFP / num_samples;
    Pd_save = [Pd_save; Pd];
    Pf_save = [Pf_save; Pf];
end
mean1 = mean(x1)
mean2 = mean(x2)
sig1 = std(x1)
sig2 = std(x2)
sig = sqrt(sig1^2 + sig2^2)
discrim = abs(mean1 - mean2)/sig % d'
plot(Pf_save, Pd_save, 'g')
sprintf('dataset4:')
dd = load('dataset4.txt');
% test samples from class 1
x1 = dd(:,1);
% test samples from class 2
x2 = dd(:,2);

Pd_save = [];
Pf_save = [];