第一章.Classification -- 02.Loss Functions for Classification翻译

So now that you understand the basics of classification, let’s talk about loss functions,

because that determines the major difference between a lot of machine learning methods.

Okay so how do we measure classification error?

Well, one very very simple way to do it is to use just a fraction of times our predictions are wrong.

So just the fraction of times the sign of f(x) is not equal to the truth y, and I can write it like that, okay? The issue with this particular way of measuring classification error is that if you want to try to minimize this, you could run into a lot of trouble because it’s computationally hard to minimize this thing.

So let’s give you the geometric picture here: the decision boundary is this line right here,

and f being positive is here and f being negative is here, and the red points are all misclassified.

And now what I’m going to do is something you might not be expecting which is that I’m going to move all the correctly classified points across the decision boundary to one side,

and all the misclassified points to the other side.

There they go, we’ll move all the misclassified points across the decision boundary and then all the correctly classified ones.

Okay, so I’m glad I did that, now I have the ones we’ve got correct over here and the ones we got wrong over there. And now the labels on this plot are wrong because we changed it, so it’s actually something like that.

Okay so over here, on the right either f is positive and y is also positive, so y-f is positive,

or they’re both negative so the product is positive again.

And then over here on the left, we have cases where the sign of f is different from y,

so the product is negative. And then the ones I suffer a penalty for are all these guys over there.

Okay so let’s keep that image in mind over here, and then I’ll put the labels up.

Now this function tells us what kind of penalty we’re going to issue for being wrong.

Okay so right now, if y times f is positive, it means we got it right and we lose 0 points.

And if we get it wrong, the point is on the wrong side and so we lose one point.

Now I’m just going to write this function another way which is like this,

okay, so we lose one

point if y-f is less than 0, and otherwise we lose no points. And this is the classic 0-1 loss.

It just tells you whether your classifier is right or wrong.

And then this thing is called a loss function, and there are other loss functions too,

and this one’s nice because – but it’s problematic because it’s not smooth,

and we have issues with things that are not smooth in machine learning.

So let’s try some more loss functions.

So while we’re doing this,

just keep in mind that these points over here are the ones that are very wrong,

because they’re on the wrong side of the decision boundary,

but they’re really far away from it too. And these points are wrong, but they’re not as bad;

they’re on the wrong side of the decision boundary but they’re pretty close to it.

And then we’ll say these points are sort of correct, and we’ll say these points are very correct.

And what we’ll really like to have are loss functions that don’t penalize the very correct ones,

but the penalty gets worse and worse as you go to the left.

But maybe we can use some other loss function,

something that – you know, maybe we get a small penalty for being sort of correct and then a bigger penalty for being sort of wrong and then a huge penalty for being very wrong.

Something that looks like this would be a deal.

So again,

this is – the horizontal axis is y times f, and this red one is 1 if y disagrees with the sign of f, and then the other curves are different loss functions and they’re actually for different machine learning algorithms.

And again, just keep in mind that on the right

– these are points that are on the correct side of the decision boundary,

they don’t suffer much penalty and on the left,

these are points that are incorrectly classified and they suffer more penalty.

This is the loss function that AdaBoost uses.

AdaBoost is one of the machine learning methods that we’ll cover in the course.

And this is the loss function that support vector machines use;

it’s a line, and then it’s another flat line.And this is the loss function for logistic regression, and we’re going to cover all three of these.

Now I’m going to write this idea about the loss functions in notation on the next slide.

Okay so start here:

the misclassification error is the fraction of times that the sign of f is not equal to the truth y that’s this.

I can rewrite it this way, okay, the number of times y times f is less than 0.

And then we’ll upper-bound this by these loss functions.

Okay, so then what is a good way to try to reduce the misclassification error which is that guy?

Well you could just try to minimize the average loss.

So if you had a choice of functions f, you could try to choose f to minimize this thing,

which hopefully would also minimize this but in a computationally easier way.

So here’s your first try for a machine learning algorithm.

Just choose the function f to minimize the average loss. And this seems like a good idea, right?

Well it is, and that’s what most machine learning methods are based on,

and how to do this minimization over models to get the best one,

that involves some optimization techniques, which go on behind the scenes.

But there’s one more thing I didn’t quite tell you,

which is that we want to do more than have a low training error.

We want to predict well on data that we haven’t seen before.

We want to, you know, generalize the new points, and that’s why we need statistical learning theory, because this algorithm that I showed you – that’s not right, and you’ll see why.

It’s pretty good,

but it’s missing this key element that tells – that encourages the model to stay simple and not over-fit.

So I’ll talk about statistical learning theory shortly.