Let’s talk about another way to produce ROC curves.
So ROC curves can be produced in two ways: the one that I just showed you,
which is for a single, real valued classifier.
In that case, the ROC curve evaluates the classifier.
There’s another way to create an ROC curve,
where you can use a single algorithm and sweep the imbalance parameter across the full range and trace out an ROC curve to evaluate the algorithm.
So one is a property of a single classifier and the other is a property of a whole algorithm.
So let’s go over this way again and I’ll repeat how to do that.
So let’s say that we have our classifier – our function, f, and it’s increasing along this direction.
And we could place a decision boundary anywhere we wanted along here.
So let’s start from the top and sweep this thing down from top to bottom,
and every time we move it, you record the true positive rate and the false positive rate.
And then you plot all of those CPRs and FPRs on the scatter plot and that’s the ROC curve,
so let’s do it.
So we swing that whole thing that way and we record the true positive rate and false positive rate as we do it.
Okay, and that traces out this whole curve. So that’s the first way to create ROC curves,
and that’s for a single classification model.
And now let’s talk about how to evaluate an algorithm.
Do you remember the c parameter from the imbalanced learning section?
That’s the one that allows you to weight the positives differently than the negatives.
So here, I’m going to sweep through values of the c parameters and fit a machine learning model each time we adjust the c.
And we’re going to start with c being tiny.
So the classifier doesn’t care about the positives at all –
it just cares about getting the negatives right. And guess what?
It got all of them right because it just classified everything as negative.
Then we adjust c and we get this one.
And then we adjust c again and get that decision boundary and this one,
and that one. And as you do this sweep, you get different models each time,
but you also get a true positive rate and a false positive rate each time.
So you could plot those values on a ROC curve.
But this ROC curve evaluates the whole algorithm and not one algorithm.
Now what’s the advantage of using one of these methods over the other?
It’s not really like that – you can’t really say that.
One of these two things is an evaluation measure for a single function and the other is a measure of quality for a whole algorithm.
Usually an algorithm that’s optimized for a specific decision point can actually do better than an algorithm that is optimized for something else.
So usually, you would expect to see the ROC curve for the whole algorithm,
which is optimized at each point on this curve.
You’d expect that to do better than just the single classifier,
though every once in a while you do get a surprise and something weird happens.
But in any case, this is the idea.
If you use the algorithm and fiddle with the class weight parameter,
you’re essentially optimizing for each point along that curve.
Whereas, if you use a fixed classifier, you might be optimizing for one point on that curve,
or for something else entirely, so you wouldn’t expect the ROC curve to be as good.
Okay, so here ends the discussion of ways to produce ROC curves.
让我们讨论另一种生产ROC曲线的方法。
所以ROC曲线可以用两种方式产生:我刚才给你们看的那个，
这是一个单一的，实值的分类器。
在这种情况下，ROC曲线评估分类器。
还有一种创建ROC曲线的方法，
你可以使用一个算法，在整个范围内扫描不平衡参数，并找出一个ROC曲线来评估算法。
一个是单个分类器的属性另一个是整个算法的属性。
我们再看一遍，我重复一下怎么做。
假设有一个分类器，函数f，它沿着这个方向增长。
我们可以在任何我们想要的地方设置一个决策边界。
让我们从上面开始，从上到下，
每次移动的时候，你都记录了正确率和假阳性率。
然后把所有的CPRs和FPRs都画在散点图上这是ROC曲线，
让我们来做它。
所以我们用这种方法来改变整个事情我们记录了真实的正确率和假阳性率。
好的，这就是曲线的轨迹。这是创建ROC曲线的第一种方法，
这是一个分类模型。
现在我们来讨论一下如何计算一个算法。
你还记得不平衡学习部分的c参数吗?
这是允许你以不同于消极的方式来衡量积极因素的方法。
在这里，我将扫过c参数的值并在每次调整c时匹配机器学习模型。
我们从c很小开始。
所以分类器并不关心所有的优点。
它只关心负负的问题。你猜怎么着?
所有这些都是正确的，因为它把所有东西都归为负。
然后我们调整c，得到这个。
然后我们再调整c，得到这个决定边界，
这一个。当你做这个扫描时，每次都得到不同的模型，
但是你也会得到一个真实的正速率和一个假的正速率。
你可以在ROC曲线上画出这些值。
但是这个ROC曲线对整个算法进行了评估，而不是一个算法。
那么使用这些方法中的一个的好处是什么呢?
这不是真的，你不能这么说。
这两种方法中的一种是对单个函数的评估，另一种是对整个算法的质量度量。
通常，针对某个特定决策点进行优化的算法实际上比针对其他内容优化的算法做得更好。
通常情况下，你会看到整个算法的ROC曲线，
在这条曲线上的每一点上都进行了优化。
你会希望它比单一分类器做得更好，
虽然每隔一段时间你都会感到意外，但奇怪的事情发生了。
但无论如何，这是一个想法。
如果你使用这个算法来摆弄类权重参数，
你对曲线上的每一点都进行了优化。
然而，如果你使用固定的分类器，你可能会在曲线上的某一点进行优化，
或者完全是为了别的东西，所以你不会期望ROC曲线会这么好。
好的，这里结束了关于生产ROC曲线的方法的讨论。