mis6324EvaluatingClassifiers

1、MIS 6324Xianjun GengOutline - Why and How to Evaluate? Why? Choose among multiple classifiers For each classifier, choose among multiple parameter settings How? Heres a list of popular measurements Accuracy % of correct classifications in evaluation data Confusion matrix MSE Lift curve / ROC curve O

2、ther evaluation criteria Speed and scalability Interpretability Robustness2Example: Good or bad credit risk? InputsActualOutputModelsPredictionCorrect/incorrect predictionSingleNo of cardsAgeIncome50KGood/Bad riskGood/Bad risk012811112560000561101X0128111AccuracyInputsActualOutputModelsPredictionCor

3、rect/incorrect predictionSingleNo of cardsAgeIncome50KGood/Bad riskGood/Bad risk012811112560000561101X0128111Number of correct classificationsClassification Accuracy RateNumber of instances in datasetIn this credit risk example, accuracy = 3/4Accuracy can be misleading! An example Model B is a nave

4、model that predicts that no customer will respond. Classification accuracy rate is 99%. But model misclassifies all respondents Very accurate, but useless Can be used as a benchmark. Need to know the source of the error not just the accuracy rate5Example: typical response rate for a mailing campaign

5、 is 1% Two models Model A accurately classifies 90% of instances in the test set. Model B accurately classifies 99% of instances in the test set. Which model would you pick?Confusion Matrix A confusion matrix records the source of error: False positives and false negatives Continuing the mailing cam

6、paign example: suppose 1,000 mails are sent out What is the accuracy rate?6PositiveNegativePositiveTrue positiveFalse negativeNegativeFalse positiveTrue negativePredicted classActualclassRespondDo not respondRespond010Do not Respond0990Predicted classActualclassUsing Confusion Matrix for Model Compa

7、rison Accuracy = (5+950)/1000 = 95.5%7RespondDo not respondRespond55Do not Respond40950RespondDo not respondRespond100Do not Respond90900 Accuracy= ? Below shows the performance of two classifiers. Which one is better based on accuracy?Your business goal decides your choice - Cost Based Classificati

8、on Cost = 5*20 + 40*1 = $1408RespondDo not respondRespond55Do not Respond40950RespondDo not respondRespond100Do not Respond90900 Cost = ? Suppose cost of mailing to a non-responder is $1, and (net) lost revenue of not mailing to a responder is $20. Now from cost perspective, which classifier is bett

9、er?Another Exercise on Cost Based Classification Revenue = ?9RespondDo not respondRespond55Do not Respond40950RespondDo not respondRespond100Do not Respond90900 Revenue = ? Now instead of thinking from cost, think from revenue. Suppose on average each responder, upon receiving a mail, contributes $2

10、1 of revenue. And the mailing cost per customer is $1. Now from revenue perspective, which classifier is better?If the target is a numerical variable - Discretization using cutoffs10If cutoff is 0.50, thirteen records are classified as “1”How many 1s are wrongly classified as 0? Vice versa?If cutoff

11、 is 0.80, how many records are classified as “1”? If the target is a numerical variable - Using Mean Squared Error (MSE)Actual $ spent: a1,a2,anPredictions: p1,p2,pnError ei=(pi - ai)Mean Square Error MSE = e12 + e22 + e1n/nRoot Mean Square Error: Square root of MSEMean Absolute Error (MAE)Relative

12、ErrorIf prediction is 450 and actual was 500, relative error is 10%.11Mean Squared Error (MSE)Model to estimate $ spent on next catalog offer.12MSE = (83-80)2+(131.3-140)2+(178-175)2+(166-168)2 +(117-120)2+(198-189)2 /6= 31.86Exaggerates the effect of outliers normalize data firstLiftLift: improveme

13、nt obtained via modelingI.e., the ratio between the results obtained with and without the predictive model.Similar to “lift” for association rulesExampleMass mail of a promotional offer to 1 million householdsExpected response rate is 0.1% (1000 responders)A data mining tool identifies a subset of 1

14、00k households for which the response rate is 0.4% (400 responders). Lift is 0.4/0.1 = 4.13Constructing Lift ChartsSort cases in the test set based on their predicted probability ofbelonging to the class of interest, in the descending orderRanking of a case is more important than its predicted proba

15、bilityDivide sorted cases into deciles each interval represents 1/10th of the entire sampleFor each decile, calculate success rate # of correct predictions divided by the # of cases in the decileFor each decile, calculate lift factor Success rate of decile divided by success rate of the entire test

16、set14Constructing Lift Charts step 115RankPredicted prob. of responseActual Class10.95Yes20.93Yes30.93No40.88Yes50.86Yes60.85Yes70.82Yes80.80Yes90.80NoSort records in the test set based on their predicted probability of responseWhy?16Decile# of Cases # of ResponsesResponse Rate120,000700.35%220,0004

17、00.20%320,000300.15%420,000200.10%520,000150.08%620,000100.05%720,00050.03%820,00050.03%920,00030.02%1020,00020.01%Constructing Lift Charts step 2Divide records cases into deciles each interval represents 1/10th of the entire sampleResponse rate drops as # of mailings increaseWHY?Constructing Lift C

18、harts step 317Two extreme points in this plot: No solicitations sent no respondentsEveryone receives offers 1,000 respondentsHow many letters would you like to send out?Plot # of responses as a function of sample size (e.g., # of mails sent)Receiver Operating Characteristic (ROC) CurvePlots performance of a classifier in terms of its true positive rate (vertical axis) and false positive rate (horizontal axis)True positive rate= # true positives / (# true positives + # false negatives)(= Sensitivity)False positive rate = # false positives /