SLEP025: Losing Accuracy in Scikit-Learn

[lorentzenchr]

What

This SLEP is for finding a consensus on how to removing accuracy as a default metric for classifiers which currently is: classifiers.score.

Why

Because accuracy has many severe weaknesses and we even baked a probability threashold of 50% into it.

What else

I hope the irony of the title is not lost by its passive aggressiveness.

This SLEP is based on @amueller's proposal scikit-learn/scikit-learn#28995.

@scikit-learn/core-devs @scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/documentation-team ping

[jjerphan]

Thank you for writing the SLEP.

While discussing the rectification of this design choice is appropriate (I really think it is valuable), I find the title a bit aggressive or at least detrimental to the intent of the proposal (improving scikit-learn's theoretical consistency): how about "Redefining default metrics"?

I think this could be implemented as part of scikit-learn 2.0.

[reshamas]

@lorentzenchr How about: Removing the accuracy metric in scikit-learn scoring

[betatim]

Do we need to choose the same metric for all classifiers?

I think the answer is yes because people will use the results of est1.score(X, y) and est2.score(X, y) to evaluate which one is the better estimator. It seems very hard to educate people that they can't compare scores from different estimators

(This is almost a rhetorical question, but I wanted to double check my thinking)

[lorentzenchr]

Given your assumption that users will continue to compare score results of different estimators, and given that a generally satisfying metric does not exist, the conclusion is to remove the score method.

My currently best choice for a general classifier metric is the skill score (R2) variant of the Brier score. Classifiers and regressors would then have the same metric, which is nice.