Geek asks and answers

The role of class_weight in loss functions for linearSVC and LogisticRegression

I am trying to figure out what the formula of the loss function is and how I can calculate it manually when class_weight='auto'in the case svm.svc, svm.linearSVCand linear_model.LogisticRegression.

For balanced data, for example, you have a trained classifier: clf_c. Logistic loss should be (am I right?):

def logistic_loss(x,y,w,b,b0):
    '''
    x: nxp data matrix where n is number of data points and p is number of features.
    y: nx1 vector of true labels (-1 or 1).
    w: nx1 vector of weights (vector of 1./n for balanced data).
    b: px1 vector of feature weights.
    b0: intercept.
    '''
    s = y
    if 0 in np.unique(y):
        print 'yes'
        s = 2. * y - 1
    l = np.dot(w, np.log(1 + np.exp(-s * (np.dot(x, np.squeeze(b)) + b0))))
    return l

I realized what logisticRegression has predict_log_proba(), which gives you exactly this when the data is balanced:

b, b0 = clf_c.coef_, clf_c.intercept_
w = np.ones(len(y))/len(y)
-(clf_c.predict_log_proba(x[xrange(len(x)), np.floor((y+1)/2).astype(np.int8)]).mean() == logistic_loss(x,y,w,b,b0)

Note. np.floor((y+1)/2).astype(np.int8)just maps y = (- 1,1) to y = (0,1).

But this does not work when the data is unbalanced.

, , (, logisticRegression) ( ), class_weight=None , , class_weight='auto'. ( ) .

, class_weight = 'auto' ? class_weight = {-1 : (y==1).sum()/(y==-1).sum() , 1 : 1.} , , class_weight = {-1 : 1./(y==-1).sum() , 1 : 1./(y==1).sum()}?

. , , . .

+4
1

class_weight

class_weight='auto', :

class_weight = {-1 : (y == 1).sum() / (y == -1).sum(), 
                1 : 1.}

, , .

, , class_weight="auto", . : weight = none auto svm scikit learn.

:

, , ( ), , 1, , (y), , , . .

, ;).

0.18. , class_weight='balanced'.

"" .

:

"" y : n_samples / (n_classes * np.bincount(y)).

np.bincount(y) - i, i.

:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.utils import compute_class_weight

n_classes = 3
n_samples = 1000

X, y = make_classification(n_samples=n_samples, n_features=20, n_informative=10, 
    n_classes=n_classes, weights=[0.05, 0.4, 0.55])

print("Count of samples per class: ", np.bincount(y))
balanced_weights = n_samples /(n_classes * np.bincount(y))
# Equivalent to the following, using version 0.17+:
# compute_class_weight("balanced", [0, 1, 2], y)

print("Balanced weights: ", balanced_weights)
print("'auto' weights: ", compute_class_weight("auto", [0, 1, 2], y))

:

Count of samples per class:  [ 57 396 547]
Balanced weights:  [ 5.84795322  0.84175084  0.60938452]
'auto' weights:  [ 2.40356854  0.3459682   0.25046327]

: ?

, , .

SVC linearSVC

C _weight [i] * C SVC.

, svm, .

, . , liblinear libsvm, .

, class_weight , predict_proba. , .
, , , , ( ):

lr = LogisticRegression(class_weight="auto")
lr.fit(X, y)
# We get some probabilities...
print(lr.predict_proba(X))

new_lr = LogisticRegression(class_weight={0: 100, 1: 1, 2: 1})
new_lr.fit(X, y)
# We get different probabilities...
print(new_lr.predict_proba(X))

# Let cheat a bit and hand-modify our new classifier.
new_lr.intercept_ = lr.intercept_.copy()
new_lr.coef_ = lr.coef_.copy()

# Now we get the SAME probabilities.
np.testing.assert_array_equal(new_lr.predict_proba(X), lr.predict_proba(X))

, .

+4

More articles:


All Articles