Multi-class Logistic Regression

Multi-class Logistic Regression#

We demonstrate multi-class logistic regression.

Handwritten Digits#

We will demonstrate multi-class logistic regression using a handwritten digits dataset. The data are in scikit-learn, and our example follows very closely this example.

First, let’s load the dataset.

from sklearn import datasets

digits = datasets.load_digits()

print(digits.DESCR)

.. _digits_dataset:

Optical recognition of handwritten digits dataset
--------------------------------------------------

**Data Set Characteristics:**

    :Number of Instances: 1797
    :Number of Attributes: 64
    :Attribute Information: 8x8 image of integer pixels in the range 0..16.
    :Missing Attribute Values: None
    :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
    :Date: July; 1998

This is a copy of the test set of the UCI ML hand-written digits datasets
https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits

The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.

Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.

For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
1994.

.. topic:: References

  - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
    Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
    Graduate Studies in Science and Engineering, Bogazici University.
  - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
  - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
    Linear dimensionalityreduction using relevance weighted LDA. School of
    Electrical and Electronic Engineering Nanyang Technological University.
    2005.
  - Claudio Gentile. A New Approximate Maximal Margin Classification
    Algorithm. NIPS. 2000.

The images are in a 3D array:

print(digits.images.shape)

(1797, 8, 8)

Each row of this array is an 8x8 image (which is just a matrix). Here is the first image as just numbers:

print(digits.images[0])

[[ 0.  0.  5. 13.  9.  1.  0.  0.]
 [ 0.  0. 13. 15. 10. 15.  5.  0.]
 [ 0.  3. 15.  2.  0. 11.  8.  0.]
 [ 0.  4. 12.  0.  0.  8.  8.  0.]
 [ 0.  5.  8.  0.  0.  9.  8.  0.]
 [ 0.  4. 11.  0.  1. 12.  7.  0.]
 [ 0.  2. 14.  5. 10. 12.  0.  0.]
 [ 0.  0.  6. 13. 10.  0.  0.  0.]]

These numbers correspond to the darkness of each pixel. The greater the value the darker the pixel. Here is how we can visualie the first image:

fig, ax = plt.subplots()
ax.imshow(
    digits.images[0],
    cmap=plt.cm.gray_r,
    interpolation='nearest'
);

../_images/40ecc4d11de01a727475973c3ca4ba867fda4b6af329c3d9a82572f792e7145c.svg

That’s clearly a 0. Now each one of the images comes we predetermined labels that we can use to train models. Here is where you can find the labels:

print(digits.target)

[0 1 2 ... 8 9 8]

and notice that the first label is a 0, which is great. Let’s now plot several images just to gain some intuition about them:

fig, axes = plt.subplots(4, 4)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(
    axes.flatten(),
    images_and_labels[:16]
):
    ax.set_axis_off()
    ax.imshow(
        image,
        cmap=plt.cm.gray_r,
        interpolation='nearest'
    )
    ax.set_title(f'Training: {label}')
plt.tight_layout()
sns.despine(trim=True);

../_images/83cb4da3a582f702d234153282f4255516faac66f2d5cd30ee8d8c736de9215e.svg

We will apply the multi-class logistic regression classifier with 64 linear features, one per pixel. First, we will vectorize the images. We turn them from \(8\times 8\) matrices to \(64\)-dimensional arrays. Here is how we can do this:

n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
print(data.shape)

(1797, 64)

Let’s split the dataset into training and validation sets. We will use the functionality of scikit-learn for this:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    data,
    digits.target,
    test_size=0.5,
    shuffle=True
)

The model we are going to fit is:

\[ p(y=k|\mathbf{x}, \mathbf{W}) = \operatorname{softmax}_k\left(\mathbf{w}_1^T\mathbf{x},\dots,\mathbf{w}_K^T\mathbf{x}\right), \]

where \(\mathbf{x}\) is the vectorized version of the image. Let’s do it:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=2000,
    penalty=None,
    fit_intercept=True
)
model.fit(X_train, y_train);

Here is how you can get the matrix of all weights \(\mathbf{W}\):

print(model.coef_.shape)

(10, 64)

Here are point predictions for (picking the label with the highest probability):

predicted = model.predict(X_test)
print('#\tTruth\tPrediction')
print('-' * 26)
for i, (yt, yp) in enumerate(zip(y_test, predicted)):
    print(f'{i}\t{yt}\t{yp}')

Show code cell output Hide code cell output

#	Truth	Prediction
--------------------------
6	6
6	6
7	7
2	2
1	1
8	9
0	0
7	7
5	5
1	1
7	7
5	5
8	8
6	6
5	6
1	1
7	7
0	0
0	0
7	7
2	2
0	0
1	1
7	7
1	1
8	6
2	2
6	6
4	4
4	4
4	4
7	7
5	5
8	8
6	6
5	5
8	8
4	4
0	0
6	6
5	5
3	3
1	4
4	4
7	7
0	0
9	3
6	6
5	5
7	7
2	2
1	1
1	1
8	8
8	8
6	6
0	0
5	5
9	9
7	7
5	5
8	1
6	6
1	1
1	1
7	7
3	3
8	8
2	2
3	3
0	0
9	9
5	5
4	4
3	3
5	5
0	0
6	6
5	5
1	1
3	3
4	4
1	1
1	1
2	2
5	5
8	8
5	5
2	2
1	1
8	1
8	8
6	6
0	0
1	1
8	8
5	5
1	1
6	6
1	1
7	7
2	2
7	7
9	9
2	2
4	4
7	7
1	1
5	5
6	6
5	5
3	3
1	1
6	6
1	1
2	1
4	4
6	6
0	0
4	4
8	8
2	2
4	4
2	2
7	7
1	1
7	7
3	3
8	8
1	1
5	5
0	0
1	1
3	3
5	5
7	7
0	0
0	0
2	2
1	1
9	9
4	4
0	0
1	1
0	0
5	5
4	4
8	1
2	3
6	6
0	0
5	5
0	0
4	4
3	3
1	1
7	7
8	8
6	6
8	8
7	7
7	7
1	1
3	3
6	6
4	4
8	8
1	1
6	6
7	7
7	7
2	2
9	9
3	3
4	4
2	2
5	5
6	6
9	9
6	6
0	0
6	6
4	4
3	3
8	8
6	6
6	6
4	4
4	4
7	7
3	3
7	7
2	2
1	1
6	6
6	6
8	8
7	7
5	5
7	7
1	1
7	7
9	9
9	9
2	2
5	5
6	6
1	1
9	9
6	6
6	6
2	2
3	3
5	5
7	7
2	2
9	9
2	2
3	3
3	3
7	7
1	1
0	0
0	0
4	4
3	3
8	8
3	3
4	4
5	5
7	7
1	1
5	5
1	1
9	9
8	8
2	1
6	6
2	2
8	1
2	2
1	1
1	1
8	8
5	5
7	7
7	4
0	0
9	9
1	1
7	7
0	0
2	2
3	3
1	1
8	6
0	0
9	9
3	3
7	7
4	7
5	5
2	2
9	9
4	4
7	7
2	2
5	5
9	9
2	2
1	1
2	2
8	8
3	3
7	7
3	3
4	4
7	7
0	0
8	8
0	0
2	2
1	1
1	1
0	0
1	1
2	2
1	1
9	9
0	0
2	2
2	2
8	8
5	5
1	9
6	6
4	4
4	4
0	0
9	9
5	3
6	6
7	7
8	2
3	3
7	1
8	8
7	7
5	5
0	0
9	9
5	5
0	0
3	3
6	6
3	3
0	0
8	8
3	3
3	3
4	4
9	9
7	7
2	2
4	4
4	4
0	0
8	8
2	2
9	9
3	3
6	6
9	5
6	6
5	5
7	7
8	8
5	5
5	5
7	7
9	9
0	0
5	5
8	6
3	3
0	0
5	9
9	9
0	0
8	8
8	6
0	0
0	0
3	3
5	5
1	1
9	9
6	6
9	9
4	4
0	0
4	4
5	5
4	4
9	9
4	4
3	3
4	4
9	9
9	9
6	6
9	9
5	5
9	9
0	0
8	1
9	9
6	6
7	7
0	0
9	9
7	7
3	3
9	9
5	5
3	3
5	5
6	6
9	9
2	2
3	3
7	7
1	1
7	7
0	0
7	7
0	0
3	3
3	3
0	0
0	0
1	1
0	0
4	4
5	5
1	1
6	6
8	8
3	3
0	0
8	8
4	4
8	8
4	4
1	1
2	1
0	0
4	4
3	3
3	3
3	3
6	6
1	1
6	6
2	2
5	5
2	2
5	5
2	2
1	1
9	9
7	7
7	7
0	0
2	2
6	6
8	8
9	9
5	5
3	3
9	9
6	6
2	2
6	6
6	6
6	6
9	9
2	1
6	6
9	9
3	3
7	7
7	7
8	6
4	4
3	3
8	8
4	4
1	1
8	8
6	6
9	9
3	3
1	1
5	5
8	8
0	0
3	3
3	3
9	9
8	8
7	7
3	3
7	7
8	1
0	0
6	6
0	0
0	0
8	8
3	3
8	8
6	6
3	3
4	4
0	0
5	5
9	9
9	9
1	1
6	6
8	8
1	1
5	5
8	8
8	8
1	1
1	1
0	0
6	6
7	7
4	4
8	8
0	0
0	0
0	0
8	6
2	2
4	4
8	8
4	4
0	0
0	0
7	7
8	8
9	8
5	5
8	8
6	6
5	5
3	3
9	9
8	8
5	5
7	7
0	0
2	2
5	5
5	5
3	8
9	9
5	5
8	1
5	5
2	2
1	1
6	6
6	6
1	1
1	1
6	6
7	7
9	9
0	0
1	1
5	5
7	7
3	3
2	2
2	2
7	7
9	9
6	6
7	7
4	4
7	7
2	2
6	6
9	9
5	5
5	5
3	3
9	9
9	9
0	0
6	6
3	3
5	5
6	6
7	7
3	3
8	8
3	3
8	8
3	3
9	9
5	5
2	2
8	8
2	2
1	1
1	1
6	6
8	8
2	2
6	6
8	8
7	7
9	9
7	7
9	9
6	6
0	0
8	8
4	4
8	8
5	5
8	8
2	2
7	7
3	3
7	7
3	3
5	5
1	1
3	3
2	2
0	0
7	7
6	6
1	1
5	5
3	3
0	0
8	8
3	3
1	1
4	4
0	0
3	3
3	3
3	3
8	8
7	7
2	2
6	6
9	9
1	1
9	9
2	2
8	8
2	2
7	7
6	6
9	9
6	6
7	7
2	2
0	0
0	0
2	2
6	6
8	8
3	3
8	8
1	1
3	3
9	9
4	4
2	2
9	8
9	8
3	3
9	9
1	1
2	2
5	7
4	4
8	8
0	0
3	3
3	3
0	0
1	1
0	0
4	4
4	4
0	0
4	4
7	7
0	0
9	9
4	4
6	6
1	1
5	5
5	9
6	6
6	6
8	8
3	3
9	9
4	4
3	3
9	9
1	1
3	3
3	3
8	8
6	6
7	7
3	8
2	1
5	5
2	2
2	1
6	6
6	6
7	7
3	3
0	0
8	8
8	8
0	0
7	7
0	0
6	6
4	4
0	0
3	3
1	1
1	1
7	7
9	9
5	5
4	4
4	4
2	2
4	4
8	1
1	1
8	8
7	3
5	5
5	5
2	2
3	3
6	6
3	3
0	0
7	7
5	5
6	6
7	4
0	0
2	2
0	0
5	5
5	5
5	5
2	2
6	6
1	1
7	7
1	1
7	7
9	9
2	2
4	4
8	6
5	5
4	4
4	4
9	9
0	0
5	5
5	5
1	1
6	6
8	8
9	9
7	7
6	6
8	8
4	4
6	6
1	1
4	4
5	5
9	9
9	9
4	4
4	4
0	0
3	3
8	8
8	8
2	2
4	4
7	7
9	9
9	9
8	8
8	8
5	5
9	9
9	9
4	4
3	3
2	3
2	2
9	9
9	9
8	5
9	6
3	3
5	5
2	2
9	9
4	4
2	2
8	8
8	8
4	4
6	6
5	5
8	1
6	6
6	6
3	3
9	9
2	2
7	7
0	0
4	4
7	7
9	9
5	5
2	2
8	8
3	3
2	2
3	3
7	7
1	1
2	2
0	0
0	0
4	4
7	7
1	1
6	6
5	5
4	4
3	3
4	4
4	4
4	4
3	3
8	8
3	3
9	9
3	3
5	5
0	0
8	8
5	5
4	4
9	9
2	2
0	0
6	6
9	9
8	8
4	4
9	9
0	0
0	0
4	4
1	1
1	1
2	2
4	4
1	1
0	0
9	9
9	6
8	8
0	0
7	7
2	2
0	0
7	7
1	1
6	6
9	9
1	1
4	4
8	8
4	4
8	8
6	6
6	6
6	6
4	4
2	2
8	8
7	7
6	6
7	7
6	6
0	0
5	5
1	1
5	5
0	0
7	7
4	4
2	2

But we can also make probabilistic predictions:

../_images/75eb981810ddb48122413afe312ec83cdaf2bce72c108ae9aa58417d290ea03b.svg

Scikit-learn can run many accuracy metrics at once for you. Here is everything, including the confusion matrix:

from sklearn import metrics
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

print(f"Classification report for model {model}")
print(
    metrics.classification_report(y_test, predicted)
)

cm = confusion_matrix(
    y_test,
    predicted,
    labels=model.classes_
)

disp = ConfusionMatrixDisplay(
    confusion_matrix=cm,
    display_labels=model.classes_
)
disp.plot();

Classification report for model LogisticRegression(max_iter=2000, penalty=None)
              precision    recall  f1-score   support

           0       1.00      1.00      1.00        95
           1       0.84      0.98      0.90        87
           2       0.99      0.90      0.94        83
           3       0.95      0.98      0.96        89
           4       0.96      0.99      0.98        83
           5       0.98      0.94      0.96        90
           6       0.90      1.00      0.95        93
           7       0.98      0.96      0.97        92
           8       0.94      0.81      0.87        99
           9       0.95      0.92      0.94        88

    accuracy                           0.95       899
   macro avg       0.95      0.95      0.95       899
weighted avg       0.95      0.95      0.95       899

../_images/72869ed52b1f11b53922d17c60c61895d6ce6ce58b1ced0f34d6a4f3233ad4fb.svg

Questions#

Look at the precision matrix carefully and identify the digits for which the most mistakes are made. Why does this happen? Write code to visualize some of the wrong predictions.

Multi-class Logistic Regression

Contents

Multi-class Logistic Regression#

Handwritten Digits#

Questions#