Logistic Regression Keras vs Sklearn

In [1]:
import time 
import numpy as np
import pandas as pd 
from matplotlib import pylab as plt
from sklearn.preprocessing import MinMaxScaler 
from sklearn.model_selection import train_test_split 

Getting the data

In [2]:
from keras.datasets import mnist 
# the data, shuffled and split between train and test sets 
(X_train, y_train), (X_test, y_test) = mnist.load_data()
Using TensorFlow backend.

Reshaping and normalizing the inputs

In [3]:
input_dim = 784 #28*28 
X_train = X_train.reshape(60000, input_dim) 
X_test = X_test.reshape(10000, input_dim) 
x_train = X_train.astype('float32') 
x_test = X_test.astype('float32') 
x_train /= 255 
x_test /= 255
In [4]:
y_train[y_train != 3] = 0
y_train[y_train == 3] = 1
In [5]:
y_test[y_test != 3] = 0
y_test[y_test == 3] = 1

Convert class vectors to binary class matrices

Build Keras Logistic Regression

  • Settings
In [6]:
batch_size = 256 
nb_epoch = 30
optimizer = "adam"

Without L1 Regularization

In [7]:
from keras import regularizers
from keras.models import Sequential 
from keras.layers import Dense, Activation 
start = time.time()
model = Sequential() 
model.add(Dense(1, input_dim=input_dim, activation='sigmoid', kernel_regularizer=regularizers.l1(0.00))) 
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy']) 
history = model.fit(x_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1, validation_data=(x_test, y_test)) 
pred_test = model.predict(x_test).reshape([-1])>0.5
time_cost = time.time() - start

print("Time Cost of Logistic Regression: ", time_cost)
print("Accuracy: ", np.mean(pred_test == y_test))
/home/r7user1/anaconda2_local/lib/python2.7/site-packages/ipykernel_launcher.py:8: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  
Train on 60000 samples, validate on 10000 samples
Epoch 1/30
60000/60000 [==============================] - 1s 13us/step - loss: 0.2572 - acc: 0.9128 - val_loss: 0.1614 - val_acc: 0.9512
Epoch 2/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1441 - acc: 0.9554 - val_loss: 0.1220 - val_acc: 0.9624
Epoch 3/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1209 - acc: 0.9628 - val_loss: 0.1069 - val_acc: 0.9666
Epoch 4/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1103 - acc: 0.9661 - val_loss: 0.0981 - val_acc: 0.9701
Epoch 5/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1041 - acc: 0.9682 - val_loss: 0.0924 - val_acc: 0.9718
Epoch 6/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1000 - acc: 0.9694 - val_loss: 0.0891 - val_acc: 0.9733
Epoch 7/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0971 - acc: 0.9703 - val_loss: 0.0861 - val_acc: 0.9744
Epoch 8/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0950 - acc: 0.9711 - val_loss: 0.0845 - val_acc: 0.9743
Epoch 9/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0932 - acc: 0.9717 - val_loss: 0.0827 - val_acc: 0.9759
Epoch 10/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0917 - acc: 0.9722 - val_loss: 0.0816 - val_acc: 0.9756
Epoch 11/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0906 - acc: 0.9727 - val_loss: 0.0803 - val_acc: 0.9757
Epoch 12/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0895 - acc: 0.9729 - val_loss: 0.0791 - val_acc: 0.9762
Epoch 13/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0885 - acc: 0.9736 - val_loss: 0.0784 - val_acc: 0.9756
Epoch 14/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0878 - acc: 0.9734 - val_loss: 0.0779 - val_acc: 0.9756
Epoch 15/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0869 - acc: 0.9738 - val_loss: 0.0772 - val_acc: 0.9761
Epoch 16/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0863 - acc: 0.9742 - val_loss: 0.0762 - val_acc: 0.9773
Epoch 17/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0857 - acc: 0.9741 - val_loss: 0.0759 - val_acc: 0.9766
Epoch 18/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0850 - acc: 0.9744 - val_loss: 0.0756 - val_acc: 0.9770
Epoch 19/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0846 - acc: 0.9746 - val_loss: 0.0753 - val_acc: 0.9770
Epoch 20/30
60000/60000 [==============================] - 1s 14us/step - loss: 0.0842 - acc: 0.9749 - val_loss: 0.0746 - val_acc: 0.9776
Epoch 21/30
60000/60000 [==============================] - 1s 16us/step - loss: 0.0837 - acc: 0.9751 - val_loss: 0.0746 - val_acc: 0.9778
Epoch 22/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0833 - acc: 0.9752 - val_loss: 0.0741 - val_acc: 0.9777
Epoch 23/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0829 - acc: 0.9751 - val_loss: 0.0744 - val_acc: 0.9784
Epoch 24/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0825 - acc: 0.9754 - val_loss: 0.0738 - val_acc: 0.9780
Epoch 25/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0823 - acc: 0.9753 - val_loss: 0.0736 - val_acc: 0.9781
Epoch 26/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.0820 - acc: 0.9755 - val_loss: 0.0732 - val_acc: 0.9785
Epoch 27/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0815 - acc: 0.9754 - val_loss: 0.0731 - val_acc: 0.9782
Epoch 28/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0813 - acc: 0.9757 - val_loss: 0.0734 - val_acc: 0.9784
Epoch 29/30
60000/60000 [==============================] - 1s 14us/step - loss: 0.0813 - acc: 0.9761 - val_loss: 0.0726 - val_acc: 0.9786
Epoch 30/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.0809 - acc: 0.9756 - val_loss: 0.0720 - val_acc: 0.9790
('Time Cost of Logistic Regression: ', 21.73831605911255)
('Accuracy: ', 0.979)

With L1 Regularization

In [8]:
start = time.time()
model = Sequential() 
model.add(Dense(1, input_dim=input_dim, activation='sigmoid', kernel_regularizer=regularizers.l1(0.001))) 
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy']) 
history = model.fit(x_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1, validation_data=(x_test, y_test)) 
pred_test = model.predict(x_test).reshape([-1])>0.5
time_cost = time.time() - start

print("Time Cost of Logistic Regression: ", time_cost)
print("Accuracy: ", np.mean(pred_test == y_test))
/home/r7user1/anaconda2_local/lib/python2.7/site-packages/ipykernel_launcher.py:5: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  """
Train on 60000 samples, validate on 10000 samples
Epoch 1/30
60000/60000 [==============================] - 1s 14us/step - loss: 0.3074 - acc: 0.9004 - val_loss: 0.2051 - val_acc: 0.9410
Epoch 2/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1874 - acc: 0.9503 - val_loss: 0.1677 - val_acc: 0.9572
Epoch 3/30
60000/60000 [==============================] - 1s 13us/step - loss: 0.1653 - acc: 0.9589 - val_loss: 0.1530 - val_acc: 0.9624
Epoch 4/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1554 - acc: 0.9620 - val_loss: 0.1452 - val_acc: 0.9663
Epoch 5/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1497 - acc: 0.9639 - val_loss: 0.1405 - val_acc: 0.9670
Epoch 6/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1460 - acc: 0.9654 - val_loss: 0.1374 - val_acc: 0.9694
Epoch 7/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1434 - acc: 0.9663 - val_loss: 0.1350 - val_acc: 0.9698
Epoch 8/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1414 - acc: 0.9672 - val_loss: 0.1328 - val_acc: 0.9707
Epoch 9/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1399 - acc: 0.9675 - val_loss: 0.1315 - val_acc: 0.9717
Epoch 10/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1387 - acc: 0.9683 - val_loss: 0.1303 - val_acc: 0.9719
Epoch 11/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1376 - acc: 0.9688 - val_loss: 0.1293 - val_acc: 0.9727
Epoch 12/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1366 - acc: 0.9690 - val_loss: 0.1289 - val_acc: 0.9724
Epoch 13/30
60000/60000 [==============================] - 1s 13us/step - loss: 0.1358 - acc: 0.9695 - val_loss: 0.1278 - val_acc: 0.9726
Epoch 14/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1351 - acc: 0.9697 - val_loss: 0.1270 - val_acc: 0.9730
Epoch 15/30
60000/60000 [==============================] - 1s 10us/step - loss: 0.1345 - acc: 0.9698 - val_loss: 0.1265 - val_acc: 0.9733
Epoch 16/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1338 - acc: 0.9703 - val_loss: 0.1263 - val_acc: 0.9731
Epoch 17/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1334 - acc: 0.9705 - val_loss: 0.1253 - val_acc: 0.9738
Epoch 18/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1328 - acc: 0.9706 - val_loss: 0.1245 - val_acc: 0.9737
Epoch 19/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1324 - acc: 0.9708 - val_loss: 0.1240 - val_acc: 0.9736
Epoch 20/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1320 - acc: 0.9708 - val_loss: 0.1235 - val_acc: 0.9735
Epoch 21/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1316 - acc: 0.9710 - val_loss: 0.1236 - val_acc: 0.9742
Epoch 22/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1311 - acc: 0.9710 - val_loss: 0.1227 - val_acc: 0.9741
Epoch 23/30
60000/60000 [==============================] - 1s 13us/step - loss: 0.1308 - acc: 0.9714 - val_loss: 0.1228 - val_acc: 0.9747
Epoch 24/30
60000/60000 [==============================] - 1s 13us/step - loss: 0.1304 - acc: 0.9712 - val_loss: 0.1223 - val_acc: 0.9744
Epoch 25/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1301 - acc: 0.9712 - val_loss: 0.1220 - val_acc: 0.9740
Epoch 26/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1298 - acc: 0.9713 - val_loss: 0.1218 - val_acc: 0.9745
Epoch 27/30
60000/60000 [==============================] - 1s 11us/step - loss: 0.1296 - acc: 0.9714 - val_loss: 0.1213 - val_acc: 0.9743
Epoch 28/30
60000/60000 [==============================] - 1s 10us/step - loss: 0.1294 - acc: 0.9710 - val_loss: 0.1219 - val_acc: 0.9745
Epoch 29/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1291 - acc: 0.9714 - val_loss: 0.1208 - val_acc: 0.9738
Epoch 30/30
60000/60000 [==============================] - 1s 12us/step - loss: 0.1289 - acc: 0.9713 - val_loss: 0.1202 - val_acc: 0.9752
('Time Cost of Logistic Regression: ', 21.32376980781555)
('Accuracy: ', 0.9752)
In [9]:
test_acc  = history.history['val_acc']
xc        = history.epoch

plt.figure(figsize=(8,5))
plt.plot(xc, test_acc)
plt.legend(["Validation Set"],fontsize=24)
plt.xlabel("Epoch",fontsize=24)
plt.ylabel("Accuracy",fontsize=24)
Out[9]:
Text(0,0.5,'Accuracy')

SKlearn Logistic Regression

Without L1 Regularization

In [10]:
from sklearn.linear_model import LogisticRegression

start = time.time()
lr = LogisticRegression(penalty="l1", C = 1000000000000000)
lr.fit(x_train, y_train)
pred_test = lr.predict(x_test)
time_cost = time.time() - start

print("Time Cost of Logistic Regression: ", time_cost)
print("Accuracy: ", np.mean(pred_test == y_test))
/home/r7user1/anaconda2_local/lib/python2.7/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
('Time Cost of Logistic Regression: ', 11.690162897109985)
('Accuracy: ', 0.9776)

With L1 Regularization

In [11]:
from sklearn.linear_model import LogisticRegression

start = time.time()
lr = LogisticRegression(penalty="l1", C = 1000)
lr.fit(x_train, y_train)
pred_test = lr.predict(x_test)
time_cost = time.time() - start

print("Time Cost of Logistic Regression: ", time_cost)
print("Accuracy: ", np.mean(pred_test == y_test))
('Time Cost of Logistic Regression: ', 11.077100038528442)
('Accuracy: ', 0.9778)