Number of Parameters in Dense and Convolutional Layers in Neural Networks

I find it hard to picture the structures of dense and convolutional layers in neural networks. It helps to use some examples with actual numbers of their layers. Here are some examples to demonstrate and compare the number of parameters in dense and convolutional neural networks using Keras.

Code is here: https://github.com/yang-zhang/deep-learning/blob/master/number_parameters_dense_conv_nn.ipynb

img_rows, img_cols = 224, 224
colors = 3
input_size = img_rows * img_cols * colors
input_shape = (img_rows, img_cols, colors)
num_classes = 10

Dense layers

from keras.models import Sequential
from keras.layers import Dense
model = Sequential([
    Dense(32, activation='relu', input_shape=(input_size,)),
    Dense(64, activation='relu'),
    Dense(128, activation='relu'),
    Dense(num_classes, activation='softmax')
])
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_5 (Dense)              (None, 32)                4816928   
_________________________________________________________________
dense_6 (Dense)              (None, 64)                2112      
_________________________________________________________________
dense_7 (Dense)              (None, 128)               8320      
_________________________________________________________________
dense_8 (Dense)              (None, 10)                1290      
=================================================================
Total params: 4,828,650
Trainable params: 4,828,650
Non-trainable params: 0
_________________________________________________________________

These assertions show how the numbers of parameters of the layers depend on input, output, and each other: output_size * (input_size + 1) == number_parameters

assert 32 * (input_size + 1) == 4816928
assert 64 * (32 + 1) == 2112
assert 128 * (64 + 1) == 8320
assert num_classes * (128 + 1) == 1290

Conv layers

from keras.models import Sequential
from keras.layers import Conv2D

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=input_shape),
    Conv2D(64, (3, 3), activation='relu'),
    Conv2D(128, (3, 3), activation='relu'),
    Dense(num_classes, activation='softmax')
])
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 222, 222, 32)      896       
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 220, 220, 64)      18496     
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 218, 218, 128)     73856     
_________________________________________________________________
dense_9 (Dense)              (None, 218, 218, 10)      1290      
=================================================================
Total params: 94,538
Trainable params: 94,538
Non-trainable params: 0
_________________________________________________________________

These assertions show how the numbers of parameters of the layers depend on input, output, and each other: again, output_size * (input_size + 1) == number_parameters. For convnets, output_channels * (input_channels * window_size + 1) == number_parameters. For the this particular example, window_size=3*3

assert 32 * (3 * (3*3) + 1) == 896
assert 64 * (32 * (3*3) + 1) == 18496
assert 128 * (64 * (3*3) + 1) == 73856
assert num_classes * (128 + 1) == 1290

Conv and dense layers

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_4 (Conv2D)            (None, 222, 222, 32)      896       
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 220, 220, 64)      18496     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 110, 110, 64)      0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 110, 110, 64)      0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 774400)            0         
_________________________________________________________________
dense_10 (Dense)             (None, 128)               99123328  
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_11 (Dense)             (None, 10)                1290      
=================================================================
Total params: 99,144,010
Trainable params: 99,144,010
Non-trainable params: 0
_________________________________________________________________

These assertions show how the numbers of parameters of the layers depend on input, output, and each other.

assert 32 * (3 * (3*3) + 1) == 896
assert 64 * (32 * (3*3) + 1) == 18496
assert 110 * 110 * 64 == 774400
assert 128 * (774400 + 1) == 99123328
assert num_classes * (128 + 1) == 1290