A deeper dive into neural network architecture

Introduction to Deep Learning with PyTorch

Maham Faisal Khan

Senior Data Scientist

Layers are made of neurons

  • Linear layers are fully connected
  • Each neuron of a layer connected to each neuron of previous layer

 

  • A neuron of a linear layer:
    • computes a linear operation using all neurons of previous layer
    • contains N+1 learnable parameters
    • where N = dimension of previous layer's outputs

a linear layer

Introduction to Deep Learning with PyTorch

Layer naming convention

hidden layers

Introduction to Deep Learning with PyTorch

Tweaking the number of hidden layers

  • Input and output layers dimensions are fixed.
    • input layer depends on the number of features n_features
    • output layer depends on the number of categories n_classes
model = nn.Sequential(nn.Linear(n_features, 8),
                      nn.Linear(8, 4),
                      nn.Linear(4, n_classes))

  • We can use as many hidden layers as we want

  • Increasing the number of hidden layers = increasing the number of parameters = increasing the model capacity

Introduction to Deep Learning with PyTorch

Counting the number of parameters

Given the following model:

model = nn.Sequential(nn.Linear(8, 4), 
                      nn.Linear(4, 2))

Manually calculating the number of parameters:

  • first layer has 4 neurons, each neuron has 8+1 parameters = 36 parameters
  • second layer has 2 neurons, each neuron has 4+1 parameters = 10 parameters
  • total = 46 learnable parameters

Using PyTorch:

  • .numel(): returns the number of elements in the tensor
total = 0
for parameter in model.parameters():
    total += parameter.numel()
print(total)

46
Introduction to Deep Learning with PyTorch

Let's practice!

Introduction to Deep Learning with PyTorch

Preparing Video For Download...