Hidden layers and parameters

Introduction to Deep Learning with PyTorch

Jasmin Ludolf

Senior Data Science Content Developer, DataCamp

Stacking layers with nn.Sequential()

# Create network with three linear layers
model = nn.Sequential(

    nn.Linear(n_features, 8),

    nn.Linear(8, 4),
    nn.Linear(4, n_classes)

)
  • Input is passed through the linear layers
  • Layers within nn.Sequential() are hidden layers
Introduction to Deep Learning with PyTorch

Stacking layers with nn.Sequential()

# Create network with three linear layers
model = nn.Sequential(
    nn.Linear(n_features, 8), # n_features represents number of input features
    nn.Linear(8, 4),
    nn.Linear(4, n_classes) # n_classes represents the number of output classes  
)
  • Input is passed through the linear layers
  • Layers within nn.Sequential() are hidden layers
  • n_features and n_classes are defined by the dataset
Introduction to Deep Learning with PyTorch

Adding layers

Illustration of a neural network with two hidden layers

Introduction to Deep Learning with PyTorch

Adding layers

Illustration of a neural network with four hidden layers

Introduction to Deep Learning with PyTorch

Adding layers

Illustration of a neural network with four hidden layers

Introduction to Deep Learning with PyTorch

Adding layers

Illustration of a neural network with four hidden layers

Introduction to Deep Learning with PyTorch

Adding layers

# Create network with three linear layers
model = nn.Sequential(
    nn.Linear(10, 18),
    nn.Linear(18, 20),
    nn.Linear(20, 5)
)
Introduction to Deep Learning with PyTorch

Adding layers

# Create network with three linear layers
model = nn.Sequential(
    nn.Linear(10, 18), # Takes 10 features and outputs 18
    nn.Linear(18, 20),
    nn.Linear(20, 5)
)
  • Input 10 ➡ output 18 ➡ output 20 ➡ Output 5
Introduction to Deep Learning with PyTorch

Adding layers

# Create network with three linear layers
model = nn.Sequential(
    nn.Linear(10, 18),
    nn.Linear(18, 20), # Takes 18 and outputs 20
    nn.Linear(20, 5)
)
  • Input 10 ➡ output 18 ➡ output 20 ➡ Output 5
Introduction to Deep Learning with PyTorch

Adding layers

# Create network with three linear layers
model = nn.Sequential(
    nn.Linear(10, 18),
    nn.Linear(18, 20),
    nn.Linear(20, 5) # Takes 20 and outputs 5
)
  • Input 10 ➡ output 18 ➡ output 20 ➡ Output 5
Introduction to Deep Learning with PyTorch

Layers are made of neurons

$$

  • Fully connected when each neuron links to all neurons in the previous layer

An illustration of an input and output layer with arrows connecting the layers

Introduction to Deep Learning with PyTorch

Layers are made of neurons

$$

  • Fully connected when each neuron links to all neurons in the previous layer

$$

  • A neuron in a linear layer:

An illustration of an input and output layer with arrows connecting the layers and a circle around one output neuron

Introduction to Deep Learning with PyTorch

Layers are made of neurons

$$

  • Fully connected when each neuron links to all neurons in the previous layer

$$

  • A neuron in a linear layer:
    • Performs a linear operation using all neurons from the previous layer

An illustration of an input and output layer with arrows connecting the layers and a circle around one output neuron and highlighted arrows

Introduction to Deep Learning with PyTorch

Layers are made of neurons

$$

  • Fully connected when each neuron links to all neurons in the previous layer

$$

  • A neuron in a linear layer:
    • Performs a linear operation using all neurons from the previous layer
    • Has N+1 parameters: N from inputs and 1 for the bias

N+1

Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

$$

Manual parameter calculation:

  • First layer has 4 neurons, each neuron has 8+1 parameters. 9 times 4 = 36 parameters
Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

$$

Manual parameter calculation:

  • First layer has 4 neurons, each neuron has 8+1 parameters. 9 times 4 = 36 parameters
Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

$$

Manual parameter calculation:

  • First layer has 4 neurons, each neuron has 8+1 parameters. 9 times 4 = 36 parameters
  • Second layer has 2 neurons, each neuron has 4+1 parameters. 5 times 2 = 10 parameters
Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

$$

Manual parameter calculation:

  • First layer has 4 neurons, each neuron has 8+1 parameters. 9 times 4 = 36 parameters
  • Second layer has 2 neurons, each neuron has 4+1 parameters. 5 times 2 = 10 parameters
  • 36 + 10 = 46 learnable parameters
Introduction to Deep Learning with PyTorch

Parameters and model capacity

  • More hidden layers = more parameters = higher model capacity

$$

Code block for a two layer network

$$

Using PyTorch:

  • .numel(): returns the number of elements in the tensor
total = 0
for parameter in model.parameters():
    total += parameter.numel()
print(total)

46
Introduction to Deep Learning with PyTorch

Balancing complexity and efficiency

Balance between complexity and efficiency

Introduction to Deep Learning with PyTorch

Let's practice!

Introduction to Deep Learning with PyTorch

Preparing Video For Download...