Training a network in TensorFlow

Introduction to TensorFlow in Python

Isaiah Hull

Visiting Associate Professor of Finance, BI Norwegian Business School

The image shows a 3D-plot of a complicated objective function called the eggholder function.

Random initializers

Often need to initialize thousands of variables
- tf.ones() may perform poorly
- Tedious and difficult to initialize variables individually
Alternatively, draw initial values from distribution
- Normal
- Uniform
- Glorot initializer

Initializing variables in TensorFlow

import tensorflow as tf

# Define 500x500 random normal variable
weights = tf.Variable(tf.random.normal([500, 500]))

# Define 500x500 truncated random normal variable
weights = tf.Variable(tf.random.truncated_normal([500, 500]))

Initializing variables in TensorFlow

# Define a dense layer with the default initializer
dense = tf.keras.layers.Dense(32, activation='relu')

# Define a dense layer with the zeros initializer
dense = tf.keras.layers.Dense(32, activation='relu',\
    kernel_initializer='zeros')

Neural networks and overfitting

The image shows the predictions from simple and complex models in the training set.

The image shows the predictions from simple and complex models in the validation set.

Applying dropout

This image shows a neural network without dropout.

This image shows a neural network with dropout applied to random nodes.

Implementing dropout in a network

import numpy as np
import tensorflow as tf

# Define input data
inputs = np.array(borrower_features, np.float32)

# Define dense layer 1
dense1 = tf.keras.layers.Dense(32, activation='relu')(inputs)

Implementing dropout in a network

# Define dense layer 2
dense2 = tf.keras.layers.Dense(16, activation='relu')(dense1)

# Apply dropout operation
dropout1 = tf.keras.layers.Dropout(0.25)(dense2)

# Define output layer
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(dropout1)

Let's practice!

Introduction to TensorFlow in Python