Batch training

Introduction to TensorFlow in Python

Isaiah Hull

Visiting Associate Professor of Finance, BI Norwegian Business School

What is batch training?

This image shows data for the price, size, and number of bedrooms for houses in King County.

This image shows data for the price, size, and number of bedrooms for houses in King County, divided into batches.

The chunksize parameter

pd.read_csv() allows us to load data in batches
- Avoid loading entire dataset
- chunksize parameter provides batch size

# Import pandas and numpy
import pandas as pd
import numpy as np

# Load data in batches
for batch in pd.read_csv('kc_housing.csv', chunksize=100):
    # Extract price column
    price = np.array(batch['price'], np.float32)

    # Extract size column
    size = np.array(batch['size'], np.float32)

Training a linear model in batches

# Import tensorflow, pandas, and numpy
import tensorflow as tf
import pandas as pd
import numpy as np

# Define trainable variables
intercept = tf.Variable(0.1, tf.float32)
slope = tf.Variable(0.1, tf.float32)

# Define the model
def linear_regression(intercept, slope, features):
    return intercept + features*slope

Training a linear model in batches

# Compute predicted values and return loss function
def loss_function(intercept, slope, targets, features):
    predictions = linear_regression(intercept, slope, features)
    return tf.keras.losses.mse(targets, predictions)

# Define optimization operation
opt = tf.keras.optimizers.Adam()

Training a linear model in batches

# Load the data in batches from pandas
for batch in pd.read_csv('kc_housing.csv', chunksize=100):
    # Extract the target and feature columns
    price_batch = np.array(batch['price'], np.float32)
    size_batch = np.array(batch['lot_size'], np.float32)

    # Minimize the loss function
    opt.minimize(lambda: loss_function(intercept, slope, price_batch, size_batch), 
                 var_list=[intercept, slope])

# Print parameter values
print(intercept.numpy(), slope.numpy())

Full sample versus batch training

Full Sample
1. One update per epoch
2. Accepts dataset without modification
3. Limited by memory

Batch Training
1. Multiple updates per epoch
2. Requires division of dataset
3. No limit on dataset size

Let's practice!

Introduction to TensorFlow in Python