Incorporating validation and testing

Scalable AI Models with PyTorch Lightning

Sergiy Tkachuk

Director, GenAI Productivity

Why incorporate validation and testing?

Validation
- Identify model performance issues early
- Prevent overfitting and underfitting
Testing
- Performance on unseen data

Validation and test diagram

Implementing validation

Evaluate model performance at each epoch
Aggregate metrics for a more stable view

def validation_step(self, batch, batch_idx):
  x, y = batch
  preds = self(x)
  loss = F.cross_entropy(preds, y)
  self.log('val_loss', loss)


def validation_epoch_end(self, outputs):
  avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
  self.log('avg_val_loss', avg_loss)

Implementing testing

Assess final model performance on unseen data
Benchmark real-world effectiveness
Provide metrics for model deployment

def test_step(self, batch, batch_idx):
  x, y = batch
  y_hat = self(x)
  loss = F.cross_entropy(y_hat, y)
  self.log('test_loss', loss)


def test_epoch_end(self, outputs):
  avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
  self.log('avg_test_loss', avg_loss)

Evaluation with Torchmetrics

Monitor metrics such as accuracy
Easily integrate into Lightning workflow
Initialize accuracy
Calculate accuracy at each validation step

from torchmetrics import Accuracy

class BaseModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.accuracy = Accuracy()


    def validation_step(self, batch, batch_idx):
        x, y = batch
        preds = self(x)
        acc = self.accuracy(preds, y)
        self.log('val_acc', acc)

Connecting DataModule, validation, and testing

Data logic centralized in DataModule
Consistent train/val/test data splits
Automatic validation metric logging
Reproducible pipeline from prep to reporting

DataModule, validation and testing diagram

Let's practice!

Scalable AI Models with PyTorch Lightning