GitHub Actions workflow for Hyperparameter Tuning

CI/CD for Machine Learning

Ravi Bhadauria

Machine Learning Instructor

Branching workflow

Separate feature branches for training and hyperparameter tuning
- Intended job should trigger
- Other job should not trigger
- Implemented using if condition
Hyperparameter tuning
- Print statistics table for analysis
- Automatically open a new PR with parameter changes
Training
- Read new parameter file in training PR

Setting conditionals

Hyperparameter Tuning

jobs:
  hp_tune_and_publish_report:
    # Run when branch name starts with hp_tune/
    if: startsWith(github.head_ref, 'hp_tune/')

    steps:
      ...
      - name: |
          DVC pipeline for hyperparameter tuning
        run: dvc repro -f hp_tune

Training

jobs:
  train_and_publish_report:
    # Run when branch name starts with train/
    if: startsWith(github.head_ref, 'train/')

    steps:
      ...
      - name: Run DVC pipeline for training
        run: dvc repro train

Setup workflow permissions

Repository Settings > Actions > General

Screenshot of settings page to configure workflow permissions

Hyperparameter tuning job kickoff

Make sure to prefix branch name with hp_tune/

Screenshot of GHA workflow showing hyperparameter tuning job running and training job skipped

Hyperparameter tuning job metrics

Screenshot of PR showing GridSearch results as a comment

Creating a training PR from hyperparameter run

steps:
  - name: Create training branch
    env:
      REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |
      # Branch name begins with train/
      export BRANCH_NAME=train/$(git rev-parse --short "${{ github.sha }}")

      # Create PR for training
      cml pr create \
        --user-email [email protected] \
        --user-name HPBot \
        --message "Hyperparameter tuning" \
        --branch $BRANCH_NAME \
        --target-branch main \
          rfc_best_params.json

New training branch PR

Top screenshot of new model training PR created as an outcome of GHA workflow

Bottom screenshot of new model training PR created as an outcome of GHA workflow

New training branch PR

Screenshot of best parameter file diff in the new model training PR

Starting training run manually

GITHUB_TOKEN cannot trigger workflows on self created PRs
- Prevention from recursive runs
Workarounds
- Use a Personal access token with proper permissions
```
steps:
  - env:
      GITHUB_TOKEN: ${{ secrets.MY_TOKEN }}
```
- Run training job right after hyperparameter tuning in GHA pipeline
- Force push the code to trigger a run (forces inspection)
```
-> git checkout train/1f34fs
-> git commit --amend --no-edit && git push -f
```

¹ https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow

Training job kickoff

Screenshot of GHA workflow showing training job running and hyperparameter tuning job skipped

Let's practice!

CI/CD for Machine Learning