Wrap-up

Creating Robust Workflows in Python

Martin Skarzynski

Co-Chair, Foundation for Advanced Education in the Sciences (FAES)

Principles

  • DRY (Don't repeat yourself)
  • Modularity
  • Abstraction

cat owner versus veterinarian perspective Booch, G. et al. Object-Oriented Analysis and Design with Applications. Addison-Wesley, 2007, p. 45.

dripping faucet

stacking lego blocks

Creating Robust Workflows in Python

Documentation

sphinx logo

  • Includes:
    • Docstrings triple quote
    • Type hints x: int
Creating Robust Workflows in Python

Project templates

cookiecutter logo

pyscaffold logo

sphinx logo

  • Includes:
    • Docstrings triple quote
    • Type hints x: int
Creating Robust Workflows in Python

Tests

  • pytest testing framework
    • pytest.ini configuration file
      • doctest: run docstring examples
      • mypy: check types

pytest logo

mypy logo

Creating Robust Workflows in Python

Jupyter notebooks

  • Create and edit notebooks
    • nbformat
  • Convert notebooks to other formats
    • nbconvert
  • Execute notebooks with parameters
    • papermill
  • Access notebook data
    • scrapbook
  • Check out rmarkdown! rmarkdown r package logo

jupyter project logo

papermill logo

scrapbook logo

Creating Robust Workflows in Python

Pipelines

gnu make book cover

snakemake logo

airflow logo

dask logo

Creating Robust Workflows in Python

Virtual environments

pipenv logo

  • Create virtual Python environments
    • venv, virtualenv, or pipenv
  • Install Python packages
    • pip or pipenv
  • Not limited to Python conda logo
Creating Robust Workflows in Python

Packaging

python package index logo

  • Package Python code
    • setuptools
  • Deploy packages to PyPI
    • twine

flit logo

poetry logo

Creating Robust Workflows in Python

Keep learning!

Creating Robust Workflows in Python

Preparing Video For Download...