Code profiling for memory usage

Writing Efficient Python Code

Logan Thomas

Scientific Software Technical Trainer, Enthought

Quick and dirty approach

import sys

nums_list = [*range(1000)]
sys.getsizeof(nums_list)

9112

import numpy as np

nums_np = np.array(range(1000))
sys.getsizeof(nums_np)

8096
Writing Efficient Python Code

Code profiling: memory

  • Detailed stats on memory consumption
  • Line-by-line analyses
  • Package used: memory_profiler
pip install memory_profiler
  • Using memory_profiler package
%load_ext memory_profiler

%mprun -f convert_units convert_units(heroes, hts, wts)
Writing Efficient Python Code

Code profiling: memory

  • Functions must be imported when using memory_profiler
    • hero_funcs.py

 

from hero_funcs import convert_units

%load_ext memory_profiler

%mprun -f convert_units convert_units(heroes, hts, wts)
Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics”

Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with Line # column highlighted”

Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with Mem usage column highlighted”

Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with Increment column highlighted”

Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with Line contents column highlighted”

Writing Efficient Python Code

%mprun output

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics”

Writing Efficient Python Code

%mprun output caveats

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with mebibytes (MiB) in output highlighted”

Writing Efficient Python Code

%mprun output caveats

Data used in this example is a random sample of 35,000 heroes.

(not original 480 superheroes dataset)

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics”

Writing Efficient Python Code

%mprun output caveats

Small memory allocations could result in 0.0 MiB output.

(using original 480 superheroes dataset)

%mprun -f convert_units convert_units(heroes, hts, wts)

alt=”Magic command mprun tabular output summarizing memory usage profiling statistics with all rows in Increment column showing 0.0 MiB”

Writing Efficient Python Code

%mprun output caveats

  • Inspects memory by querying the operating system
  • Results may differ between platforms and runs
    • Can still observe how each line of code compares to others based on memory consumption
Writing Efficient Python Code

Let's practice!

Writing Efficient Python Code

Preparing Video For Download...