套用動態量化

Scalable AI Models with PyTorch Lightning

Sergiy Tkachuk

Director, GenAI Productivity

為何使用量化？

$$

  • 記憶體縮減

代表計算記憶體縮減的圖示

Scalable AI Models with PyTorch Lightning

為何使用量化？

$$

  • 記憶體縮減
  • CPU 加速

代表計算記憶體縮減與閃電的圖示

Scalable AI Models with PyTorch Lightning

為何使用量化？

$$

  • 記憶體縮減
  • CPU 加速
  • 行動端推論

代表計算記憶體縮減、閃電與行動裝置的圖示

Scalable AI Models with PyTorch Lightning

什麼是動態量化？

$$

  • 降低權重與運算精度
  • 提升推論速度
  • 適合資源受限裝置部署

$$

import torch
from torch.quantization 
import quantize_dynamic


model_quantized = quantize_dynamic(
    model, 
    {torch.nn.Linear}, 
    dtype=torch.qint8
)
Scalable AI Models with PyTorch Lightning

評估量化影響

天平上原始模型與量化模型的示意圖，呈現準確度與效率之間的權衡

Scalable AI Models with PyTorch Lightning

效能比較

$$

  • ⚡ 比較推論速度與記憶體占用

$$

  • 📊 判斷可接受的準確度取捨

$$

  • ⛗ 依部署需求決定量化是否合適
Scalable AI Models with PyTorch Lightning

比較效能

import time

def measure_time(model, data_loader):
      model.eval()  # Set model to evaluation mode
    start_time = time.time()
    for inputs in data_loader:

        _ = model(inputs)
    end_time = time.time()
    return end_time - start_time


original_time = measure_time(model, test_loader)
quant_time = measure_time(model_quant, test_loader)

print(f"Original Model Time: {original_time:.2f}s")
print(f"Quantized Model Time: {quant_time:.2f}s")
Scalable AI Models with PyTorch Lightning

一起來練習吧！

Scalable AI Models with PyTorch Lightning

Preparing Video For Download...