Panoptic segmentation

Deep Learning for Images with PyTorch

Michal Oleszak

Machine Learning Engineer

Panoptic segmentation challenge

Original image

Semantic segmentation

Instance segmentation

Panoptic segmentation

Panoptic segmentation workflow

Combining semantic and instance segmentation can be complex:
- Overlaps
- Ensuring unique instance IDs
Our workflow:
1. Generate semantic masks
2. Combine them into a single mask
3. Initialize the panoptic mask as the semantic mask
4. Generate instance masks
5. Iterate over instance masks and overlay detected objects onto the semantic mask

Semantic masks

model = UNet()

with torch.no_grad():
    semantic_masks = model(image_tensor) 
    print(semantic_masks.shape)

torch.Size([1, 3, 427, 640])

semantic_mask = torch.argmax(
  semantic_masks, dim=1
)

Instantiate the model
Produce semantic masks for the input image
Choose highest-probability class for each pixel

Semantic mask

Instance masks

model = MaskRCNN()


with torch.no_grad():
    instance_masks = model(image_tensor)[0]["masks"]
    print(instance_masks.shape)

torch.Size([80, 1, 427, 640])

Load instance segmentation model
Produce instance masks

Instance mask

Panoptic masks

panoptic_mask = torch.clone(semantic_mask)


instance_id = 3
for mask in instance_masks:

    panoptic_mask[mask > 0.5] = instance_id

    instance_id += 1

Initialize panoptic mask as semantic_mask
Iterate over instance masks
Set panoptic mask to instance ID where mask > 0.5
Increase instance ID counter

Panoptic mask

Let's practice!

Deep Learning for Images with PyTorch