Zero-shot semantic segmentation

• Traditional semantic segmentation: train the model on the seen classes and test the model on the seen classes

• Zero-shot semantic segmentation: train the model on the seen classes and test the model on the unseen classes

Attempt 1

Inserting Text Projection Layer to ZegFormer 

What is ZegFormer?

MaskFormer:

Generate class-agnostic segments

CLIP:

zero-shot classification on the segments

• Dataset: COCO-Stuff

  • 156 seen classes + 15 unseen classes

• Metric

  • Mean IoU between the prediction and the ground truth

  • Seen, Unseen, Harmonic

Experiment

• Add a text projection layer to further transform the text feature given by the CLIP text encoder

• Implemented as a MLP (512 → 384 → 512)

• Alleviates the overfitting to the seen classes

Modification

Result

• mIoU:
  • Decreases for seen classses
  • Increases for unseen clasess
• Overfitting for seen classes is slightly improved

Attempt 2

What is CRIS?

What is CRIS?

• Text-to-pixel matching
• The loss function helps to pull semantically similar pixels together and push dissimilar pixels away

Modification 1

Modification 2

Modifications

• Improve the segmentation performance on small objects
• Modification 1: image level
• Modification 2: pixel level

Modification 1: Image-level 

• In the vanilla setting, a large object segment has the same weight as a small object segment.

• The total loss is less sensitive to the mismatching of smaller objects.

• Therefore, we multiply the loss with a coefficient beta that is inversely proportional to the segment size.

Modification 2: Pixel-level 

• A smaller object occupies a smaller area.
• The pixels in the positive set P are much less than in the negative set N.
• Missing a                   matching in P is not that significant.
• Therefore, we multiply the loss on the positive side with an alpha > 1, so as to make the positive matching to be more oustanding.

• Dataset: COCO-Stuff

  • 156 seen classes + 15 unseen classes

• Metric

  • Mean IoU between the prediction and the ground truth

  • Seen, Unseen, Harmonic

• Prompt Engineering

  • e,g, "This is a photo of <class name>"

  • Performing zero-shot semantic segmentation guided by the class name

Experiment

Result

Result Analysis

Examples

Reflection

• Construct Consistent Evaluation Metrics.
• Modify the MaskFormer part in ZegFormer to improve the class-agnostic segmentation performance.
  
• Polish the CRIS model to segment all objects in an image simultaneously.
  
• Produce a comprehensive survey among ZegFormer, CRIS-like and traditional models.
  

Milestones

ZS3Net

SPNet