Text Alignment in RETFound
Why try text at all?
- RETFound only image information (structure)
- EyeCLIP image + text (semantics)
- semantic anchors for glaucoma, DR, and AMD.
Q: “does clinical meaning make them more useful for RETFound?”
Architecture
Reference baselines : (a) CLIP (b) EyeCLIP (c) RETFound
| No text | None | Baseline |
| Clinical | Disease vectors | Clinical meaning |
| Orthogonal | Three artificial separated vectors | Separation alone |
| Collapsed | The same information | Generic regularization |
| Condition | Prototype target | What it tests |
|---|---|---|
| No text | None | Baseline without TextAlign |
| Clinical | EyeCLIP disease vectors | Does clinical meaning help? |
| Orthogonal | Three artificial separated vectors | Is separation alone enough? |
| Collapsed | The same vector for every disease | Does generic regularization explain the effect? |
Reference baselines : (a) CLIP (b) EyeCLIP (c) RETFound
Clinical text should perform better compared to all three controls.
Reference baselines : (a) CLIP (b) EyeCLIP (c) RETFound
| No text | None | Baseline |
| Clinical | Disease vectors | Clinical meaning |
| Orthogonal | Three artificial separated vectors | Separation alone |
| Collapsed | The same information | Generic regularization |
| Condition | Prototype target | What it tests |
|---|---|---|
| No text | None | Baseline without TextAlign |
| Clinical | EyeCLIP disease vectors | Does clinical meaning help? |
| Orthogonal | Three artificial separated vectors | Is separation alone enough? |
| Collapsed | The same vector for every disease | Does generic regularization explain the effect? |
*similar performance.
*similar performance.
why the increase?
Architecture
Q: What's the contribution of Text and BCE?
Architecture
Direction Similarity
PILOT!!!
Direction Similarity
PILOT!!!
Direction Similarity
BCE and text share similar information.
PILOT!!!
Magnitude
BCE gives more valuable information than text.
PILOT!!!
Take-home message
- text alignment contribution at the beginning.
- disease-language semantics not helping*
- similar results between disease-language and artificial
- geometry/regularization effect, not a language-specific benefit.
BUT
- Binary Class Entropy takes over
“The text would help, but the task to fine-tune is the main contributor. Hence, the text effect might not be significant.
Take-home message
- text alignment contribution at the beginning.
- disease-language semantics not helping*
- similar results between disease-language and artificial
- geometry/regularization effect, not a language-specific benefit.
BUT
- Binary Class Entropy takes over
“The text would help, but the task (BCE) to fine-tune is the main contributor. Hence, the text effect might not be significant. PILOT would help.