Google Veo 3

Code used to create above plots

Startups working on Diffusion Models

AI Provenance

NSF and DARPA

mention AI control as core GenAI challenge

Autoregressive Formulation in Traditional LLMs

Each token prediction depends on previous tokens.

Sequential sampling

No parallelism

Error compounding

Sequential sampling

Large Language Diffusion Models - Loss

LLaDA’s core is a mask predictor: a model \( p_\theta(\cdot \mid x_t) \) that takes a masked sequence \( x_t \) and predicts all masked tokens (set \( M \)) simultaneously.

Pre-training

Supervised Finetuning

Sampling

We want to match

these by training \( E_\theta \)

Blind Watermarking: No additional metadata other than a watermarked image is required for detection.

Input Image \( I_0 \)

Watermarked Image \( I_w \)

Payload

Robustness

Invisibility

Payload

Robustness

Invisibility

• Watermarking within T2I generation.
• Object Level watermark localization.
• Towards training-free methods.
• Natural control for watermarking.

Additional constraints while using

Generative models for watermarking.

• Watermarking within T2I generation.
• Object Level watermark localization.
• Towards training-free methods.
• Natural control for watermarking.

Datasets:

MS COCO, WikiArt

Image similarity:

PSNR, SSIM, FID

Robustness Metrics:

Basic and Adversarial Attacks

 Multiple Object Watermarking

• Diffusion models (e.g., Stable Diffusion) can adapt to new subjects or styles with just a few images. Textual Inversion and DreamBooth achieve high-fidelity personalization rapidly.

Power of Personalization

Risks and Concerns

• Privacy: Risk of misuse of private or identity-sensitive data.

• Intellectual Property: Unauthorized style or content cloning.

• Security: Uncontrolled replication of copyrighted material

• Pixel-space perturbations make samples unlearnable. However...
  • Visibly degrade quality (noise/artifacts).
  • Vulnerable to purification defenses (e.g., DiffPure).

Limitations of Existing Defenses

Proposed Method

• Shift perturbations into latent space, altering the diffusion trajectory.
• Ensure perturbations are imperceptible yet robust to personalization and purification.
• Practical and scalable defense for safeguarding sensitive data.

Approach: Introduce adversarial noise perturbations to ensure that certain images become unlearnable.

Goal: Prevent a diffusion model from learning and reproducing specific images while maintaining its generalization capability.

We consider Latent Diffusion Models for Image Generation

Textual Inversion and DreamBooth as Personalization models

Diffusion Noise-Matching Loss

We consider Latent Diffusion Models for Image Generation

Textual Inversion and DreamBooth as Personalization models

Textual Inversion Loss

DreamBooth Loss

We consider Latent Diffusion Models for Image Generation

Textual Inversion and DreamBooth as Personalization models

where the personalization loss at the latent level is defined as

We consider Latent Diffusion Models for Image Generation

Textual Inversion and DreamBooth as Personalization models

Overall Loss:

\( \lambda \) - hyper-parameter

Few-step Diffusion Models maintain data distribution integrity while allowing perturbations in reconstruction:

Quantitative results comparison to baseline methods: Metrics: (1) Imperceptibility (PSNR, SSIM, FID), (2) Face-specific metrics (ISM, BRIS, FDFR), (3) Non-Face metrics (SDS, ISM, BRIS, IQAC, LIQE). The metrics presented utilize  \( 10/255 \) as the maximum budget, with \( 15 \) steps of DiffPure purification after Unlearnable Sample Generation and TI training with the prompt ``A photo of an sks person/object'' for personalization. We generate \( 16 \) images per prompt post personalization to report personalization metrics. We notice from the above table that our method shows significant improvements in imperceptible perturbation at the image-level while maintaining enhanced counter personalization metrics.

• All Current unlearnable sample methods = static: perturbations crafted once after personalization model is fixed.

• But real-world personalization is continual → models adapt incrementally.

• Need defenses that also adapt continually to evolving personalization systems.

This min-max formulation can be written as:

This setup better reflects real-world usage, where users may upload new training data, and personalization systems are incrementally updated.

What is Prompt Inversion?

Existing hard prompt inversion methods produce prompts that are either incoherent or too brittle under downstream token edits, so we invert a reference image into a coherent, edit-robust prompt that reliably reconstructs it via a T2I model, sidestepping the slow trial-and-error of manual prompt engineering.

• Prompt engineering for text-to-image (T2I) models is a slow trial-and-error process.
• Hard prompt inversion aims to recover a discrete text prompt that reconstructs an input reference image. However, existing inversion techniques optimize only for one-shot reconstruction.
• Prompts by existing methods are incoherent (PEZ) or brittle; a small token edit can collapse the whole image.
• Goal: prompts that are interpretable, aligned, and stay stable under token swap & append edits.

Interpretable and Edit-Friendly Prompt Inversion

◆ Edit-friendly objective. Hard inversion that directly optimizes alignment under token swap/append edits.

◆ CLIP-guided discrete diffusion. A dLLM decoder steered by CLIP for faithful, fully-readable prompts.

◆ Fast & plug-and-play. ~10x faster than baselines; no fine-tuning of the dLLM or any T2I model.

Contributions

Pipeline

Method

Method

Results - Quantitative

Results - Prompt Inversion comparison

Results - Concept Swap

Results - Concept Append

Results - TIFA and LLM as a Judge

Incremental concept addition without forgetting

Arjun Ramesh Koushik et al. 2026

Arjun Ramesh Koushik et al. 2026

Internal-signal guided Hallucination Detection and Mitigation

Internal-signal guided Hallucination Detection and Mitigation

Internal-signal guided Hallucination Detection and Mitigation

Validator-guided Hallucination Mitigation in Diffusion LLMs and VLMs

ongoing work

ongoing work

ongoing work

The floor is open to questions :)