• 1000 images each night, 15 TB/night for 10 years


• 18,000 square degrees, observed once every few days


• Tens of billions of objects, each one observed ∼1000 times

SDSS

Image credit: Peter Melchior

DES

Image credit: Peter Melchior

Image credit: Peter Melchior

HSC (proxy for LSST)

The Deep Learning Boom in Astrophysics

astro-ph abstracts mentioning Deep Learning, CNN, or Neural Networks

Huertas-Company & Lanusse (2023)

The Limits of Traditional Deep Learning

• Limited Supervised Training Data
  • ​Rare or novel objects have by definition few labeled examples
     
  • In Simulation Based Inference (SBI), training a neural compression model requires many simulations
     
• Limited Reusability
  • Existing models are trained supervised on a specific task, and specific data.

Huang et al. (2019)

Zhang, Bloom, Gaudi, Lanusse, Lam, Lu (2021)

The Rise of The Foundation Model Paradigm

• Foundation Model approach
  • Pretrain models on pretext tasks, without supervision, on very large scale datasets.
  • Adapt pretrained models to downstream tasks. 
  • Combine pretrained modules in more complex systems.

Bommasani et al. 2021

The Advantage of Scale of Data and Compute

Liu et al. 2022

Linearly Accessible Information

• Backbone of modern architectures embed input images  as vectors in         where d can typically be between 512 to 2048.
   
• Linear probing refers to training a single matrix to adapt this vector representation to the desired downstream task.

Can we translate these innovations into a similar paradigm shift in deep learning for scientific applications?

Polymathic

What is CLIP?

Contrastive Language Image Pretraining (CLIP)
(Radford et al. 2021)

One model, many downstream applications!

The AstroCLIP approach

• We use spectra and multi-band images as our two different views for the same underlying object.
   
• DESI Legacy Surveys (g,r,z) images, and DESI EDR galaxy spectra.

The Information Point of View

• The InfoNCE loss is a lower bound on the Mutual Information between modalities

Shared physical information about galaxies between images and spectra

van den Oord et al. (2018)

• Redshift Estimation From Images

Supervised baseline

• Zero-shot prediction                    
  • k-NN regression

• Few-shot prediction
  • MLP head trained on top of frozen backbone

Evaluation of the model: Parameter Inference

• Galaxy Physical Property Estimation from Images and Spectra

We use estimates of galaxy properties from the PROVABGS catalog (Hahn et al. 2023) (Bayesian spectral energy distribution (SED) modeling of DESI spectroscopy and photometry method)

of regression

Negative Log Likelihood of Neural Posterior Inference

• Galaxy Morphology Classification

Classification Accuracy

We test a galaxy morphology classification task using as labels the GZ-5 dataset (Walmsley et al. 2021)

The AstroCLIP Model (v2, Parker et al. in prep.)

• For images, we use a ViT-L Transformer (300M).
   
• For spectra, we use a decoder only Transformer working at the level of spectral patches.

(Dosovitskiy et al 2021)

DiNOv2 (Oquab et al. 2023) Image Pretraining

• Common practice for SOTA CLIP models is to initially pretrain the image encoder before CLIP alignment
• We adopt the DiNOv2 state of the art Self-Supervised Learning model for the initial large scale training of the model.

• We pretrain the DiNOv2 model on ~70 million postage stamps from DECaLS   

PCA of patch features

Dense Semantic Segmentation

Dense Depth Estimation

Spectrum Transformer Pretraining by Masked Modeling

• To pretrain the spectrum embedder, we use a simple Masked Image Modeling strategy

Detecting Galaxy Tidal Features Using Self-Supervised Representation Learning

Project led by Alice Desmons, Francois Lanusse, Sarah Brough

Evaluation of the model: Similarity Search

• Cross-Modal similarity search

Image Similarity

Spectral Similarity

New Generation of Token-Based Multimodal Models

Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al. 2022)

All-to-All Foundation Models

Yu et al. (2023)

Why Is It Interesting to Us?

Going Further: Data Collection and Curation

• Development of large models requires access to "web scale" datasets
   
• Astrophysics generates large amounts of publicly available data,
  • BUT, data is usually not stored or structured in an ML friendly way.
     
• Accessing and using scientific data requires significant expertise, for each dataset.

The MultiModal Universe Project

• Goal: Assemble the first large-scale multi-modal dataset for machine learning in astrophysics.
• Main pillars:
  • Engage with a broad community of AI+Astro experts.
  • Adopt standardized conventions for storing and accessing data and metadata through mainstream tools (e.g. Hugging Face Datasets).
  • Target large astronomical surveys, varied types of instruments, many different astrophysics sub-fields.

=> Official release October 2024

https://github.com/MultimodalUniverse/MultimodalUniverse

Accepted at NeurIPS 2024 🎉

Scientific Data Tokenization

Mizrahi et al. (2023)

Our strategy: 

• Develop modality specific but universal tokenizers, i.e. a single model to embed all type of astronomical images
   
• This requires specific innovations to take into account the metadata of observations.

Example of strategy to embed different bands

The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning

• 55B tokens from 3M frames
  => First ImageNet scale dataset for fluids
   
• 18 subsets spanning problems in astro, bio, aerospace, chemistry, atmospheric science, and more.
   
• Simple self-documented HDF5 files, with pytorch readers provided.

Accepted at NeurIPS 2024 🎉