Traditional Drug Discovery Process

• ​Takes 10+ years for the development of a single drug!

• Lower estimates for the development of a drug is roughly ~$2B!

Drug Discovery 101

​

• Exploration in the compound universe leads to a combinatorial explosion! (1e20 -1e60 molecules)

• What we want is targeted and directed exploration.

• What if we had a magic box that could generate novel molecules with the desired properties?

Cue Deep Learning!​

Different representations

• But how do you represent a molecule to train these models?

• Molecules are usually represented in two primary formats:

  • Graphs

  • SMILES 

Different representations

• SMILES stands for Simplified Molecular Input Line-Entry System.

• Textual representation obtain from a Depth-First Traversal of a molecular compound.

• Chemical rules are established for validity.

Transformers to the Rescue

• The famous Transformer architecture!

• The daunting model has two broad parts:

  • Encoder : Learn the latent embeddings.

  • Decoder : Generate output auto-regressively.

Transformers to the Rescue

Encoder Only

• BERT based models.

• Uses contexts bidirectionally for effective learning.

• Proposed Masked Language Modeling (MLM).

Transformers to the Rescue

Decoder Only

• GPT based models.

• Unidirectional Autoregressive model.

Transformers to the Rescue

Encoder-Decoder

Best of both worlds!

Mega-Mol-BART

• 500M molecules sampled randomly from ZINC15

• BART

  • ​Developed by FAIR.

  • Span Masking.

Results

• Usage:

  • SMILES strings sampled around the region of a single input SMILES molecule.

  • SMILES strings sampled at regularly spaced intervals between two input molecules.​

• Results:

  • ​Validity: 0.98

  • Uniqueness: 0.32

  • Novelty: 0.36

Results