LDA

1

2

two simple assumptions:

slide 1

LDA

1

2

slide 1

Each document is a mixture of topics.

Each topic is a probability distribution over words.

Latent Dirichlet Allocation

LDA

1

2

slide 1

Each document is a mixture of topics.

Each topic is a probability distribution over words.

Latent Dirichlet Allocation

LDA

1

2

slide 1

Each document is a mixture of topics.

Each topic is a probability distribution over words.

Latent Dirichlet Allocation

Example

slide 3

Each topic is a probability distribution over words.

slide 3

quantum

basketball

slide 3

quantum

basketball

slide 3

slide 3

slide 3

topic 1

topic 2

topic 3

slide 3

topic 1

topic 2

topic 3

slide 3

slide 4

\( \alpha \)

\( \theta \)

\( z \)

\( w\)

\( \beta \)

\( M \)

\( N \)

\( \phi \)

slide 4

\( \alpha \)

\( \theta \)

\( z \)

\( w\)

\( \beta \)

\( M \)

\( N \)

\( \phi \)

election

word-topic distribution

slide 5

Initial Random Allocation

topic 2

slide 5

topic 2

quantum

basketball

election

ballot

Initial Random Allocation

slide 5

topic 2

quantum

basketball

election

ballot

• We cannot compute these positions analytically. 
• LDA uses approximate inference,
  typically Gibbs sampling.

slide 5

Outlook

In 2003, LDA was groundbreaking

• It was unsupervised (no labeled data was needed).

• It provided interpretable topics.

• It was efficient for small to medium-sized datasets.

Choosing a Topic Modeling Approach

• Today, transformer-based models offer more advanced topic modeling.

• BERTopic clusters transformer-generated sentence embeddings.

• These capture semantics better than LDA, providing more meaningful topics.

A Challenge For You

github.com/carinahausladen/deer-in-a-chalet

Our Gibbs Sampler does not converge.

Work in pairs to achieve convergence!

Happy Modeling!

github.com/carinahausladen/deer-in-a-chalet

A common goal is to summarize or classify documents based on their content.

Why start with LDA in 2026?

LDA remains foundational because it introduced the core grammar of modern topic modeling:

1. Latent variables: topics as unobserved structure
2. Mixed membership: documents as topic distributions
3. Approximate inference: Gibbs / variational methods
4. Generative modeling: explicit assumptions about text

Modern approaches (neural topic models, BERTopic, embeddings) change the machinery — not the underlying problem:

• How do we infer latent thematic structure from text under uncertainty?

How does LDA uncover meaningful topics without knowing the meaning of words?

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

Topic Assignment Process

slide 3

count

election election ballot 
ballot
quantum

election election ballot 
ballot
quantum

– 1

– 1

\( P(\text{topic 3}|d) = \frac{(0 + 0.1)}{(5 + 3 \times 0.1)} \approx 0.23 \)

election election ballot 
ballot
quantum

\( P(\text{election}|\text{topic 3}) = \frac{(0+0.01)}{(1 + 4 \times 0.01)} \approx 0.4 \)

\( \times \)

\( P(\text{topic 3}|d) = \frac{(0 + 0.1)}{(5 + 3 \times 0.1)} \approx 0.23 \)

election election ballot 
ballot
quantum

election election ballot 
ballot
quantum

0.064

0.24

election election ballot 
ballot
quantum

0.064

.

0.058

election election ballot 
ballot
quantum

0.064

.

0.058

election election ballot 
ballot
quantum

election election ballot 
ballot
quantum

election election ballot 
ballot
quantum

slide 4

Initial random allocation
of topics to words.

• The new document is more monochromatic.

• The new word-topic associations are more consistent.

• Train LDA for different k values 

• Compute both perplexity and coherence scores.

  • Choose the smallest k that minimizes perplexity and maximizes coherence.

• Verify topics manually (ensure they are interpretable).

• α → Document-Topic Distribution

  • High α → Documents cover many topics.

  • Low α → Documents focus on few topics.

  • Default: $50/K$ (K = topics).

• β → Topic-Word Distribution

  • High β → Topics use many words.

  • Low β → Topics use fewer words.

  • Default: 0.01 (sparse topics).
     

How to Choose α & β?
Grid Search → Test values, compare perplexity/coherence.

Gibbs Sampling: Convergence

• Convergence Criteria in Gibbs Sampling:

  • Topic distribution stability – Minimal change in topic-word & document-topic distributions.

• Iterations Required:

• • Typically 1000–2000 iterations.

  • First 50–200 iterations discarded as burn-in.

  • Convergence depends on dataset size, number of topics, and hyperparameters.

• Key Takeaway:

  • No strict stopping rule – check metrics stabilization!

Gibbs Sampling

• A Markov Chain Monte Carlo (MCMC) method that iterates over every word and samples topic assignments based on the current state of the model. Over many iterations, it converges to an approximation of the true posterior distribution.

• Pros:

  ✅ More accurate in the long run (closer to the true posterior).

  ✅ Works well for small to medium-sized datasets.

  ✅ No need for complicated derivations—just sample iteratively.

• Cons:

  ❌ Slow convergence – requires many iterations.

  ❌ Not scalable to large datasets (millions of documents).

  ❌ No clear stopping criterion – needs manual tuning of iterations.

• Best Use Cases:

  🔹 Small to medium corpora (thousands to hundreds of thousands of documents).

  🔹 When accuracy is more important than speed.

  🔹 When computational resources are not a major constraint.

Variational Inference

Converts Bayesian inference into an optimization problem. Uses an approximate but deterministic method to find a simpler distribution (e.g., mean-field assumption) that is close to the true posterior.

• Pros:

  ✅ Much faster – works well with large-scale datasets.
  ✅ Has a clear convergence criterion (minimizing KL divergence).
  ✅ Deterministic – always produces the same output for the same data.

• Cons:

  ❌ Less accurate than Gibbs Sampling (approximates the posterior but may not match it exactly).
  ❌ Requires more mathematical derivation (especially for complex models).
  ❌ Can get stuck in local optima instead of the true posterior.

• Best Use Cases:

  🔹 Large-scale document corpora (millions of documents).
  🔹 When speed and scalability are more important than perfect accuracy.
  🔹 Online learning settings where models need to be updated continuously.

• MCMC methods like Gibbs Sampling are used to sample from high-dimensional distributions when direct computation is infeasible.

• In LDA, we want to sample from the posterior \( p(\theta, z, \phi \mid w, \alpha, \beta) \) but it’s intractable. Instead, Gibbs sampling iteratively samples from conditional distributions, building a Markov Chain that eventually converges to the true posterior.
  
  Marcov Property

• In MCMC, each new sample depends only on the previous state.

• In LDA’s Gibbs sampling, we sample \( z_{dn} \) (the topic of word \( w_{dn} \)) based on the topic assignments of all other words \( Z_{-dn} \), making it a Markov Chain.

Why is Gibbs Sampling an MCMC Method?

Why can we approximate?

• Baye's Rule: \( Posterior =  \frac{Likelihood \times Prior}{Evidence (Marginal  Likelihood)} \)

• Exact Bayesian inference:
  \( p(topic 1|quantum) = \frac{p(quantum|topic 1)p(topic 1)}{ p(quantum)} \)

• In Gibbs sampling, we skip computing \( p(quantum) \) / \( p(w|\alpha, \beta) \) because:

  • It's impossible hard to compute

  • We only care about relative probabilities (so we do not need it)

  • So we use:
    p(topic1|quantum) \( \propto\) p(quantum|topic1)p(topic1)

Details on the Dirichlet Distribution

K-dimensional Dirichlet distribution the probability density function (PDF) is:

P(θ∣α)=1B(α)∏i=1Kθiαi−1

where:

θ=(θ1,θ2,...,θK

• \( \theta = (\theta_1, ..., \theta_K) \) represents a probability vector (e.g., a document's topic distribution).α=(α1,α2,...,αK

• \( \alpha = (\alpha_1, ..., \alpha_K) \) is the Dirichlet parameter vector controlling the shape of the distribution.

• B(α)\( B(a) \) is the normalizing constant (Beta function).

  • It ensures that the Dirichlet distribution integrates to 1.

$$ P(\theta \mid \alpha) = \frac{1}{B\alpha} \prod_{i=1}^{K} \theta_i^{\alpha_i-1} $$

• The Dirichlet distribution for three topics (\(K=3\)) is:
  \( P(\theta | \alpha) = \frac{1}{B(\alpha)} \theta_1^{\alpha_1 - 1} \theta_2^{\alpha_2 - 1} \theta_3^{\alpha_3 - 1} \)
  
   \(\theta_1 + \theta_2 + \theta_3 = 1, \quad \text{with } \theta_i \geq 0 \)
   

• where the Beta function \( B(\alpha) \) is:
  \( B(\alpha) = \frac{\Gamma(\alpha_1) \Gamma(\alpha_2) \Gamma(\alpha_3)}{\Gamma(\alpha_1 + \alpha_2 + \alpha_3)} \)
   

• and \( \Gamma(x) \) is the Gamma function:
  \(\Gamma(x) = \int_0^\infty t^{x-1} e^{-t} \, dt\)
  \( e \): Euler's number \( \approx 2.718 \)
  \( t \): Integration Variable

\( B(\alpha) = \frac{\prod_{i=1}^{K} \Gamma(\alpha_i)}{\Gamma\left( \sum_{i=1}^{K} \alpha_i \right)} \)

where \( \Gamma(\cdot) \) is the Gamma function:

\( \Gamma(x) = \int_0^\infty t^{x-1} e^{-t} \, dt \)

The Multivariate Beta Function