We discuss discrete diffusion models that offer a unified framework for jointly modeling categorical data such as text and images. We present a new model that we have developed for language generation called the Anchored Diffusion Language Model (ADLM). ADLM is grounded in a novel two-stage framework that first predicts distributions over important tokens via an anchor network (e.g., key words or low-frequency words that anchor a sentence), and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches. Project page:  https://anchored-diffusion-llm.github.io/

Seminar Recording