Paper accepted at ICLR 2024.

The paper “Closing the Curious Case of Neural Text Degeneration” emerged from Matt Finlayson’s research as an intern at AI2. It frames the challenge of sampling good texts from an LLM as figuring out which possible next token truly have a nonzero probability and which tokens only look like it because of some smoothing that the LLM implicitly performs. The paper makes a beautiful mathematical argument to explain why existing methods for truncation sampling are effective but crude at addressing this challenge, and proposes a new sampling strategy that directly attempts to predict next tokens with nonzero probability.