Four papers accepted at ACL 2025.

The paper “LLMs syntactically adapt their language use to their conversational partner” summarizes Florian Kandra’s BSc thesis, co-supervised by Alexander Koller and Vera Demberg. Kandra finds that GPT-4o adapts the syntax of its language use to that of its conversational partner, based just on the way that the partner talks and not on any particular instructions on language use. These findings mirror earlier results about the way that humans adapt their language use in dialogue.

In “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks”, we introduce JUDGE-BENCH, a dataset that compares human judgments on a variety of NLP tasks to judgments made by LLMs. We find widely different correlations between human and LLM judgments: on some tasks, LLMs reliably give similar judgments as humans, whereas they are totally unreliable on others. Thus, researchers who want to use LLMs as judges should carefully evaluate their suitability on their specific task.

In “Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs”, Yuekun Yao along with our alumna Xiulin Yang investigates whether LLMs are free of an inductive bias, i.e. they could learn all languages equally well, or whether they have inductive biases that make them better at learning humanlike languages. Through experiments with impossible and typologically unattested languages, they find that LLMs do have an inductive bias towards human languages, but it is much weaker than among humans.

In “Exploring Graph Representations of Logical Forms for Language Modeling”, Michael Sullivan introduces GFoLDS, a novel neural language model that reads graph representations of the meaning of a sentence as input. The paper evaluates GFoLDS across a range of NLP tasks - making it the first logic-based LM that can be applied so broadly - and shows that it performs on par with conventional models (such as BERT) while requiring a fraction of the training data.

Congratulations to all authors!