I’m doing it with LLMs or I’m not doing it at all.
📜 google scholar | ✉️ aholtzman@uchicago.edu |
- Head of Conceptualization Lab
- Assistant Professor at
- UChicago CS
- UChicago DSI
- TTIC (by courtesy)
- Office: Searle 213
- I am actively looking for PhD students to apply through both UChicago CS and the Data Science Institute. Experience with LLMs/generative models is a plus, but not required! I no longer have time to reply to all emails from prospective PhD students but I highly encourage you to check out my lab website and apply!
- Previously:
- Post-Doc @ FAIR @ Meta
- PhD from UW advised by Luke Zettlemoyer
- Natural Language Generation
- Machine Communication
- Generative Models as a Complex Systems Science
Events
- 2025/06—excited to give a talk at MMLS!
- 2024/10—The UChicago Communication and Intelligence Symposium was a great success!
- 2024/07—Officially started at UChicago as an Assistant Professor!
- 2023/09—Started a Post-doc at Meta
- 2023/09—Gave a keynote talk at the International Conference on Social Computing about LLMs as Linguistic Spies
Nightly Research Statement
- Large language models farm corpora for the heuristics and strategies embedded in data. How can we extract the abstractions crystalized in LLMs?
- What can we do with LLMs that has never been done, or even thought of, before? Can we pushback against the current bounds of ineffability to help humans articulate what was previously inarticulable? Can we make narrative video games with a dense graph of states where one learns through bad decisions instead of replaying failures?
Research Foci
My primary interest is in generative models, how they work and how we can get them to generate text and other media that communicate with humans is useful and novel ways. Lately, I’ve been thinking about how language models fit the definition of complex systems, systems in which we understand the low-level components (neurons) but can’t explain or even fully describe the high-level behaviors (e.g., in-context learning) as they emerge with more data and parameters. In the spirit of complex systems, I want to create a taxonomy of model behavior, analogous to the periodic table of elements in Chemistry, which hardly explains complex chemical processes in its own right, but gives a description of elementary components and their interactions that can be used to build-up more complex hypotheses. Currently, we rely on benchmark performance or vague intuitive descriptions to pin-point specific phenomena, which means most hypotheses rely on imprecise vocabulary that won’t stand the test of time.
In the short-term, I’m interested in thinking how we can map out what models can and can’t do, which I believe will naturally relate to long-form generation. It is incredibly difficult to evaluate long-form generation rigorously, and it is hard to show long-form generations in power point slides, which has made coordinating the issues in long-form generation difficult for the academic community. In the medium-term, I think we need to tackle the non-objective aspects of language, as almost all communication is open to interpretation, relying instead on the pragmatic attempt at cooperation to bridge this gap. Focusing on easy-to-evaluate aspects of language doesn’t do it justice. Perhaps looking at indirect evaluation, where we evaluate what generated language can be used for, rather than whether it is “correct” can help move researchers in that direction. My long-term goal is to create discursive machines.
Selected Publications
Generative Models as a Complex Systems Science:
How can we make sense of large language model behavior?
Ari Holtzman, Peter West, Luke Zettlemoyer
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
EMNLP 2021
=Ari Holtzman, =Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer
= equal contribution
The Curious Case of Neural Text Degeneration
ICLR 2019
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi