Ari Holtzman

I’m doing it with LLMs or I’m not doing it at all.

📜 google scholar | ✉️ aholtzman@uchicago.edu |

image
  • Head of Conceptualization Lab
  • Assistant Professor at
  • Office: Searle 213
  • I am actively looking for PhD students to apply through both UChicago CS and the Data Science Institute. Experience with LLMs/generative models is a plus, but not required! I no longer have time to reply to all emails from prospective PhD students but I highly encourage you to check out my lab website and apply!
  • Previously:
  • Natural Language Generation
  • Machine Communication
  • Generative Models as a Complex Systems Science
Holtzman CV.pdf110.5KB

Events

  • 2025/06—excited to give a talk at MMLS!
  • 2024/10—The UChicago Communication and Intelligence Symposium was a great success!
  • 2024/07—Officially started at UChicago as an Assistant Professor!
  • 2023/09—Started a Post-doc at Meta
  • 2023/09—Gave a keynote talk at the International Conference on Social Computing about LLMs as Linguistic Spies

Nightly Research Statement

  1. Large language models farm corpora for the heuristics and strategies embedded in data. How can we extract the abstractions crystalized in LLMs?
  2. What can we do with LLMs that has never been done, or even thought of, before? Can we pushback against the current bounds of ineffability to help humans articulate what was previously inarticulable? Can we make narrative video games with a dense graph of states where one learns through bad decisions instead of replaying failures?

Research Foci

My primary interest is in generative models, how they work and how we can get them to generate text and other media that communicate with humans is useful and novel ways. Lately, I’ve been thinking about how language models fit the definition of complex systems, systems in which we understand the low-level components (neurons) but can’t explain or even fully describe the high-level behaviors (e.g., in-context learning) as they emerge with more data and parameters. In the spirit of complex systems, I want to create a taxonomy of model behavior, analogous to the periodic table of elements in Chemistry, which hardly explains complex chemical processes in its own right, but gives a description of elementary components and their interactions that can be used to build-up more complex hypotheses. Currently, we rely on benchmark performance or vague intuitive descriptions to pin-point specific phenomena, which means most hypotheses rely on imprecise vocabulary that won’t stand the test of time.

In the short-term, I’m interested in thinking how we can map out what models can and can’t do, which I believe will naturally relate to long-form generation. It is incredibly difficult to evaluate long-form generation rigorously, and it is hard to show long-form generations in power point slides, which has made coordinating the issues in long-form generation difficult for the academic community. In the medium-term, I think we need to tackle the non-objective aspects of language, as almost all communication is open to interpretation, relying instead on the pragmatic attempt at cooperation to bridge this gap. Focusing on easy-to-evaluate aspects of language doesn’t do it justice. Perhaps looking at indirect evaluation, where we evaluate what generated language can be used for, rather than whether it is “correct” can help move researchers in that direction. My long-term goal is to create discursive machines.

Selected Publications

image

Generative Models as a Complex Systems Science:

How can we make sense of large language model behavior?

Ari Holtzman, Peter West, Luke Zettlemoyer

[paper]

image

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers*, Artidoro Pagnoni*, Ari Holtzman, Luke Zettlemoyer

[paper] [code]

image

Contrastive Decoding: Open-ended Text Generation as Optimization

ACL 2023

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis

[paper] [code]

image

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

EMNLP 2022

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

[paper] [code]

image

Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right

EMNLP 2021

=Ari Holtzman, =Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer

= equal contribution

[paper] [project page] [code]

image

The Curious Case of Neural Text Degeneration

ICLR 2019

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi

[paper] [code] [huggingface] [openai api] [fairseq]

Useful Stuff

Materials from the Academic Job Market

Holtzman_Ariel_Cover_Letter_UoChicago.pdf115.9KB
Holtzman_Ariel_CV.pdf105.8KB
Holtzman_Ariel_DEI_Statement.pdf50.6KB
Holtzman_Ariel_Research_Statement.pdf2026.5KB
Holtzman_Ariel_Teaching_Statement.pdf45.4KB