Ari Holtzman

I’m doing it with LLMs or I’m not doing it at all.

📜 google scholar | ✉️ aholtzman@uchicago.edu | 💡ideahub

image
Holtzman CV.pdf110.5KB

I work on pragmatic narrative—stories about models and models that produce stories.

Events

Some things I’m interested in right now:

  • I want to figure out how Transformer LLMs communicate with themselves in the residual stream. In my opinion, both the alignment and MechInterp communities have become somewhat less ambitious. I’ve often said that I think MechInterp is overrated. I think MechInterp is very cool—just the vast majority of students want to do MechInterp, when I think behavioral work is where much of our insight comes from. But, I’m slowly becoming convinced that Transformer LLMs are simpler than I thought, they just don’t line-up with the kind of explanations people were looking for, so I’m throwing my hat back into MechInterp after being briefly excited and then abandoning it in 2021.
  • I think we should build communication games, games where the main mechanic is communication. There are some (Disco Elysium, Chants of Sennaar, Keep Talking and Nobody Explodes, etc.), but I want ones that are NPC-driven (not MMORPGs or friend-group co-ops) and use LLMs to build a novel social ecosystem you have to navigate. This will happen, let’s be a part of it!
  • The jury is still out on whether passive-learning based AI can ever produce truly interesting media. My guess is yes, but I’m excited to look at this either way. Let’s see if LLMs have a story to tell or two!
  • For other ideas, see my twitter or IdeaHub

Selected Publications

image

AbsenceBench: Language Models Can’t Tell What’s Missing

Harvey Yiyun Fu, Aryan Shrivastava, Jared Moore, Peter West, Chenhao Tan, Ari Holtzman

[paper][dataset][code]

image

Prompting as Scientific Inquiry Ari Holtzman & Chenhao Tan

[paper]

image

Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling

Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, Ari Holtzman

[paper]

image

Generative Models as a Complex Systems Science:

How can we make sense of large language model behavior?

Ari Holtzman, Peter West, Luke Zettlemoyer

[paper

image

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

EMNLP 2022

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

[paper] [code]

image

Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right

EMNLP 2021

=Ari Holtzman, =Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer

= equal contribution

[paper] [project page] [code]

image

The Curious Case of Neural Text Degeneration

ICLR 2019

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi

[paper] [code] [huggingface] [openai api] [fairseq]

Useful Stuff

Materials from the Academic Job Market

Holtzman_Ariel_Cover_Letter_UoChicago.pdf115.9KB
Holtzman_Ariel_CV.pdf105.8KB
Holtzman_Ariel_DEI_Statement.pdf50.6KB
Holtzman_Ariel_Research_Statement.pdf2026.5KB
Holtzman_Ariel_Teaching_Statement.pdf45.4KB