In our latest paper, published in Science Advances and accessible here, we argue that a fruitful direction for researching LLMs is engaging them in behavioral experiments inspired by psychology that have traditionally been aimed at understanding human cognition and behavior. In our article, we highlight and summarize theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a “machine psychology” for LLMs that goes beyond performance benchmarks and focuses instead on computational insights that move us toward a better understanding and discovery of emergent abilities and behavioral patterns in LLMs.
Podcast on my latest research
I recently had the pleasure of engaging in an extensive conversation with Stephan Dalügge for his podcast Prioritäten. If you’re interested, you can listen to the first of two episodes here (in German):
To support the incredible work Stefan is doing, please subscribe to his podcast on Apple Podcasts or Spotify.
Superhuman intuitions in language models
Our most recent paper on human-like intuitive decision-making in language models was published at Nature Computational Science. The research is also featured in a newspaper article (in German). We show that large language models, most notably GPT-3, exhibit behavior that strikingly resembles human-like intuition – and the cognitive errors that come with it. However, language models with higher cognitive capabilities, in particular ChatGPT, learned to avoid succumbing to these errors and perform in a hyperrational, superhuman manner. For our experiments, we probe language models with tests that were originally designed to investigate intuitive decision-making in humans.