The LoCoMo benchmark

What LoCoMo contains

LoCoMo was introduced in “Evaluating Very Long-Term Conversational Memory of LLM Agents” (arXiv:2402.17753). Its authors start from the observation that earlier long-term dialogue work evaluated contexts of no more than five chat sessions.

• Conversations come from a machine–human pipeline. LLM-based agents talk with each other, grounded in personas and temporal event graphs, and can share and react to images.
• Human annotators verified and edited the generated conversations for long-range consistency and fidelity to the event graphs.
• Each conversation averages about 300 turns and 9K tokens, over up to 35 sessions.
• Three evaluation tasks: question answering, event summarization and multimodal dialogue generation.

The paper reports that long-context LLMs and RAG help, but still lag substantially behind human performance, especially on long-range temporal and causal dynamics.

Sources: Maharana et al., “Evaluating Very Long-Term Conversational Memory of LLM Agents”, arXiv:2402.17753

Gold answers and adjudication

Like any human-annotated set, LoCoMo has items where reasonable answers differ from the reference. A published score should say how answers were judged (exact match, F1 or a model judge), which tasks and question categories were included, and how ambiguous items were handled.

Publishing a comparable result

1. 01

   Fix the answer model

   Report it and its settings.

2. 02

   Report per category

   Not only an average.

3. 03

   Publish the judge prompt

   And agreement with human labels on a sample.

4. 04

   Include baselines

   Full-context and no-memory runs with the same model.

What LoCoMo does not test

• Multi-user access control.
• Business data outside conversation (CRM, documents).
• Histories much longer than its conversations — around 9K tokens on average.

Frequently asked questions