LongMemEval

What LongMemEval contains

LongMemEval was introduced by Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang and Dong Yu, and accepted at ICLR 2025. It has 500 questions covering five abilities: information extraction, multi-session reasoning, knowledge updates, temporal reasoning and abstention.

• An attribute-controlled pipeline, inspired by needle-in-a-haystack tests, builds a coherent, timestamped chat history for each question.
• The system must take in the sessions as they happen and answer only after all of them.
• Three files are released: longmemeval_oracle (evidence sessions only) and two longer haystacks, longmemeval_s and longmemeval_m. A cleaned version was published in September 2025 to remove history that interfered with answer correctness.
• Question types include single-session (user, assistant, preference), multi-session, temporal-reasoning and knowledge-update. Abstention questions are marked by an _abs suffix on the question id.
• Evidence turns carry has_answer labels and each question lists its evidence session ids, so memory recall can be scored at turn and session level, separately from answer accuracy.

Sources: LongMemEval repository (README) · Wu et al., ICLR 2025, arXiv:2410.10813 · Cleaned dataset

Why the update and abstention categories matter most

In production, users change their minds and facts go stale. A system that returns the first-mentioned value fails knowledge-update questions. A system that always answers, even when nothing supports it, fails abstention — which is where hallucinated memory shows up.

Running it reproducibly

• Pin the dataset version (original or cleaned) and the file (_s, _m or oracle).
• Feed sessions in timestamp order with their original dates.
• Use one answer model across systems, and the repository’s evaluation script for QA metrics.
• Report retrieval recall from the evidence labels as well as answer accuracy.
• Publish raw answers and judge decisions.

LongMemEval and Auld

No Auld result is published. The knowledge-update and abstention categories map directly to supersession and the abstained flag on answer, so we expect evaluation there to be the most informative. The authors have also announced LongMemEval-V2, focused on long-term memory in agentic settings.

Frequently asked questions