Evaluation

Benchmarks and methodology

How agent memory should be evaluated, what the public suites measure, and how to read scores — including ours, when we publish them.

Auld on public suitesresults

No Auld measurement is published.

When Auld publishes a result on public suites, it will include the configuration, the judge and its prompts, raw per-question outputs, and the baselines in the methodology.

The public suites

Suite	What it stresses	Page
LoCoMo	Very long multi-session conversations (about 300 turns over up to 35 sessions); question answering, event summarization and multimodal dialogue generation	LoCoMo
LongMemEval	500 questions on five abilities: information extraction, multi-session reasoning, knowledge updates, temporal reasoning, abstention	LongMemEval
BEAM	100 generated conversations up to 10M tokens, with 2,000 validated questions across many memory abilities	BEAM
MemoryBench (THUIR)	Whether LLM systems learn from accumulated user feedback: 28 datasets, 3 domains, simulated feedback	MemoryBench

What a score can and cannot tell you

Scores depend on the answer model, the judge and its prompt as much as on the memory system.
Small suites have wide confidence intervals; a few points apart may not be a real difference.
Public conversations differ from your data — run your own test set too.

Auld results

This site does not publish Auld benchmark scores. We would rather publish one well-documented result than a table of numbers without method. The methodology page is the checklist we hold ourselves to.

Evaluate on your own data.

Load a sample with real timestamps and run the four-operation test. We are happy to help design it.

Talk to us Quickstart