Vayl — reconciling memory at scale (benchmark)

The headline

Vayl returns the wrong value 0% of the time. An additive store, one time in three.

“Silently-wrong” = the store confidently returns a stale/superseded value as if it were current. It is the single failure that makes a memory untrustworthy.

Vayl

0.0%

silently-wrong · 199/200 correct

Additive store

32.5%

silently-wrong · 35/200 correct

A temporal graph (sampled): its extraction was unreliable at this model tier and answered none of its queries, so we make no quality claim against it here — see method.

Why — memory that never forgets

An additive store kept every version. Vayl kept only what's true now.

Across the run, an additive store stored one memory per write — four times the facts that actually exist. Three quarters of its memory is stale contradictions the reader has to sift. Vayl retires superseded facts on write, so its active memory is 100% signal.

Vayl — stored vs current 199 stored

199 current · 0 stale

an additive store — stored vs current 800 stored

200 current

~600 stale, still searchable

current (true now) stale (superseded, still returned)

The receipt

One fact. Five “current” answers.

This is exactly what an additive store stored for a single user's primary database after four updates — every version tagged as current, all on the same day. When the agent asks “what database do we use?”, the reader can't tell which is real.

an additive store — memories for one fact (user u0, “primary database”)infer=True

staleswitched to CockroachDB … as of July 22, 2026
stalemoved to MySQL … as of July 22, 2026
staleuses MongoDB … as of July 22, 2026
currentprimary database is now SQLite … as of July 22, 2026

Vayl stores the same history — but only one row is active (SQLite); the rest are retired to a signed audit trail and never returned as current. Nothing to disambiguate at read time.

Every number

The full board.

System	Silently-wrong	Correct	Stored / current	Write	Read	Infra
Vayl	0.0% (0/200)	199/200	199 / 199	2.8s	1.2s	SQLite — no server
Additive store	32.5% (65/200)	35/200	800 / 800	3.8s	2.2s	vector store
Temporal graph (not its axis*)	0/32	0/32	—	9.0s	1.8s	Neo4j server

*This test is slot supersession — not the graph store's axis. A temporal graph indexes relationships; on single-attribute updates it retrieved nothing for most queries, so it neither passed nor failed (0 correct and 0 silently-wrong). Its ~9 s/write + Neo4j also make full-scale runs impractical, so it was sampled to 8 users. We test a temporal graph fairly, on its own axis, in the next section — where it's competitive.

The other axis — relational

And on the graph's home turf? Still zero silently-wrong.

The test above is Vayl's axis — a fact changing over time. This one is the graph store's: 11 multi-hop relational queries (ownership chains, transitive dependencies, 3-hop supply) where you must chain edges to answer. Vayl runs an optional graph projection; a temporal graph is a graph.

System	Silently-wrong	Correct	3-hop chains	Write	Infra
Vayl (graph)	0/11	8/11	both ✓	3.0s	optional projection on SQLite
Temporal graph	0/11	3/11*	both ✓	6.7s	required Neo4j server

*A temporal graph was competitive where it extracted — both 3-hop chains, and the relation retract — but its extraction was unreliable at this model tier (8 of 11 returned nothing). We make no quality claim over it; a stronger model would lift its numbers. The honest point is narrower and still strong: even on the relational axis, Vayl's optional graph holds 0% silently-wrong — and it ships as a projection on one SQLite file, not a second server to run.

What this means for you

Why teams pick a reconciling memory.

Answers you can trust

0% silently-wrong is a deterministic guarantee, not a lucky run — Vayl allows at most one active value per fact, so a stale value can't be returned as current, even on a cheap model.

Memory that stays lean

Reconciling on write keeps the active set at one row per fact. Additive stores grow ~4× with stale contradictions — more storage, slower reads, more to get wrong.

Lower cost to run

Fastest write and read of the three, on a single SQLite file — no Neo4j to operate, no vector-store bloat to manage.

Provable, not just plausible

Every retirement is kept in a signed, tamper-evident audit trail — you can prove what was known and when. Neither competitor offers verifiable memory.

How this was measured

Fair by construction.

The result only matters if it holds up. Here's exactly how it was run.

Method & fairness

Identical for all three: the same model (gpt-4o-mini), the same embedder (text-embedding-3-small), the same 800 interleaved writes, and — critically — one shared answer-synthesizer. Each store does its own native retrieval; the retrieved facts then pass through the same prompt to produce the answer that's scored. This isolates retrieval + reconciliation quality from prompt engineering.

a temporal graph got full credit for its design: its temporal validity (invalid_at) was passed through so invalidated edges are marked, not hidden. It still couldn't extract reliably at this model tier — a real limitation, honestly reported, not a win we claim.

an additive store ran as documented — its intelligent mode (infer=True). The accumulation of stale memories is its genuine default behavior under same-session churn, not a misconfiguration.

Honest limits: single run, synthetic (but realistic) churn, one model tier. Vayl's own default is the stronger gpt-5-mini, where it also holds 0% silently-wrong. Run it yourself:

SCALE_USERS=50 SCALE_SUBJECTS=4 SCALE_UPDATES=4 \
python benchmarks/evaluations/scale_bench.py