benchmark.html

<title>Vayl — reconciling memory at scale (benchmark)</title>
<style>
  :root{
    --paper:#F6F7F9; --ink:#0F1216; --muted:#5B6472; --faint:#8A94A6;
    --hair:#E4E7EC; --card:#FFFFFF; --vayl:#12876A; --vayl-soft:#E4F1EC;
    --alarm:#CC4436; --alarm-soft:#FBE9E7; --grey:#9AA3B0; --grey-soft:#EDEFF2;
    --mono:ui-monospace,"SF Mono",Menlo,Consolas,monospace;
    --sans:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
  }
  @media (prefers-color-scheme:dark){
    :root{--paper:#0C0E11; --ink:#EDEFF2; --muted:#9AA3B0; --faint:#6B7482;
      --hair:#222730; --card:#14171B; --vayl:#33C89E; --vayl-soft:#12352C;
      --alarm:#F0736C; --alarm-soft:#3A1D1B; --grey:#7A8494; --grey-soft:#1C2129;}
  }
  :root[data-theme="dark"]{--paper:#0C0E11; --ink:#EDEFF2; --muted:#9AA3B0; --faint:#6B7482;
    --hair:#222730; --card:#14171B; --vayl:#33C89E; --vayl-soft:#12352C;
    --alarm:#F0736C; --alarm-soft:#3A1D1B; --grey:#7A8494; --grey-soft:#1C2129;}
  :root[data-theme="light"]{--paper:#F6F7F9; --ink:#0F1216; --muted:#5B6472; --faint:#8A94A6;
    --hair:#E4E7EC; --card:#FFFFFF; --vayl:#12876A; --vayl-soft:#E4F1EC;
    --alarm:#CC4436; --alarm-soft:#FBE9E7; --grey:#9AA3B0; --grey-soft:#EDEFF2;}

  *{box-sizing:border-box}
  body{margin:0;background:var(--paper);color:var(--ink);font-family:var(--sans);
    line-height:1.6;-webkit-font-smoothing:antialiased}
  .wrap{max-width:820px;margin:0 auto;padding:0 24px}
  .num{font-family:var(--mono);font-variant-numeric:tabular-nums}
  h1,h2,h3{line-height:1.15;text-wrap:balance;letter-spacing:-.02em}

  header.top{padding:72px 0 40px;border-bottom:1px solid var(--hair)}
  .eyebrow{font-family:var(--mono);font-size:12.5px;letter-spacing:.14em;text-transform:uppercase;
    color:var(--vayl);margin:0 0 18px}
  h1{font-size:clamp(30px,5.2vw,46px);font-weight:700;margin:0 0 18px}
  .lede{font-size:clamp(17px,2.4vw,19px);color:var(--muted);max-width:60ch;margin:0}
  .setup{margin-top:24px;font-family:var(--mono);font-size:13px;color:var(--faint);
    display:flex;flex-wrap:wrap;gap:8px 18px}
  .setup b{color:var(--ink);font-weight:600}

  section{padding:52px 0;border-bottom:1px solid var(--hair)}
  .kicker{font-family:var(--mono);font-size:12px;letter-spacing:.12em;text-transform:uppercase;
    color:var(--faint);margin:0 0 20px}
  h2{font-size:clamp(22px,3.4vw,28px);font-weight:650;margin:0 0 8px}
  p{margin:0 0 16px;max-width:64ch}
  p.sub{color:var(--muted)}

  /* hero stat cards */
  .stats{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-top:8px}
  .stat{border:1px solid var(--hair);border-radius:14px;padding:26px;background:var(--card)}
  .stat .big{font-family:var(--mono);font-size:clamp(42px,9vw,64px);font-weight:600;
    line-height:1;letter-spacing:-.03em;font-variant-numeric:tabular-nums}
  .stat .lab{margin-top:12px;font-size:14px;color:var(--muted)}
  .stat .who{font-family:var(--mono);font-size:12px;letter-spacing:.08em;text-transform:uppercase;
    display:inline-block;margin-bottom:14px;padding:3px 9px;border-radius:999px}
  .win{border-color:color-mix(in srgb,var(--vayl) 40%,var(--hair))}
  .win .big{color:var(--vayl)} .win .who{background:var(--vayl-soft);color:var(--vayl)}
  .lose .big{color:var(--alarm)} .lose .who{background:var(--alarm-soft);color:var(--alarm)}
  .stat small{color:var(--faint)}

  /* footprint bars */
  .bars{margin-top:26px;display:flex;flex-direction:column;gap:22px}
  .barrow .barhead{display:flex;justify-content:space-between;align-items:baseline;margin-bottom:8px}
  .barrow .name{font-weight:600} .barrow .name span{color:var(--faint);font-weight:400}
  .barrow .tot{font-family:var(--mono);font-size:14px;color:var(--muted);font-variant-numeric:tabular-nums}
  .track{height:34px;border-radius:8px;overflow:hidden;display:flex;background:var(--grey-soft)}
  .seg{height:100%;display:flex;align-items:center;padding:0 10px;font-family:var(--mono);
    font-size:12px;color:#fff;white-space:nowrap;overflow:hidden}
  .seg.cur{background:var(--vayl)} .seg.stale{background:var(--alarm)}
  .legend{display:flex;gap:18px;margin-top:16px;font-size:13px;color:var(--muted);flex-wrap:wrap}
  .legend i{display:inline-block;width:11px;height:11px;border-radius:3px;margin-right:7px;vertical-align:middle}
  .dot-cur{background:var(--vayl)} .dot-stale{background:var(--alarm)}

  /* ledger example */
  .ledger{margin-top:22px;border:1px solid var(--hair);border-radius:12px;overflow:hidden;background:var(--card)}
  .ledger .lh{padding:12px 16px;border-bottom:1px solid var(--hair);font-family:var(--mono);
    font-size:12.5px;color:var(--muted);display:flex;justify-content:space-between}
  .ledger ul{list-style:none;margin:0;padding:8px 0}
  .ledger li{font-family:var(--mono);font-size:13px;padding:7px 16px;display:flex;gap:12px;align-items:baseline}
  .ledger li .tag{font-size:10px;letter-spacing:.06em;text-transform:uppercase;padding:2px 7px;
    border-radius:5px;flex-shrink:0}
  .stale-tag{background:var(--alarm-soft);color:var(--alarm)} .cur-tag{background:var(--vayl-soft);color:var(--vayl)}
  .ledger li.is-stale{color:var(--faint);text-decoration:line-through;text-decoration-color:var(--alarm)}

  /* table */
  .scroll{overflow-x:auto;margin-top:20px}
  table{border-collapse:collapse;width:100%;min-width:560px;font-size:14px}
  th,td{text-align:left;padding:12px 14px;border-bottom:1px solid var(--hair)}
  th{font-family:var(--mono);font-size:11.5px;letter-spacing:.06em;text-transform:uppercase;color:var(--faint);font-weight:500}
  td.n{font-family:var(--mono);font-variant-numeric:tabular-nums}
  tr.vayl td{background:var(--vayl-soft)} tr.vayl td:first-child{font-weight:700;color:var(--vayl)}
  .g{color:var(--vayl);font-weight:600} .r{color:var(--alarm);font-weight:600}

  /* takeaways */
  .cards{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-top:8px}
  .tk{border:1px solid var(--hair);border-radius:12px;padding:22px;background:var(--card)}
  .tk h3{font-size:16px;margin:0 0 8px} .tk p{font-size:14px;color:var(--muted);margin:0}

  details{margin-top:10px;border:1px solid var(--hair);border-radius:10px;background:var(--card)}
  summary{cursor:pointer;padding:14px 18px;font-weight:600;font-size:15px}
  details[open] summary{border-bottom:1px solid var(--hair)}
  .dbody{padding:18px}
  code{font-family:var(--mono);font-size:.9em;background:var(--grey-soft);padding:2px 6px;border-radius:5px}
  .repro{font-family:var(--mono);font-size:12.5px;background:var(--card);border:1px solid var(--hair);
    border-radius:10px;padding:14px 16px;overflow-x:auto;color:var(--muted)}
  footer{padding:44px 0 72px;color:var(--faint);font-size:13px}
  footer a{color:var(--vayl);text-decoration:none}
  .caveat{font-size:13.5px;color:var(--faint);border-left:2px solid var(--hair);padding-left:14px;margin-top:20px}

  @media (max-width:640px){
    .stats,.cards{grid-template-columns:1fr}
    header.top{padding:52px 0 32px}
  }
</style>

<header class="top">
  <div class="wrap">
    <p style="margin:0 0 22px"><a href="index.html" style="font-family:var(--mono);font-size:13px;letter-spacing:.04em;color:var(--faint);text-decoration:none">&larr; Vayl</a></p>
    <p class="eyebrow">Benchmark · reconciling memory at scale</p>
    <h1>When the facts keep changing, does your memory still tell the truth?</h1>
    <p class="lede">We ran Vayl head-to-head against an additive store and a temporal graph on the one thing an agent's
      memory has to get right: after a fact is updated many times, does a lookup return the
      <em>current</em> value — or a stale one, confidently?</p>
    <div class="setup">
      <span><b>800</b> writes</span><span><b>200</b> facts, each changed up to 4×</span>
      <span><b>50</b> users</span><span>model <b>gpt-4o-mini</b></span><span>same embedder + synthesizer for all</span>
    </div>
  </div>
</header>

<section>
  <div class="wrap">
    <p class="kicker">The headline</p>
    <h2>Vayl returns the wrong value 0% of the time. An additive store, one time in three.</h2>
    <p class="sub">“Silently-wrong” = the store confidently returns a stale/superseded value as if it
      were current. It is the single failure that makes a memory untrustworthy.</p>
    <div class="stats">
      <div class="stat win">
        <span class="who">Vayl</span>
        <div class="big">0.0%</div>
        <div class="lab">silently-wrong · <span class="num">199/200</span> correct</div>
      </div>
      <div class="stat lose">
        <span class="who">Additive store</span>
        <div class="big">32.5%</div>
        <div class="lab">silently-wrong · <span class="num">35/200</span> correct</div>
      </div>
    </div>
    <p class="caveat">A temporal graph (sampled): its extraction was unreliable at this model tier and
      answered none of its queries, so we make <b>no</b> quality claim against it here — see method.</p>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">Why — memory that never forgets</p>
    <h2>An additive store kept every version. Vayl kept only what's true now.</h2>
    <p class="sub">Across the run, an additive store stored one memory <em>per write</em> — four times the facts that
      actually exist. Three quarters of its memory is stale contradictions the reader has to sift.
      Vayl retires superseded facts on write, so its active memory is 100% signal.</p>

    <div class="bars">
      <div class="barrow">
        <div class="barhead"><span class="name">Vayl <span>— stored vs current</span></span>
          <span class="tot num">199 stored</span></div>
        <div class="track"><div class="seg cur" style="width:100%">199 current · 0 stale</div></div>
      </div>
      <div class="barrow">
        <div class="barhead"><span class="name">an additive store <span>— stored vs current</span></span>
          <span class="tot num">800 stored</span></div>
        <div class="track">
          <div class="seg cur" style="width:25%">200 current</div>
          <div class="seg stale" style="width:75%">~600 stale, still searchable</div>
        </div>
      </div>
    </div>
    <div class="legend">
      <span><i class="dot-cur"></i>current (true now)</span>
      <span><i class="dot-stale"></i>stale (superseded, still returned)</span>
    </div>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">The receipt</p>
    <h2>One fact. Five “current” answers.</h2>
    <p class="sub">This is exactly what an additive store stored for a single user's <b>primary database</b> after
      four updates — every version tagged as current, all on the same day. When the agent asks
      “what database do we use?”, the reader can't tell which is real.</p>
    <div class="ledger">
      <div class="lh"><span>an additive store — memories for one fact (user u0, “primary database”)</span><span>infer=True</span></div>
      <ul>
        <li class="is-stale"><span class="tag stale-tag">stale</span>switched to CockroachDB … as of July 22, 2026</li>
        <li class="is-stale"><span class="tag stale-tag">stale</span>moved to MySQL … as of July 22, 2026</li>
        <li class="is-stale"><span class="tag stale-tag">stale</span>uses MongoDB … as of July 22, 2026</li>
        <li><span class="tag cur-tag">current</span>primary database is now SQLite … as of July 22, 2026</li>
      </ul>
    </div>
    <p class="caveat">Vayl stores the same history — but only <b>one</b> row is <code>active</code>
      (<span class="num">SQLite</span>); the rest are retired to a signed audit trail and never
      returned as current. Nothing to disambiguate at read time.</p>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">Every number</p>
    <h2>The full board.</h2>
    <div class="scroll">
      <table>
        <thead><tr><th>System</th><th>Silently-wrong</th><th>Correct</th><th>Stored / current</th>
          <th>Write</th><th>Read</th><th>Infra</th></tr></thead>
        <tbody>
          <tr class="vayl"><td>Vayl</td><td class="n g">0.0% <small>(0/200)</small></td>
            <td class="n">199/200</td><td class="n">199 / 199</td><td class="n">2.8s</td>
            <td class="n">1.2s</td><td>SQLite — no server</td></tr>
          <tr><td>Additive store</td><td class="n r">32.5% <small>(65/200)</small></td>
            <td class="n">35/200</td><td class="n">800 / 800</td><td class="n">3.8s</td>
            <td class="n">2.2s</td><td>vector store</td></tr>
          <tr><td>Temporal graph <small>(not its axis*)</small></td><td class="n">0/32</td>
            <td class="n">0/32</td><td class="n">—</td><td class="n">9.0s</td>
            <td class="n">1.8s</td><td>Neo4j server</td></tr>
        </tbody>
      </table>
    </div>
    <p class="caveat">*This test is <b>slot supersession — not the graph store's axis.</b> A temporal graph indexes relationships; on single-attribute updates it retrieved nothing for most queries, so
      it neither passed nor failed (0 correct <em>and</em> 0 silently-wrong). Its ~9 s/write + Neo4j also
      make full-scale runs impractical, so it was sampled to 8 users. We test a temporal graph <b>fairly, on its
      own axis, in the next section</b> — where it's competitive.</p>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">The other axis — relational</p>
    <h2>And on the graph's home turf? Still zero silently-wrong.</h2>
    <p class="sub">The test above is Vayl's axis — a fact changing over time. This one is the graph store's:
      11 multi-hop relational queries (ownership chains, transitive dependencies, 3-hop supply) where you
      must chain edges to answer. Vayl runs an <em>optional</em> graph projection; a temporal graph <em>is</em> a graph.</p>
    <div class="scroll">
      <table>
        <thead><tr><th>System</th><th>Silently-wrong</th><th>Correct</th><th>3-hop chains</th><th>Write</th><th>Infra</th></tr></thead>
        <tbody>
          <tr class="vayl"><td>Vayl (graph)</td><td class="n g">0/11</td><td class="n">8/11</td>
            <td class="n g">both ✓</td><td class="n">3.0s</td><td>optional projection on SQLite</td></tr>
          <tr><td>Temporal graph</td><td class="n">0/11</td><td class="n">3/11*</td>
            <td class="n">both ✓</td><td class="n">6.7s</td><td>required Neo4j server</td></tr>
        </tbody>
      </table>
    </div>
    <p class="caveat">*A temporal graph was competitive where it extracted — both 3-hop chains, and the relation
      retract — but its extraction was unreliable at this model tier (8 of 11 returned nothing). We make
      <b>no</b> quality claim over it; a stronger model would lift its numbers. The honest point is
      narrower and still strong: even on the relational axis, Vayl's optional graph holds <b>0%
      silently-wrong</b> — and it ships as a projection on one SQLite file, not a second server to run.</p>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">What this means for you</p>
    <h2>Why teams pick a reconciling memory.</h2>
    <div class="cards">
      <div class="tk"><h3>Answers you can trust</h3><p>0% silently-wrong is a deterministic guarantee,
        not a lucky run — Vayl allows at most one active value per fact, so a stale value can't be
        returned as current, even on a cheap model.</p></div>
      <div class="tk"><h3>Memory that stays lean</h3><p>Reconciling on write keeps the active set at
        one row per fact. Additive stores grow ~4× with stale contradictions — more storage, slower
        reads, more to get wrong.</p></div>
      <div class="tk"><h3>Lower cost to run</h3><p>Fastest write and read of the three, on a single
        SQLite file — no Neo4j to operate, no vector-store bloat to manage.</p></div>
      <div class="tk"><h3>Provable, not just plausible</h3><p>Every retirement is kept in a signed,
        tamper-evident audit trail — you can prove what was known and when. Neither competitor offers
        verifiable memory.</p></div>
    </div>
  </div>
</section>

<section>
  <div class="wrap">
    <p class="kicker">How this was measured</p>
    <h2>Fair by construction.</h2>
    <p class="sub">The result only matters if it holds up. Here's exactly how it was run.</p>
    <details open><summary>Method &amp; fairness</summary>
      <div class="dbody">
        <p><b>Identical for all three:</b> the same model (<code>gpt-4o-mini</code>), the same embedder
          (<code>text-embedding-3-small</code>), the same 800 interleaved writes, and — critically —
          <b>one shared answer-synthesizer</b>. Each store does its own native retrieval; the retrieved
          facts then pass through the same prompt to produce the answer that's scored. This isolates
          <em>retrieval + reconciliation</em> quality from prompt engineering.</p>
        <p><b>a temporal graph got full credit for its design:</b> its temporal validity
          (<code>invalid_at</code>) was passed through so invalidated edges are marked, not hidden. It
          still couldn't extract reliably at this model tier — a real limitation, honestly reported,
          not a win we claim.</p>
        <p><b>an additive store ran as documented</b> — its intelligent mode (<code>infer=True</code>). The
          accumulation of stale memories is its genuine default behavior under same-session churn, not
          a misconfiguration.</p>
        <p><b>Honest limits:</b> single run, synthetic (but realistic) churn, one model tier. Vayl's
          own default is the stronger <code>gpt-5-mini</code>, where it also holds 0% silently-wrong.
          Run it yourself:</p>
        <div class="repro">SCALE_USERS=50 SCALE_SUBJECTS=4 SCALE_UPDATES=4 \<br>&nbsp;&nbsp;python benchmarks/evaluations/scale_bench.py</div>
      </div>
    </details>
  </div>
</section>

<footer>
  <div class="wrap">
    Vayl — reconciling memory for AI agents. Benchmark harness and raw results are open in the
    repository (<code>benchmarks/evaluations/scale_bench.py</code>, <code>docs/COMPARISON.md</code>).
    Figures are a single run on gpt-4o-mini and reproducible with the command above.
  </div>
</footer>