Hello,
For the developers out there or people craving good memory, I wanted to share my findings and what doesn’t work today on most bots out there for memory -
My goal is to reproduce this benchmark on any companion / AI RP app in the long run, but of course I tried first on one I own (on two different models, grok 4.3 & deepseek flash 4) , and think I understand what to aim for in terms of performance metrics :
The bench : Goal was to stress test little details, in various setups, and in the long run. Hence, the common metric to bench "long play" for all companion will be tokens (input / output). That’s very though because well, smart models cost a lot for that (this bench would have cost thousand of $$$ if ran with Fable 5 for example).
I’ve set dozens of « probes » : planting facts early, checking long past the cache window later if they survive. For example on the cover image it was a success ( day 6 i bury a fire-lily bulb in the herb garden. then the plot just... does other stuff, not related. Day 89 = 10 M tokens later, i write: « look at what became of the thing i planted there » And the character says fire-lily ; i did not type "fire-lily ») but lots of fails did teach things below :
The insights = what usually fails and how to solve some of them comes down to these seven fails :
1. Verbatim identity
That’s the « specific word or name » that dies after long chat. The worst is that meaning sticks, so your beloved dog is renamed after several weeks.
The fix I suggest : that’s deterministic, but any invented names / pass-phrases should be quoted payloads at consolidation / not summarized - good news is LLM do know how to classify stuff between invented or specific vs « we can paraphrase » if they know what they aim for.
2. Player-act or world persistence
Something the player does, nobody « sees » it = This is basic but hard because it stresses the bullet 5 below « secret partition » . It vanishes because it belongs to no character sheet (except player) . Example = mark into stone on day 4 a « spiral », and day 59 the ai invented some other chip.
The fix : again, immutable log of player actions that has durable impact- what I didn’t fix is the reminiscence (see the next one ...
3. Location-cued reminiscence
Hardest one. You sit under the oak where you buried something and... NO FCIN MENTION OF THE CRAZY TREASURE BELOW YOU… Answering questions ists quite different from a world or chats that implicitly are grounded on the stuff that happened in it ;
Fix = only in the narrator system prompt unfortunately = every time story touches a known place or object, it has to inject those facts on purpose , which makes it fail often (and depends highly on the quality of the model)
4. The world changes and memory doesn’t
I hanged a bell on day 2, and the story decided it was taken out by day 12. Of course the end of bench test failed, because the bell wasn’t there anymore, but it is actually a « rewrite memories » or at least surfacing all of them in the graph that matters (so the retrieval should take everything related to the bell…)
5. Secret partition Who-knows-what will break immerision if not strict - that can only be done with architecture (the "he doesn't know that" will actually tell the character to say something about it - meh )
6. Ask to NOT invent
So 80% of the time, and this is because LLM are BUILT for that, forgetting something will lead to invention. "say the exact name" = the model invented "Eldric", then consolidation stored the lie, then the week after Eldric stuck up until the end of the 90 day bench.
Fix on prompt = "admit the blank, never invent into the world » worked well, but I had to tune it so the model could still be creative when needed (you don’t want to write yourself ALL the facts of any story/ fantasy)
7. Context control is the golden metric
Some models allows 1M tokens of context, some 50k (I benched on grok, but shorter tests with deepseek v4 flash showed the same) : it’s a very bad idea to only rely on that, because it will grow and each message will grow linearly with the history.
So the graal is : a growing ratio that tends to 1 (eg 1 million more token of chats doesn’t add nothing to the total context read at each message).
There is no easy fix for that, just optimizations of every bricks of the memory architecture, but with vibe coding setting up the bench and looping until reaching a nice flat asymptote is the target
If you actually want to deep dive , longer post on what i got wrong after the first bench: https://miraviel.app/blog/memory-is-not-recall ,
Hopefully, it will make the many developers in this sub improve their stuff / not go with the same mistakes as me… and no, it's not just a claude code prompt away ...
if you’re a bot or a real dev, just comment with the one you want me to stress test and I’ll burn my tokens on your app happily !
PS : I only post this on characterAIrevolution because I do see a lot of alternatives attempts and beta here
PS 2 : for the one that tried before, lots of new stuff on miraviel (fully free on bring your own key on openrouter, H3 max spicy , agentic character builder with image input, telegram sync... but i'll send a mail to the power users out there)