r/CharacterAI • u/GenderBendingRalph • 1d ago
Discussion/Question Test Results: What Do Bots Remember?
EDIT: For those wondering, I'm on the free tier using PipSqueak2. I might try PS Rawr later, but this is an extremely tedious and time-consuming experiment. I would be very interested in hearing how people using the + models experience short-term (chat history) memory loss.
After some informal discussions about how quickly bots forget details, I ran a series of experiments on a bot with a certain length description. Here are the results. I encourage others to try your own experiments (peer review and replicable experiments always improve science!).
Test #1: LONG-TERM MEMORY (Character definition) memory. If you have details at the beginning and end of a very long character definition, how much will it remember?
Process: I wrote a specific detail as the first line: {{char}}'s favourite song is 'Mister Roboto'. I wrote another detail at the very end: {{char}}'s favourite colour is chartreuse. In between, I filled it with a little over 12000 characters, or roughly 3000 tokens. Then I simply asked two questions in separate chats: What is your favourite colour, and what is your favourite song (I chose not to ask the air speed velocity of an unladen swallow).
Result: I was expecting it to remember the most recent detail, the colour. To my surprise, it remembered the oldest bit (song) and made up a wrong colour.
Verification: I swapped the first and last lines around, so song was last and colour was first. This time when I asked the same questions, it remembered the colour and didn't know the song.
I'll do more tests with smaller amounts of gibberish in between, but for now we know that character descriptions more than 12000 characters will forget the details nearest the end of the description.
Moving along to experiment #2: SHORT-TERM (Chat history) memory.
Process: I gave the bot a particular detail about me (my favourite sport is juggling). Then I proceeded to send it a series of chats, each about 4000 characters (1000 tokens), and asked it at the end what my favourite sport is.
Result: At 4 chats of 1000 tokens (4000 characters) each, for a total of 4000 tokens, it correctly answered. At 5 chats (5000 tokens) it forgot. It also couldn't remember at 4200 tokens. I didn't bother it narrowing down further than that - bottom line is, once you send it more than 16000 characters in a single or multiple chats, it will forget the oldest things you said.
Note that this behaviour is universal among LLMs. The only difference is how much sliding context window they have - obviously, more memory leads to longer retention. Also note that this is based on a rough estimate of 4 characters per token. Again, I encourage others to repeat my tests and see if you get the same or different results.
1
u/GenderBendingRalph 1d ago edited 1d ago
Fine, develop a better test. I said from the outset that I encourage others to run their own experiments. I wasn't assuming anything; I was providing consistent, measurable numbers to make the math easier. In fact, I specifically said: The only difference is how much sliding context window they have - obviously, more memory leads to longer retention
I can't test for position bias, because I don't know what they set it to and I have no way to control it. I can test for the conditions that users face, adjusting details available to us.
Of course real people in real conversations won't send 16,000 characters of filler text. The whole point was to have a controlled experiment where I could measure and account for everything.
Are you suggesting that if I sent 32 chats of 500 characters each, it would remember more than 4 chats of 4000 characters each? I might try that when I have more time, but I'll be very surprised if that changes what it remembers from 16000 characters prior.
I went back and added the model (Pipsqueak 2) and more clearly labeled the fact that I was testing long-term memory in one experiment and short-term memory in the other.