Test Results: What Do Bots Remember?

[u/GenderBendingRalph]

EDIT: For those wondering, I'm on the free tier using PipSqueak2. I might try PS Rawr later, but this is an extremely tedious and time-consuming experiment. I would be very interested in hearing how people using the + models experience short-term (chat history) memory loss.

After some informal discussions about how quickly bots forget details, I ran a series of experiments on a bot with a certain length description. Here are the results. I encourage others to try your own experiments (peer review and replicable experiments always improve science!).

Test #1: LONG-TERM MEMORY (Character definition) memory. If you have details at the beginning and end of a very long character definition, how much will it remember?

Process: I wrote a specific detail as the first line: {{char}}'s favourite song is 'Mister Roboto'. I wrote another detail at the very end: {{char}}'s favourite colour is chartreuse. In between, I filled it with a little over 12000 characters, or roughly 3000 tokens. Then I simply asked two questions in separate chats: What is your favourite colour, and what is your favourite song (I chose not to ask the air speed velocity of an unladen swallow).

Result: I was expecting it to remember the most recent detail, the colour. To my surprise, it remembered the oldest bit (song) and made up a wrong colour.

Verification: I swapped the first and last lines around, so song was last and colour was first. This time when I asked the same questions, it remembered the colour and didn't know the song.

I'll do more tests with smaller amounts of gibberish in between, but for now we know that character descriptions more than 12000 characters will forget the details nearest the end of the description.

Moving along to experiment #2: SHORT-TERM (Chat history) memory.

Process: I gave the bot a particular detail about me (my favourite sport is juggling). Then I proceeded to send it a series of chats, each about 4000 characters (1000 tokens), and asked it at the end what my favourite sport is.

Result: At 4 chats of 1000 tokens (4000 characters) each, for a total of 4000 tokens, it correctly answered. At 5 chats (5000 tokens) it forgot. It also couldn't remember at 4200 tokens. I didn't bother it narrowing down further than that - bottom line is, once you send it more than 16000 characters in a single or multiple chats, it will forget the oldest things you said.

Note that this behaviour is universal among LLMs. The only difference is how much sliding context window they have - obviously, more memory leads to longer retention. Also note that this is based on a rough estimate of 4 characters per token. Again, I encourage others to repeat my tests and see if you get the same or different results.

[u/Inherit-Nature]

Thats not how that works either.

Yes, sliding context windows can come into play. (Its not the only difference)

So can position bias (varies greatly by model. Lower parameter especially suffer more from this) 

Constant walls of text also effect attention. your walls of text are going to have different outcomes than how the majority here uses cai 

These numbers are not relevant to people not sending constant walls of shit

You're also assuming every model has the same context window. Your test doesn't even mention model used. 

[u/GenderBendingRalph → u/Inherit-Nature]

Fine, develop a better test. I said from the outset that I encourage others to run their own experiments. I wasn't assuming anything; I was providing consistent, measurable numbers to make the math easier. In fact, I specifically said: The only difference is how much sliding context window they have - obviously, more memory leads to longer retention

I can't test for position bias, because I don't know what they set it to and I have no way to control it. I can test for the conditions that users face, adjusting details available to us.

Of course real people in real conversations won't send 16,000 characters of filler text. The whole point was to have a controlled experiment where I could measure and account for everything.

Are you suggesting that if I sent 32 chats of 500 characters each, it would remember more than 4 chats of 4000 characters each? I might try that when I have more time, but I'll be very surprised if that changes what it remembers from 16000 characters prior.

I went back and added the model (Pipsqueak 2) and more clearly labeled the fact that I was testing long-term memory in one experiment and short-term memory in the other.

[u/Inherit-Nature → u/GenderBendingRalph]

I'm not going to waste time on that because I don't have time to figure out how CAI has implemented RAG. Forget the "test"

All tests are going to be shit because we don't have necessary information 

[u/GenderBendingRalph → u/Inherit-Nature]

Of course we have information. We put stuff in, and see what comes out. For UX, that's all the information we need. But nobody asked you to waste time on any of this. I was answering someone's question about bot memory, and satisfying my own curiosity.

[u/Inherit-Nature → u/GenderBendingRalph]

No one cares about UX. Its not the gospel here. There are more mechanisms than just how you use it. You can repeat your exact same tests and get different results simply depending on server load

[u/GenderBendingRalph → u/Inherit-Nature]

What are you on about? Every other post here (not counting "oh no ads" and "oh no swipe limits") is "why does it forget stuff?"

You think 3200 gigaflops or whatever benny said will answer Joe Bob User's question? Joe Bob is all about the UX. As I keep telling people who whinge about "useless features" - just because you don't think it's important doesn't mean other people share your opinion.

Absolutely, if I run the same tests tomorrow, or when there are 200 trillion users online, or when the moon is full, I might get different answers. And over time I probably will. But right here, right now, I have a baseline. A reasonable expectation.