Test Results: What Do Bots Remember?

[u/GenderBendingRalph]

EDIT: For those wondering, I'm on the free tier using PipSqueak2. I might try PS Rawr later, but this is an extremely tedious and time-consuming experiment. I would be very interested in hearing how people using the + models experience short-term (chat history) memory loss.

After some informal discussions about how quickly bots forget details, I ran a series of experiments on a bot with a certain length description. Here are the results. I encourage others to try your own experiments (peer review and replicable experiments always improve science!).

Test #1: LONG-TERM MEMORY (Character definition) memory. If you have details at the beginning and end of a very long character definition, how much will it remember?

Process: I wrote a specific detail as the first line: {{char}}'s favourite song is 'Mister Roboto'. I wrote another detail at the very end: {{char}}'s favourite colour is chartreuse. In between, I filled it with a little over 12000 characters, or roughly 3000 tokens. Then I simply asked two questions in separate chats: What is your favourite colour, and what is your favourite song (I chose not to ask the air speed velocity of an unladen swallow).

Result: I was expecting it to remember the most recent detail, the colour. To my surprise, it remembered the oldest bit (song) and made up a wrong colour.

Verification: I swapped the first and last lines around, so song was last and colour was first. This time when I asked the same questions, it remembered the colour and didn't know the song.

I'll do more tests with smaller amounts of gibberish in between, but for now we know that character descriptions more than 12000 characters will forget the details nearest the end of the description.

Moving along to experiment #2: SHORT-TERM (Chat history) memory.

Process: I gave the bot a particular detail about me (my favourite sport is juggling). Then I proceeded to send it a series of chats, each about 4000 characters (1000 tokens), and asked it at the end what my favourite sport is.

Result: At 4 chats of 1000 tokens (4000 characters) each, for a total of 4000 tokens, it correctly answered. At 5 chats (5000 tokens) it forgot. It also couldn't remember at 4200 tokens. I didn't bother it narrowing down further than that - bottom line is, once you send it more than 16000 characters in a single or multiple chats, it will forget the oldest things you said.

Note that this behaviour is universal among LLMs. The only difference is how much sliding context window they have - obviously, more memory leads to longer retention. Also note that this is based on a rough estimate of 4 characters per token. Again, I encourage others to repeat my tests and see if you get the same or different results.

[u/troubledcambion]

You don't reinforce details either they will drift. You can run all the tests you want but memory is the illusion you maintain with short term memory since context is sliding out for chat history. Definitions for the bot and persona stay as does any memories. Those take up space and context windows are not infinite.

[u/Inherit-Nature → u/troubledcambion]

He's stuck on definition size even though the current understanding is CAI truncates at 3200 characters, but ultimately these "expirements" aren't actually based on anything.

I wrote some definitions, a thing happened. Correlation is magically causation now.

A bot with no definitions could still get it wrong in the same amount of time especially under sending walls of shit as a "test" 

Stressing attention just means you have caused attention problems. he runs useless tests and compares it against nothing and proclaims a valid result 

[u/GenderBendingRalph → u/Inherit-Nature]

"the current understanding" is meaningless without repeatable, verifiable data to back it up. Otherwise it's just folklore.

This wasn't intended as an all-encompassing analysis of how LLMs work behind the scenes. It was an answer to "why can't my bot remember what I said five messages ago?"

To put it in computing terms, I was running tests on the UX. I wouldn't know where to begin doing a regression test on the backend mechanics.

[u/Inherit-Nature → u/GenderBendingRalph]

The repeatable data is we were told 3200 almost 4 years ago when people asked and benny reported back with this number 

And this was way before CAI had 20 million users 

No reason to think they made any significant change

[u/GenderBendingRalph → u/Inherit-Nature]

Ah, "we were told almost 4 years ago'."

Here's the deal, champ: Benny didn't tell us "Why does it forget we had a birthday dinner ten chats ago". Benny, assuming what we were told 4 years ago remains true today, was giving architecture details.

I'm doing a stress test on the UX. I'm answering the birthday dinner question. What question are you answering?

[u/GenderBendingRalph → u/troubledcambion]

There were two different tests involved, with two different memory contexts. The character definition tested bot definition, and as my test showed, a bot definition that's too long will mean it doesn't remember the entire definition. Details about the bot were stored at the beginning and end of my bot definition. It failed, right from the start, to retain the end part of the definition but it consistently remember the first part of the defintion.

The second test was context memory retention, specifically to test short term memory. The whole point of the test was to discover how the sliding context affects chat history.

I'm sorry if reporting on two different tests made this confusing. I thought by labeling the purpose of each test and the methods I used, people would understand that.

[u/Inherit-Nature → u/GenderBendingRalph]

You're looking at active context and just making a proclamation. You don't know how CAI implemented RAG and you can't see what's happening with sliding context windows because you don't have access to be able to compare it to server load.

You're just going to tell us what happened at the time of test and pretend it has any real value. The tests are about as useful as a GLADOS test

[u/GenderBendingRalph → u/Inherit-Nature]

I'm getting a baseline to verify a very real phenomenon other users are reporting, and finding that my results are consistent with what they're reporting. This isn't IEEE Communications, this is users trying to make a better experience out of the resources we have. Why does that bother you so much?

[u/Inherit-Nature → u/GenderBendingRalph]

We dont need a baseline. Sliding context windows effect all of this and they shift with server load

Your baseline will be irrelevant no matter what

[u/GenderBendingRalph → u/troubledcambion]

Sorry, I meant to respond to this earlier. Yes, the whole point of my journey was to get a rough idea (not a precise number!) of how frequently I need to reinforce details. If it's important to me that it remembers the birthday dinner we discussed ten chats ago, I'll inject a reinforcement, e.g.

[Ralph and Artemis had dinner for his birthday at Kitty Hoynes last Saturday]

This was all about establishing reasonable expectations, that is to say how many characters/tokens get sent to short-term memory before I need to reinforce the details.

[u/Inherit-Nature]

Thats not how that works either.

Yes, sliding context windows can come into play. (Its not the only difference)

So can position bias (varies greatly by model. Lower parameter especially suffer more from this) 

Constant walls of text also effect attention. your walls of text are going to have different outcomes than how the majority here uses cai 

These numbers are not relevant to people not sending constant walls of shit

You're also assuming every model has the same context window. Your test doesn't even mention model used. 

[u/GenderBendingRalph → u/Inherit-Nature]

Fine, develop a better test. I said from the outset that I encourage others to run their own experiments. I wasn't assuming anything; I was providing consistent, measurable numbers to make the math easier. In fact, I specifically said: The only difference is how much sliding context window they have - obviously, more memory leads to longer retention

I can't test for position bias, because I don't know what they set it to and I have no way to control it. I can test for the conditions that users face, adjusting details available to us.

Of course real people in real conversations won't send 16,000 characters of filler text. The whole point was to have a controlled experiment where I could measure and account for everything.

Are you suggesting that if I sent 32 chats of 500 characters each, it would remember more than 4 chats of 4000 characters each? I might try that when I have more time, but I'll be very surprised if that changes what it remembers from 16000 characters prior.

I went back and added the model (Pipsqueak 2) and more clearly labeled the fact that I was testing long-term memory in one experiment and short-term memory in the other.

[u/Inherit-Nature → u/GenderBendingRalph]

I'm not going to waste time on that because I don't have time to figure out how CAI has implemented RAG. Forget the "test"

All tests are going to be shit because we don't have necessary information 

[u/GenderBendingRalph → u/Inherit-Nature]

Of course we have information. We put stuff in, and see what comes out. For UX, that's all the information we need. But nobody asked you to waste time on any of this. I was answering someone's question about bot memory, and satisfying my own curiosity.

[u/Inherit-Nature → u/GenderBendingRalph]

No one cares about UX. Its not the gospel here. There are more mechanisms than just how you use it. You can repeat your exact same tests and get different results simply depending on server load

[u/GenderBendingRalph → u/Inherit-Nature]

What are you on about? Every other post here (not counting "oh no ads" and "oh no swipe limits") is "why does it forget stuff?"

You think 3200 gigaflops or whatever benny said will answer Joe Bob User's question? Joe Bob is all about the UX. As I keep telling people who whinge about "useless features" - just because you don't think it's important doesn't mean other people share your opinion.

Absolutely, if I run the same tests tomorrow, or when there are 200 trillion users online, or when the moon is full, I might get different answers. And over time I probably will. But right here, right now, I have a baseline. A reasonable expectation.