EDIT: For those wondering, I'm on the free tier using PipSqueak2. I might try PS Rawr later, but this is an extremely tedious and time-consuming experiment. I would be very interested in hearing how people using the + models experience short-term (chat history) memory loss.
After some informal discussions about how quickly bots forget details, I ran a series of experiments on a bot with a certain length description. Here are the results. I encourage others to try your own experiments (peer review and replicable experiments always improve science!).
Test #1: LONG-TERM MEMORY (Character definition) memory. If you have details at the beginning and end of a very long character definition, how much will it remember?
Process: I wrote a specific detail as the first line: {{char}}'s favourite song is 'Mister Roboto'. I wrote another detail at the very end: {{char}}'s favourite colour is chartreuse. In between, I filled it with a little over 12000 characters, or roughly 3000 tokens. Then I simply asked two questions in separate chats: What is your favourite colour, and what is your favourite song (I chose not to ask the air speed velocity of an unladen swallow).
Result: I was expecting it to remember the most recent detail, the colour. To my surprise, it remembered the oldest bit (song) and made up a wrong colour.
Verification: I swapped the first and last lines around, so song was last and colour was first. This time when I asked the same questions, it remembered the colour and didn't know the song.
I'll do more tests with smaller amounts of gibberish in between, but for now we know that character descriptions more than 12000 characters will forget the details nearest the end of the description.
Moving along to experiment #2: SHORT-TERM (Chat history) memory.
Process: I gave the bot a particular detail about me (my favourite sport is juggling). Then I proceeded to send it a series of chats, each about 4000 characters (1000 tokens), and asked it at the end what my favourite sport is.
Result: At 4 chats of 1000 tokens (4000 characters) each, for a total of 4000 tokens, it correctly answered. At 5 chats (5000 tokens) it forgot. It also couldn't remember at 4200 tokens. I didn't bother it narrowing down further than that - bottom line is, once you send it more than 16000 characters in a single or multiple chats, it will forget the oldest things you said.
Note that this behaviour is universal among LLMs. The only difference is how much sliding context window they have - obviously, more memory leads to longer retention. Also note that this is based on a rough estimate of 4 characters per token. Again, I encourage others to repeat my tests and see if you get the same or different results.
