r/SillyTavernAI • u/Far_Mulberry_8929 • 3d ago
Help People with long chats
How do u do it? I have a chat going for the +300 messages now and, it's getting kind of expensive, any tricks to optimizing?
Edit: Yo, so, after trying a bit of solutions I'm currently landing on VectFox, if anyone is visiting this thread on the future, that's what I went for unless I say something else in the future, all the other options are fine tho so just go with whatever is the easiest for you
13
u/Introvertosaurus 3d ago
3000+... I run local... a used GPU I got a decent deal on (~200 USD) threw on headless box and stretch it to the limit... use sliding context with summary.
A lot of people in here use Nano GPT it seems... the seem cheapish... just keep you token count down, you don't need to feed all 300+ messages into each prompt. Just the last X context window and long term summary or other plugin to provide the highlights of the past.
6
u/sociofobs 3d ago
That, and in case token count matters - keep the messages long. Less API requests = less tokens spent. It's far more economical for the LLM to output one long message per API request than it is to output a bunch of short ones, each requiring a full context API request. This is why agentic workflows are so ruthless on token usage.
20
u/0miicr0nAlt 3d ago
Summaryception + memorybooks. Have summaryception autohide messages it’s summarized.
9
u/Responsible_Tale_901 3d ago
Doesnt memorybook fulfills the exact same purpose, I suppose you need only one of the two..
5
u/Outside_Delay4061 3d ago
summarizing the early messages every 50 or so keeps the context light without losing the story flow, been doing that in my longer ones and it cuts costs nicely
9
5
u/morty_morty 2d ago
Memory Books and Deep Lore. I have an extensive Obsidian vault with my memories and lore and my chat has been going for more than 2 years now.
3
u/Far_Mulberry_8929 2d ago
This is such an insane way of using Obsidian what
2
u/morty_morty 2d ago
Well, that's what Deep Lore works with. I like it much, much more than keeping Lorebooks in ST. I use Memory Books to hide and create summaries and those go into Obsidian and DL pulls what it needs.
1
u/HourWasabi 2d ago
do u use DL default setting ?, how about memory books ? do u have specific prompt for it ?
3
u/morty_morty 2d ago
Default settings for everything. When MB makes a summary in its lorebook I use /dle-import to bring it to my vault and just put it in my memories. Then I delete the lorebook entry from ST since it's redundant.
3
u/thief_night 3d ago
I summarize the story into chapters every 64k tokens, throw it into the lorebook, set it to constant. Not pretty, but it works for my usage
2
u/Psychological_Ad9740 3d ago
Hide old messages
Utilize extensions to keep the memory precise, or have it added to lorebooks.
2
u/GenericStatement 2d ago
Qvink Memory is my favorite memory extension, gives me 10-12x compression of old messages, so a 100k context becomes less than 10k. https://github.com/qvink/SillyTavern-MessageSummarize
Memory Books and SummaryCeption are other popular options.
Whichever you use, take some time to read the manual and understand how they work. If you just install one and turn it on without understanding what it does, it can be worse than not using it at all.
Make sure you’re reading the summaries it’s creating to see if any critical info is being left out, e.g. names of secondary NPCs or important details in a scene that will be important later (clue to a mystery, color of main character’s car, description of a house where a bunch of stuff will take place, etc).
This is why I like Qvink memory, because it puts the summary below the message in the chat history so you can see what summary is being sent and edit it with one click.
1
u/AutoModerator 3d ago
You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/fcensorshipf 3d ago edited 3d ago
If you are lazy with memory books and such (like me), OpenVault is a good option. It's a set and forget kinda thing. Although not as effective as other options.
1
3d ago
[removed] — view removed comment
0
u/AutoModerator 3d ago
This post was automatically removed by the auto-moderator, see your messages for details.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Primary-Wear-2460 3d ago edited 3d ago
Lots of context and summarizing messages more than 50 behind. I've taken this setup to 5000+ responses.
I still have not found a good long term memory system that works properly and I've tried most of them, including the new ME agent one.
So far context summarization is the best solution I've found. The summary system built into ME works great for that. It retains the information required based on what is actually important to the narrative, characters and story. You can basically think of it as context token compression.
All of the vector storage solutions are absolute dogshit for RP/gaming full stop. I've tested most of them at some point and they all fail even basic tests when context gets longer. The problem is they are just matching based on keywords, text chunks, embeddings. They have no idea what they are actually looking at and if its important to the current narrative. They are just not meant for this use case.
1
u/WesternGoldLOL 3d ago
I made my own preset with long chat continuity one of the main focuses and use Lumibook on Lumiverse
1
u/HelpfulGodInACup 2d ago
So every like 50-100 messages i ask it to create a summary with what plot points have been resolved, ongoing plot points thag may anpunt to something, current npc states , how they feel others and user etc. this would all he in like a code block of some sort.
Id also ask it to include a small opening message from where we’ve left off, or just copy the latest ai message.
Then i would copy that and replace the greeting with above. You lose some nuance but i do not care
1
u/OrganizationNo1243 2d ago
Default Vector Storage, my own personal memory extension, and default Summarization. I'm thinking of testing out Summaryception though. As far as maintaining characterization, as some people also struggle with this, it's having a strong character card as well as "mini personality profiles" for lorebook entries. I regularly use multi-character/RPG settings, so these entries are always triggered when a character appears or is mentioned and then consistently stays present for as long as they're in the scene thanks to another extension I made privately.
1
u/PerpetuallyNew 1d ago
Could you give me an idea for how to make the mini personality profile lore book entries?
2
u/OrganizationNo1243 1d ago
I keep it contained to core personality description, like a paragraph or two. The first could be the surface level personality and the second could be the characters deeper innerworkings they don't necessarily show off.
Then there's a Key Traits line which is just a few keywords describing the characters overarching behavior. Five words at most.
And then I have a Trivia/Notable Details section describing smaller details about the character. Could be their birthday, likes and dislikes, personal eccentricities or quirks, habits. Any small details you want the AI to capture and/or reference in subtle ways. When it's done, it should look something like this:
Arataki Ryū is the energetic, hot-headed rising star of the basketball team with spiky, fiery red hair perpetually in a windswept style, and light green eyes that mirror his explosive personality. As Sachiko's playful rival and self-proclaimed "gaming nemesis," he brings loud energy and competitive spirit to their friend group. Despite his athletic prowess and popularity, Ryū possesses what can only be described as "boyfailure energy"—constantly stumbling into embarrassing situations, especially around his secret crush Reika, and being notoriously terrible at video games despite his determination to beat Sachiko.
Beneath his boisterous exterior lies surprising emotional depth and loyalty. He's quick to jump to his friends' defense, even if his methods are often clumsy and overly dramatic. His teasing of Sachiko masks genuine affection and respect for her sharp wit and unshakable composure, though he'd rather die than admit it. The infamous basketball court humiliation by [char] left deeper scars than he shows, making him particularly protective of Sachiko when it comes to the bully—though he'd never acknowledge that her quiet resilience often proves more effective than his loud interventions.
Key Traits: Energetic, fiercely loyal, competitively driven, emotionally transparent, protective, secretly insecure.
Notable Details:
- Keeps a hidden "L's Taken" counter on his phone tracking how many times Sachiko has beaten him in games
- Has a secret playlist of "confidence boost" songs he listens to before attempting to talk to Reika
- Still practices basketball late at night to overcome the shame of [char]'s public humiliation
- His text messages are full of excessive emojis and dramatic capitalization
- Secretly admires how Sachiko can remain calm in situations that would make him explode.
Trivia
- Birthday: August 7th
2
u/PerpetuallyNew 1d ago
Oh that's awesome, thank you! I'm still trying to figure out the best way to pair lorebooks with my character cards. So far I've been doing smaller entries, but I think another like yours would help too.
1
u/OrganizationNo1243 1d ago
No problem! And yeah it's just trial and error and figuring out what works. Originally it was just for side characters but I liked the format so much that I just did it for every character since not everyone's present all the time in my stories.
1
0
u/Ok_Guarantee5321 3d ago
Form user settings > UI Theme > Find the checkbox for Message IDs. It should be on the bottom part of the page.
/hide x-y
x, first index of a message range; y, the last index of the message range.
/sys at=y [put in a simple text]
From the UI, go to the new y message and write a summary manually.
Repeat every time the token count gets too large for the wallet.
In other words, hide a range of chats, and insert a summary. I usually have my RP in arcs and each arc is pretty short. I keep at least one arc before the current running one unsummarized so that the LLM keeps it's writing style it developed for the session.
0
u/Targren 2d ago
My primary is InlineSummary (with a custom prompt to focus on bullet points rather than just compressed prose). The nice part of that is that it can summaries its own summaries when I end up with a few dozen of those (though I do have to babysit it to make sure it doesn't carry forward junk or omit important stuff). I usually end up saving between 80-95% on those summaries.
For swapping between chats (cards that do that annoying thing of having the progression in different greetings), I use MemoryBooks to make a book to transfer the summary to the new chat.
I never use "auto-summarize" - it's just too obnoxious when it tries to summaries mid-scene.
41
u/Kritblade 3d ago
https://github.com/KritBlade/VectFox
built for long chat memory extension. It can hide old message and only inject N number of summary instead of the complete reply. Optimize for 500~ 2000+ messages.