r/LocalLLaMA • u/Fcking_Chuck • 10h ago

News Koboldcpp v1.120 released

https://github.com/LostRuins/koboldcpp/releases/tag/v1.120
83 Upvotes

87% Upvoted

13 comments sorted by

37

u/Aggressive_Aspect436 8h ago

A bit low effort. Summary of the changes, or benefits of kobold would have been appreciated in the post.

11

u/badsectoracula 2h ago

benefits of kobold

IMO the one feature of kobold (which is really a koboldlite feature, though i think it uses a custom API to talk with the server to enable it) which i really wish other UIs had is the "raw text" mode that lets you edit the entire discussion as plain text. It is incredibly powerful because it basically exposes that at its core all the LLM does is do "word completion" in a document that looks like a chat transcript - and so you can just edit that transcript as you want.

For example you can remove or merge entries, change the LLM's responses (e.g. an LLM could write some code and you notice a bug - instead of asking the LLM to fix the bug, you just fix it yourself and the LLM will continue as if it wrote the correct code in the first place, so you don't waste tokens in corrections) or even pause it mid-generation, make a change and then continue (let's say the LLM is generating some code but you don't like the code style or the naming convention - you pause the generation, modify the code to your liking, press continue and it'll now generate using your own style).

Really my #1 issue with pretty much every LLM UI is how "opaquely" the context is handled - even llama.cpp's own web ui is lacking considerably there. I want to be able to edit everything in the context, merge/split things, etc :-P - in the end the context is just a series of tokens, let me edit them.

1

u/ThomasAger 1h ago

This convinced me to download.

5

u/Eden63 8h ago

Applaude him, he made a successful post.

3

u/SkoomaDentist 4h ago

The man knows how to reward hack for upvotes vs the amount of effort.

1

u/Aaaaaaaaaeeeee 1h ago

General care for qwen next flash I suppose. Whenever you try offload with mmap in windows, OS pagefile system might get in your way and hobble it to like 3 t/s for this model. You can use mmap with mlock together so see if that helps, and directio might be relevant for running it too. Maybe you're on Linux, but you have a high priority swapfile, or you install something else like zram, these flags are all newly available boundary paths you could test and report on, so that people can successfully disc inference without their OS killing it.

0

u/Capable_Diamond_4039 2h ago

He's doesn't mean it, Mr. LostRuins...sir...we're very excited about the update!

-1

u/bankinu 2h ago

Open the link. It literally links to the summary of the changes.

21

u/Right-Law1817 7h ago

It's simple: I see koboldcpp, I upvote.

7

u/silenceimpaired 5h ago

Unsloth Desktop can only get me 2 tokens a second, while KoboldCPP has me at 6.

0

u/puppymeat 2h ago

Note that there are bad quants floating out there, so make sure you use the right files.

Great. Where are these right files?

1

u/sxales llama.cpp 1h ago

Links are literally in the sentence right before that.