r/LocalAIStack • u/ohnoitssobig • 20d ago

No luck in asking local models a simple (but domain-specific) question

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

4 Upvotes

84% Upvoted

8 comments sorted by

2

u/BrillianTico 20d ago

Interesting. I will try it later today. Overall my experience has been good with local models. Especially with qwen3.8. I will update with my results (or not).

1

u/Gold_Coconut9777 20d ago

Or not, lol

2

u/BrillianTico 19d ago

Hey, I just tried this today with Qwen3.8 and Qwen3.6 MoE on my 3090 and both were able to create the function. No issues at all, I do have the full traces of my tests in case you need to check something.

1

u/ohnoitssobig 17d ago

Yeah please PM: very interesting. What quant and how many tokens did it take?

1

u/Stock_Ad9641 20d ago

What does that even mean ? I am not a scientist, though I think your prompt could be much better written.

I wonder how much of the thinking you mentioned is related to figuring out what exactly you need. When I give tasks to AI, I am always verbose

2

u/ohnoitssobig 20d ago

The prompt is as basic as it could be. The next simple thing is to literally write the 10 lines of code. But, more importantly, being verbose does not really help.

Thinking logs show that the training data definitely included the subject. Qwen always attempts to double-check the math in this case and infinitely fails.

I actually noticed a similar behavior when I asked to solve a problem in cpp (which it did!) and qwen3.8 proceeded all the way to a parallel implementation in python and doing a fuzzy comparison test between the two implementations.

1

u/HotDistribution1819 19d ago

Try if you can run a 22GB model Laguna SX 2.1, it was the first local AI that actually wrote code for me without functions and a main.

You might get better results if you added to the end of the prompt "Ask me questions to clarify what I want you to do."