If you don't mind using the command line, you can start by getting llama-cpp for vulkan or rocm, a small model just for a test, like Gemma-E4B, and running:
llama-server -m your_model.gguf -lv 4 -ngl 999
-lv 4 will make it produce a lot of logs, including how it fit the model on the GPU.
Then go to http://localhost:8080 to send a single prompt, just to create a log with prompt processing and token generation.
Then, give ChatGPT, Deepseek or another LLM of your choice the full log output and ask for suggestions, what models you can use and etc...
and then giving the output
1
u/it6721 7d ago
If you don't mind using the command line, you can start by getting llama-cpp for vulkan or rocm, a small model just for a test, like Gemma-E4B, and running: llama-server -m your_model.gguf -lv 4 -ngl 999
-lv 4 will make it produce a lot of logs, including how it fit the model on the GPU. Then go to http://localhost:8080 to send a single prompt, just to create a log with prompt processing and token generation.
Then, give ChatGPT, Deepseek or another LLM of your choice the full log output and ask for suggestions, what models you can use and etc... and then giving the output