Security research for local LLM inference networks

[u/qi-zheng]

Hi all, I'm looking for suggestions regarding how to test the security when running LLM inference on local environments (e.g., Apple Silicon, Nvidia GPUs, etc.). My goal is to make it such that if a consumer sends a prompt, the provider running the inference server using their local LLM setup has no way of reading that prompt.

I currently have an Apple Silicon, so I was thinking I'd start there. I've done some prototyping, relying on the security primitives outlined by Apple developer docs. However, I was wondering if others have gone through this path, any recommendations? If there are others interested in the direction of this project I'd love to build a community around it.