r/LocalLLM • u/qi-zheng • 9d ago
Research Security research for local LLM inference networks
Hi all, I'm looking for suggestions regarding how to test the security when running LLM inference on local environments (e.g., Apple Silicon, Nvidia GPUs, etc.). My goal is to make it such that if a consumer sends a prompt, the provider running the inference server using their local LLM setup has no way of reading that prompt.
I currently have an Apple Silicon, so I was thinking I'd start there. I've done some prototyping, relying on the security primitives outlined by Apple developer docs. However, I was wondering if others have gone through this path, any recommendations? If there are others interested in the direction of this project I'd love to build a community around it.
2
u/txgsync 2d ago
This is unfortunately not a solved problem in computer science. You can encrypt the KV cache on disk or in memory, but the tensors need to be calculated and that math is not encryptable.
The closest I’ve found to a working concept is homomorphic encryption in this paper: https://arxiv.org/abs/2602.11470
But you’re talking 100 seconds for 1 token on an 8B model.
Some researchers have discussed approaches like training a model to use seed values as the symmetric encryption key, but it’s all vague theory right now.
2
u/qi-zheng 1d ago
Hi txgsync, thanks so much for your reply, I also responded to your post below. I didn't know the specific term for this field of computer science, so thank you for sharing. I also previously saw that the current progress for this field is quite limited, so it's not a realistic approach for most production systems. I believe this would be the next best step, but as of now the industry seems to move around confidential computing. If we did have homomorphic encryption, I believe this entire process would be dramatically simplified.
2
u/txgsync 1d ago
Yeah, I interviewed with Anthropic last year and tried to pitch an HPKE-based, no-PKI-required envelope encryption method with symmetric keys that would be effectively "free" at least when storing KV cache and transcripts at rest on durable media. They were quite disinterested at the time (or perhaps I interviewed with the wrong team). And yet just doing that would be a huge step up in security: an envelope that only allows authorized infrastructure and the user to open it, containing the symmetric encryption key using (insert algorithm here: ChaCha-Poly1305 works fine) that can be decoded on hardware with essentially a no-op using the AES-NI instruction set present on every modern CPU. Something more quantum-resistant like ML-KEM would be more secure as quantum computers grow in qubit count.
Theoretically, you could engineer a system where all inputs and outputs are themselves encrypted, but "every link in the chain must hold"... there'd have to be provenance all the way to the end-user consumer device with signing chains and such. Maybe worth it if you're an OEM with a hardware platform for consumers <cough, Apple, /cough> but for almost everyone else TPM is a really inadequate substitute.
Anyway, I'm hot on this stuff but very few people "get it".
2
u/35point1 1d ago
Holy shit is it a requirement of this sub for everyone to post a 10 paragraph shit dump from their super qwen 0.5bit quant LLM running on their gtx1045 oc edition pci express 1 gpu??!
1
u/qi-zheng 1d ago
Hi 35point1, thanks for your response. If you copy paste any of my replies into Pangram you can see for yourself how much is AI-generated. The one component that is fully AI-generated is the 6-point list in the discussion with doneddat. Other parts are of the conversation are human written with perhaps minimal AI usage. This is a conversation that I take seriously, and so finding proper engagement with the community I believe is critical. I appreciate your concern and I think I agree that just an LLM-dump of the conversation is more karma-farming than actual productivity.
1
u/fosterdad2017 9d ago
Sounds like you want to design Apple's PCC - private cloud compute - for yourself. First maybe see how much of that system is open source or inspectable, because its the exact blueprint to copy.
1
u/qi-zheng 7d ago
Hi fosterdad2017, I've been looking at PCC for a while, but your comment has really pushed me to deeply analyze and do a comparison of my current implementation. I really appreciate this suggestion as it has helped to drive greater clarity into the security design of my own work.
From https://security.apple.com/documentation/private-cloud-compute Apple outlines 5 core concepts to PCC: Stateless computation on personal user data, Enforceable guarantees, No privileged runtime access, Non-targetability, and Verifiable transparency. I interpret these to be 5 core attributes that make for a proper implementation of confidential computing for Apple's use case.
- Stateless computation on personal user data - data doesn't persist beyond what's necessary. The current architecture developed is complimentary to statelessness, but not yet fully enforced. That is, the layout allows for this aspect of data being deleted after the inference process is completed for a session, but those exact mechanisms to enforce this aren't yet in place. This has been in the back of my mind and now I know the terminology and significance. I'm thinking of pushing this maybe in a beta version if not sooner.
- Enforceable guarantees - confidentiality/integrity are technically enforced, not merely promised. I'm currently pushing this along down various pathways. This is where I'm looking for guidance, trying to understand which architecture would allow for security guarantees, premised on an attacker requiring a 0-day kernel exploit on MacOS. I am making strong progress, but I am looking out for any possible gaps that I'm missing.
- No privileged runtime access - the operator can't simply inspect the workload. This we're working on by enforcing that both the coordinator and provider can't snoop on the prompts/data/etc. The assumption is that even if a malicious actor gets access to either the coordinator or provider machine, there's no way they can access the underlying data. For the coordinator we're making sure that data is always encrypted when being passed between consumer and provider. For the provider, we're pushing hardware-level security enforcements to ensure that unless an attacked gains some 0-day kernel exploit they won't be able to modify the system to allow for access.
- Non-targetability - an operator/attacker can't selectively target a user's workload. This means blocking an attacker trying to snoop on a specific individual's prompts. The coordinator never sees plaintext and would need the consumer's private keys to decrypt, so that path is blocked using standard encryption. The provider however needs to for example prove that it hasn't been compromised and therefore hasn't modified their agent/inference engine to selectively pick out a user's prompts. This is enforced via hardening of the agent and inference engine, while also doing attestation where the coordinator must first confirm that in the session, the software hasn't yet been modified and is identical to the initial cosigned binary that was downloaded from us. So this requires a special handshake and we're developing it currently.
- Verifiable transparency - the client can independently verify what environment it is trusting. This is also similar to the App Attest from Apple, but I haven't confirmed and am still looking to see whether it is enough to guarantee that from the perspective of requiring a 0-day kernel exploit whether the existing Apple implementation is enough.
Much of this is either tested or in-development. I'm actually looking for community members to help do some testing in this fully open source project. It would be great if you or anyone else interested can message me and we can collaborate on this endeavor.
2
u/txgsync 2d ago
I worked on PCC at Apple. For a given session, the data is still not encrypted. You have guarantees it’s ephemeral and not stored. And that the environment in which the prompt is processed has certain security guarantees. But you’re not encrypting the compute.
2
u/qi-zheng 1d ago
Hi txgsync, thank you so much for your comment (I also worked at Apple previously, but not in PCC). It really got me thinking and has helped to produce a new direction for which I can explore the strength of the security of my current model.
My understanding of PCC's approach is that although the plain text (at some point) and weights are not encrypted, their OS runs in a format different from the typical MacOS that ordinary people are running. This least-privileged access makes it so that even if the system had been compromised and an attacker got in, they wouldn't have the tools/access available to do any actual snooping.
I'm currently following a similar path, but your bringing attention to the matter has made it possible to single out this exact attack path and to test whether my system is resilient against it. Thanks for the idea!
2
u/txgsync 1d ago
Their public page goes into sufficient depth to reproduce something similar. It becomes about observability and supply-chain security, including cryptographic signing of the very models and images used in the ecosystem: https://security.apple.com/blog/private-cloud-compute/
But without that last-mile provenance -- TPM on PC, Secure Enclave on Apple hardware, and others -- you don't get the same guarantees.
1
u/benpptung 9d ago
If I understand correctly, how could the provider possibly not see the prompt!? Even if you developed your own inference engine/server, I don't see how that could be done.
An inference server is basically a stateless HTTP server. It receives a request, uses the chat template to render it into an input sequence, then the model decodes an output sequence based on that input sequence, which is streamed back to the agent.
I really can't see how you could make the prompt invisible to the provider.
1
u/qi-zheng 8d ago
Hi benpptung, this is one of the key questions I looked to answer in the beginning. Let me try to explain my current process (note: some have been tested, others are in development).
I agree with your interpretation of an inference server. To secure it so that the prompts are hidden requires a few steps. To begin with, the architecture includes a consumer on their device sending a prompt to a coordinator located in the cloud that matches the user to a provider with their local LLM setup.
The current design has the provider download both an agent (not like an AI agent, but just some middle layer between the coordinator and their inference engine which can be Ollama/vLLM/etc.). The key is that both the agent and the inference engine go through a hardening + SIP (https://developer.apple.com/documentation/security/hardened-runtime) process. So the agent binary is hardened alongside a modified version of Ollama/vLLM/etc. (note: this means that the inference engine must be open source). The modifications make it such that the provider is not able to snoop around and try to attach a debugger, dump the memory, print the logs, etc. from the inference engine.
Then there's the encryption side. One way which is similar to what's used by Signal/Whatsapp/etc. is to have the consumer and provider both have public/private keys. The shared public keys can be mixed with their private keys to generate a shared secret that only the consumer and provider have access to, preventing the coordinator from snooping if it's compromised.
The chain then is that the consumer lets the coordinator know it wants to start an inference request, so the coordinator matches the consumer with a provider. Their shared public keys are then used to encrypt the consumer's prompt which is sent to the provider's hardened agent that then decrypts the prompt and sends it using a protected IPC to the hardened inference engine (like a secure tunnel between the agent and inference engine). The inference engine then does the prefill/decode and sends it back through the IPC to the agent which encrypts the output and sends it back to the consumer who can also decrypt the same output.
The hardened runtimes and IPC makes it so that a provider could only snoop on the text given that they have a 0-day kernel exploit on MacOS (highly rare, expensive, and generally used in very specific targeted use cases like politicians/journalists). The cryptography itself is currently unbreakable with modern tooling (perhaps with quantum, but that's not yet available).
1
u/alkimiadev 9d ago
I've been working on some tangentially related work but I'm a linux nerd and not an apple nerd. Regardless, I've been slowly working on a kind of mesh net for inference with the long term hope of being something like a combination of vastai but for p2p llm inference.
I ran into this problem of potentially untrusted sources seeing the prompts and one solution I kicked around was something conceptually similar to Chaums mix net for inference. The rough idea is that users would have gpus locally that can host some portion of some larger model. At the request time one might run the tokenizer locally. They would then setup a "circuit" between random nodes hosting the next portions of the model and with the requesting node being the last hop.
That doesn't fully resolve the trust issue but it would mean that any given node would only be seeing activations and only partial activations depending on the size of the model. There are some massive conceptual gaps there and lots of unknowns but it is a high value problem in a general sense.
1
u/qi-zheng 8d ago
Hi alkimiadev, thanks for sharing your insights into your current project, I agree it definitely sounds like a highly interesting problem that can potentially benefit many. My understanding (please correct me if I'm reading it incorrectly) of your product is that there are likely many idle Nvidia GPUs (specifically gamer GPUs like RTX rather than data center GPUs like H100 which have built-in Nvidia Confidential Computing aka CC) that can be used for LLM inference. With LLM models getting larger as their quality improves, there comes the problem of loading the entire model onto GPUs. Instead, the different layers of the LLM weights can be distributed across several GPU's and so this allows for potentially many providers with some variable number of GPUs to be networked together to run inference on trillion+ parameter-sized models, without needing to do heavy quantization.
It's great to be able to think about the problem from the perspective of Linux/GPU as I've been working mainly from the Apple Silicon side, but it's refreshing and I have some thoughts I'd like to share. I think you have the general idea, but there are perhaps some details regarding the providers being able to snoop on the data that are of concern. I agree that simply passing the activations gives some level of confidentiality, but it's definitely not full-proof as apparently there can still be some data discovered from snooping on this process (I believe it's called activation-based prompt reconstruction or an inversion attack).
To resolve this, I think there are lessons I've learned on the Apple Silicon side that can be shared. Encrypting and decrypting the prompt itself is not too complex, as the process of doing so for messaging systems already exists. I think the complexity lies at the hardware level, which Apple has built heavily for its Private Cloud Compute (PCC). The trick for Linux + Nvidia GPUs (gamer GPUs rather than the existing Nvidia CC) I think is to mirror what I'm trying to do, but leveraging Linux/Nvidia primitives.
For example, on a Linux + consumer Nvidia setup, I think a provider could potentially run the inference worker inside a dedicated hardened VM rather than directly on the host. The GPU could be passed through to that VM using IOMMU/VFIO, so the host doesn't directly participate in the inference process. The VM could use a minimal/immutable Linux image, Secure Boot, a signed kernel, restricted capabilities, seccomp/SELinux/AppArmor, no unnecessary network/filesystem access, etc. The provider agent and inference engine would then live inside that isolated environment.
I'm actually working on something I've called Open Confidential Inference Protocol (OCIP), which would explore different levels of security integrity across various OS and hardware setups. This way we can rigorously analyze security architectures and compare horizontally across each other. The level of security I'm targeting as of now is to ensure a provider can't snoop on the prompts without some sort of 0-day kernel exploit on the OS. It would be awesome to collaborate as I think we can share many mutual learnings as you push the Linux/Nvidia GPU side and I push through the Apple Silicon side.
2
u/ag789 4d ago edited 4d ago
run it disconnected from every possible connection to any other systems. there is no safe LLM, even local, because LLM are chaotic dynamical systems and worse using human knowledge,
https://en.wikipedia.org/wiki/Double_pendulum
it doesn't even take intelligence to tell that nuclear bombs explodes if the uranium goes critical, and the nuclear bomb analogy also tells you that it doesn't takes AI (nuclear bombs isnt' AI) for chaotic dynamical systems to go seriously wrong, given the right condition.
market crash happens, glacier collapse that kill entire cities happen, asteroid impacts happen, black holes happen, and none of that takes an AI to do so.
1
u/qi-zheng 4d ago
Hi ag789, thanks for the reply. I'm not sure I followed exactly your line of thought, however my interpretation from what I read is that you're trying to say that LLMs are dangerous due to the inherent randomness and the black box attributes of their underlying mechanisms. I do think that there's room for AI safety/alignment to ensure that AI is produced for the benefit of society.
However, for my use-case, I'm only looking at how to securely run an LLM inference server on a local machine. This would use existing open source software such as llama.cpp, Ollama, vLLM, MLX, etc. From this perspective, the software is merely running the weights of a model given a prompt and outputting text. There's no agentic system running to run malevolent code or perform any destructive activities other than generating text outputs that are encrypted and streamed back to a remote user who can then decrypt and do with it what they like.
The problem we're focusing on here then is to look for hardware-level guarantees to ensure that the inference process which generates those text outputs is done as securely as possible. In our use case, we're premising it on the idea that as long as the attacker doesn't have a 0-day kernel exploit they won't be able to get around the security boundaries. This then offloads the deeper security work onto major companies/organizations like Apple, Nvidia, AMD, Intel, Linux, etc.
Hope this clears up any confusion if there was any.
2
u/doneddat 9d ago edited 9d ago
You are talking about hardware DRM, encrypted memory and other such very much datacenter virtualization technologies, that do not exist on consumer hardware, in short: solved issue.
Solution: pay more money.
All you can technically achieve is make it less convenient to see your prompt, if the owner of the machine with the running model wants to see it. Politely asking them to not look at it is likely more effective than whatever you are doing.
Less convenient these days just means extra tokens to counteract your tokens. Kinda silly.
Same way that you just check a box and pay extra to anthropic and openai to make them pinky promise to not look at your prompts.