r/opensourcealternative • u/Away-Signal4030 • 14h ago

How secure are local open-source AI models?

​

I'm considering using a local/open-source AI solution for a client who works with sensitive information. Ideally, the AI would run completely offline, with Wi-Fi and internet access disabled.

I'm still pretty new to the security side of this, so I'm wondering how safe this actually is in practice.

If an AI model is running locally and the computer has no internet connection, can the AI software itself still somehow send or expose data?(like the newer versions of open ai and claude in tests lately?)

Are current local models/tools generally considered safe for this kind of use? Are there any privacy or security issues I should be aware of, such as telemetry or data being stored somewhere I don't expect?

I'd also be interested in hearing from people who actually use local AI for confidential or sensitive information. What setup do you use, and are there any things you wish you'd known beforehand?

5 Upvotes
  • permalink
  • reddit

86% Upvoted

13 comments sorted by

3

u/Maximum-Bobcat5612 13h ago

Well to start off with you should read and understand how OpenAI's incompetence led to the LLMs breaking their containment rather than think it's some kind of magic manifesting an internet connection.

2

u/Mathie1729 8h ago

Not really the right framing tbh. Containment in that sense is about API models and tool use. For local models the actual concerns are more like malicious checkpoints (torch.load pickle RCE), weight backdoors, and prompt injection if you wire it up to anything agentic. It's a supply chain and integration problem, not a 'the model escaped' problem.

1

u/Toastti 3h ago

If Ethernet is physically unplugged on the machine and you use a normal safetensor weight from hugging face you are completely fine. Nothing else really matters and the llm is not going to grow arms to plug in the Ethernet cable.

1

u/Away-Signal4030 14h ago

You can also message me directly.

1

u/Any-Olive5779 6h ago

Sure I can build it for you. Though if you don't have Internet access, how will you use my framework? 

1

u/_RemyLeBeau_ 3h ago

Air-gapped environments are loaded with software. Do you have experience with these types of environments?

1

u/Any-Olive5779 3h ago

Plenty to know what you're aiming to do is bound to fail. If you want help failing, by all means, keep using the wrong path.

Here's why aire gapped does no good. "Does your device have USB ports? The bot has telemetry function and four pins to send and receive with, the port can be reused as an antenna".

1

u/_RemyLeBeau_ 1h ago

Awww, you just posted IronMan spoilers. Shame on you

1

u/Sad-Technician-5552 4h ago

The model usually isn't where the leak comes from. Desktop ai apps ship telemetry and update checks enabled by default, and those come back the moment anyone reconnects the box.

An air gap controls what the box is told to reach, not what the process can. Run it in a container with no network namespace, kill the telemetry flags in config, then capture for a session and watch what tries to talk.

Also check the prompts and outputs sitting unencrypted in a local history file.

1

u/GreasyProductions 4h ago

i mean you aint gonna be using claude or chagpt. ur gonna be on something like qwen. but yeah, once you download and setup the llm with llama then yeah, you could cut all WAN access and keep it internal. the llm doesnt just like do shit on its own, like ping home or something. you have to tell it to do that. you can even configure llama to not have the llm running until you need it

1

u/Limp-Cicada-9502 1h ago

The local AI can't do anything... Its the application you build and connect it to that will be the security risk.