Private AI for companies: running local LLMs on your own servers (GDPR guide)

Most companies already use AI — the question is whose servers it runs on. Every prompt pasted into a public chatbot sends company data to a third party. For customer records, contracts, financials or unreleased marketing plans, that's a risk many legal and compliance teams are no longer willing to take.

Private AI is the alternative: the same kind of assistant, running inside your own walls. This guide explains what that means in practice — which use cases benefit most, which open models are good enough, what hardware you need, how the costs compare with cloud AI, how to keep answers accurate and secure, and how a local LLM fits with GDPR.

What is private AI?

Private AI is AI that runs entirely within infrastructure you control. The language model, your documents, the search index and the chat interface all live on your own servers or in a dedicated private cloud. Prompts and answers never leave your network, and nobody uses your data to train their models.

In practice, that usually means an open-weight model — one whose weights you can download and run yourself — combined with a search layer over your own data. Your team gets a ChatGPT-like assistant; your IT and data protection team gets full control over where every byte goes.

Why are companies moving away from public AI tools?

The reasons we hear most often are surprisingly consistent:

• Shadow AI. Employees already paste internal data into public chatbots because there's no approved alternative. Banning it rarely works; offering a better internal tool does.
• Data protection and client contracts. GDPR, NDAs and client agreements often forbid sending certain data to third parties — especially outside the EU.
• Vendor dependency. Prices, terms, rate limits and model behaviour can change overnight. Workflows built on one provider break with them.
• Cost at scale. Per-token pricing is cheap for experiments and expensive for company-wide daily use.

Which use cases benefit most from private AI?

Private AI pays off where two things meet: data that shouldn't leave the company, and a task people repeat every day. These are the five use cases we see most often, and what each one actually needs to work:

Internal knowledge search is usually the best first project: the value is easy to see ("where is the current travel policy?"), the risk is low, and it forces you to sort out data sources and permissions — groundwork every later use case needs anyway.

Contract analysis works best as extraction rather than open chat: the model fills a fixed set of fields — parties, term, notice period, liability cap, unusual clauses — and a person checks the result. HR and finance assistants are where private AI matters most, because the data is among the most sensitive a company holds. Support drafting is often a step towards a customer-facing agent; we describe how that looks in production in our article on our live AI support agent, which resolves 80% of requests.

Are open models good enough for business tasks?

For most everyday work, yes. Open-weight model families such as Llama, Mistral, Qwen, Gemma and DeepSeek have closed much of the gap to the big proprietary models. Where they still fall behind is the hardest multi-step reasoning — and that's rarely what an internal assistant spends its day on.

Two practical notes. First, licences differ between model families — check the licence before commercial use. Second, the model is only half of the result: good retrieval over your documents matters more for answer quality than squeezing out a slightly bigger model.

Language matters too. If your documents and users are German, French or Polish, test models in that language — quality in English doesn't automatically carry over, and smaller models in particular vary a lot between languages.

What hardware do you need for a local LLM?

Less than most people expect. The main constraint is GPU memory, which depends on model size and on how many people use the system at the same time. As a rough orientation for quantised models:

Quantisation stores model weights with fewer bits, which cuts memory needs considerably at a typically small cost in quality. Long documents and many simultaneous users also take memory, so a model that "fits" in a test can still run out of room under real load.

For many mid-sized companies, one well-equipped GPU server is enough to start. We size the setup on your real tasks and user numbers before anyone buys hardware — sometimes a private cloud instance is the better first step.

What does a private AI setup look like?

Data sources — documents, wikis, tickets, CRM or marketing data — are connected and indexed. When someone asks a question, the system retrieves the relevant passages and the local model writes an answer based on them, with references. A simple chat interface makes it usable for everyone, and access control ensures people only see answers from documents they're allowed to read. Every request is logged.

Serving tools like Ollama or vLLM run the model; a vector database handles search. None of it requires sending data outside. Where the assistant needs to trigger actions — creating a ticket, updating a CRM record — a self-hosted automation tool can do that inside your network as well; we compare the options in n8n vs. Make vs. Zapier.

How do you keep a private AI's answers accurate?

Most wrong answers aren't caused by the model. They're caused by the search step handing it the wrong passages — or none. Accuracy comes from three things:

1. Retrieval quality

Documents are split into meaningful sections — by heading or clause, not by arbitrary character count — and indexed with metadata such as date, department and document type. Combining keyword search with semantic (vector) search helps with product codes, contract numbers and names that pure semantic search tends to miss. Tables, price lists and scanned PDFs need special handling; they're where generic setups fail most often. And outdated versions should be removed from the index, or the assistant will happily quote last year's policy.

2. Citations and "no source, no answer"

Every answer should link to the passages it's based on, so users can check it in one click. Just as important: when the search finds nothing relevant, the assistant should say so instead of improvising. A clear "I couldn't find this in the available documents" builds more trust than a fluent guess.

3. An evaluation set

Collect real questions from your team, together with the correct answer and the document it comes from. Run every candidate model, every prompt change and every index update against this set before it goes live. It turns "the new model feels better" into a measurable comparison — and catches regressions before your users do.

How do you secure a private AI system beyond the model?

Keeping data on your own servers solves the third-party question. It doesn't solve everything else. Four areas deserve as much attention as the model choice:

• Permissions in retrieval. Access rules must be enforced when documents are searched — not by asking the model to "ignore documents the user may not see". If the search step returns a confidential HR file, the model will use it. Permissions from your file shares, wiki or document management system should carry over to the index and stay in sync.
• Prompt injection. Documents, emails and web pages can contain text that reads like instructions to the AI. A private model is just as susceptible as a public one. Treat retrieved content as data, give the assistant only the tools and permissions it really needs, and require human confirmation for actions with consequences.
• Logging. Logs are essential for audits and debugging — but prompts and answers often contain personal data themselves. Define who can read logs, how long they're kept and how they're handled in access and deletion requests.
• Updates. The model, the serving software, the vector database and the interface all need patches. Download models only from trusted sources, verify them, and run your evaluation set after every update.

How we handle data, access and hosting in our own projects is described on our security page.

Is local AI GDPR-compliant?

Running AI locally doesn't make a system compliant on its own — but it removes several of the hardest questions:

• No third-country transfers. If data never leaves your EU infrastructure, there's no transfer to a non-EU AI provider to justify.
• No extra processor for inference. The model provider never sees your data, so there's no additional data processing agreement for that step.
• Data minimisation and security under your control. You decide what gets indexed, who can access it and how long logs are kept.

You still need a legal basis for processing, clear access rules, deletion concepts and — for high-risk processing — a data protection impact assessment. If you use a private cloud provider, that provider is a processor and needs a data processing agreement like any other hosting service. In Germany, the works council typically has a say when a system can be used to monitor employee behaviour, which usage logs can. And if the assistant talks to customers rather than employees, the transparency rules of the EU AI Act come into play — see our guide to Article 50 for chatbots.

We design the setup together with your data protection officer. (This is general information, not legal advice.)

What does private AI cost compared with cloud AI?

The two models have different cost structures, and which one is cheaper depends on how much you use them.

Cloud AI is billed per token or per seat. There's no upfront investment, and for experiments or occasional use it's hard to beat. But costs grow with usage: per-seat licences scale with headcount, per-token costs scale with every question, every long document and every automated workflow. Price changes are decided by the provider.

Private AI turns most of this into fixed costs: hardware or a dedicated cloud instance, a one-off setup (connectors, search index, interface, access control, evaluation) and ongoing maintenance (updates, monitoring, index care). Once that's in place, an additional question costs almost nothing until you reach the capacity of your hardware.

A rough rule of thumb: the more people use AI every day and the more data each request processes, the stronger the case for private AI. A team of five testing ideas is usually better served by cloud tools; a company where hundreds of employees search documents daily, or where workflows process thousands of records, often reaches the point where fixed capacity beats per-use pricing. Don't forget the hidden costs on both sides — internal time for operation and updates on the private side, unused seats and integration work on the cloud side.

Local, private cloud or hybrid — which one should you choose?

Hybrid is often the pragmatic answer: customer and HR data stay on a local model, while non-sensitive tasks can use a large cloud model — with clear, enforced rules about what goes where. "Enforced" is the key word: the routing should happen in the system, based on data source or user group, not depend on employees remembering which tool to use.

What are the most common mistakes with private AI?

• Buying hardware before testing. The right model size depends on your tasks, languages and user numbers. Test first, then size the hardware.
• Picking the biggest model by default. A medium model with good retrieval usually beats a large model with poor retrieval — and needs far less hardware.
• Indexing everything at once. Old drafts, duplicates and outdated versions make answers worse. Start with curated, current sources.
• Ignoring permissions until later. Retrofitting access control onto a running system is painful, and one leaked salary list destroys trust in the whole project.
• No evaluation set. Without real test questions, nobody can say whether a change made things better or worse.
• Treating it as an IT project only. The data protection officer, the works council where applicable, and the teams who will use it belong at the table from the start.
• No owner after launch. Someone needs to keep sources current, review failed answers and plan updates. Without that, quality drifts.

How do you start with private AI?

The fastest path is a small, real pilot rather than a big platform decision. The 50 questions from step three become your evaluation set; the access rules from step two become the permission model for every later use case. If you'd like to find out which model and hardware fit your use case, book a free call or read more about our Private AI service.