Confidential AI, explained: where your prompts actually go
When a company calls its AI "private" or "secure," those words can mean almost anything. This page explains, in plain language, what it means for an AI system to be genuinely confidential — for the text and documents you send it to never leave hardware you control — walks through the three main technical approaches to getting there, and gives you the exact questions to ask any AI vendor before you trust them with sensitive material.
What happens to a prompt, normally
When you use a typical AI chat service, the prompt — the text you type, plus any files you attach — travels over the internet to computers owned by the AI provider. There, the provider performs inference: running a trained AI model on your input to produce an answer. Your data exists, at least briefly, in readable form on machines you do not own, operated by people you have never met, in a data center you could not point to on a map.
That is not automatically a problem. It becomes one when the material is sensitive — legal files, medical records, business plans, unpublished research — and when you cannot answer basic questions: Is the prompt logged? For how long? Who can read the logs? Is it used to train future models? Is it passed to yet another company downstream? "Confidential AI" is the discipline of making those answers knowable, and ideally making the safest answers true by construction rather than by promise.
Three approaches, from strongest to weakest
There are three broad ways to keep AI processing confidential. They differ in one crucial respect: whether the guarantee comes from architecture (how the system is physically built) or from policy (what the provider promises to do).
On-premises inference
"On-premises" means the model runs on machines you own, in a building or rack you control. Your data never crosses a network boundary belonging to someone else, so there is no third party to trust. The cost is real: you must buy, power, and maintain the hardware yourself.
Trusted execution environments
A trusted execution environment (TEE) is a sealed region inside a processor that keeps memory encrypted even while a program is running, so not even the operator of the machine can read what is inside. Paired with attestation — a cryptographic proof that the exact intended code is running — a TEE lets you rent someone else's hardware while keeping your data unreadable to them.
Zero-retention APIs
An API (application programming interface — a way for one program to call another over the network) with a zero-retention contract means the provider promises not to store your prompts or use them for training. This is the weakest form: your data still leaves your hardware in readable form, and the guarantee lives in a legal document, not in the physics of the system.
These can be combined. A careful organization might run routine work through a zero-retention API and reserve on-premises inference for its most sensitive material.
Questions to ask any AI vendor
You do not need to be an engineer to hold a vendor accountable. These questions have concrete answers; a vendor who cannot or will not answer them is telling you something.
- Where, physically, does inference happen?A country, a data center, a named cloud provider — not "the cloud."
- Who operates that hardware, and can they read my data?In transit, at rest on disk, and in memory while the model runs.
- How long are my prompts and outputs retained, and in which logs?Debug logs and abuse-monitoring systems often keep copies that marketing pages never mention.
- Is my data ever used to train or improve models?Get the answer in the contract, not the sales call.
- Which subprocessors touch my data?A subprocessor is another company your vendor passes data to — hosting providers, monitoring services, other AI providers.
- Can you prove any of this technically, or is it contractual only?Attestation reports and architecture documents are proof; a privacy policy is a promise.
The principle in practice
This site is part of a network built by someone who applies the strictest version of this principle to his own work. The research systems behind jew.tech — a project connecting Torah scholarship with computer science — run entirely on self-operated hardware, with no third-party inference API anywhere in the pipeline. Every model that processes the project's texts runs on machines the builder owns and administers himself.
An honesty note: on-premises is not a magic word. Owning the hardware removes the third-party risk, but it makes you responsible for everything a good provider would otherwise handle — physical security, patching, backups, and access control. Confidentiality is a property you maintain, not a product you buy once.
Part of a larger project
This page belongs to a network of small, honest reference sites on Judaism, computer science, and the infrastructure that connects them. Related reading:
privateaicloud.com — what a "private AI cloud" is and when it makes sense. localaihosting.com — running AI models on your own machines, from a laptop up. digitaltwinpro.com — digital twin work built on the same self-hosted principles.
The hub for the whole network, including the research it supports, is jew.tech; the full site directory is at jew.tech/sites/.