Why air-gapped RAG matters

Retrieval-augmented generation, RAG, is the technique behind every "chat with your documents" product built since 2023. It is also the single most common way sensitive material is quietly shipped to a third party. This page explains what RAG is, why running it air-gapped is not a preference but a requirement for defense and intelligence work, and how a zero-egress RAG stack is actually built. Azcuna Dynamics builds two of them: BABEL for multilingual document exploitation and ELLMENT for E-2D mission crews.

What retrieval-augmented generation is

A language model on its own answers from memory. That memory is whatever was in its training data, frozen at training time, blended together, and unattributable. Ask it a question about your documents and it will produce a fluent answer that may or may not be true and cannot point to a source.

RAG changes the order of operations. First the system searches your documents for the passages most relevant to the question. Then it hands those passages to the model with an instruction: answer using this evidence, cite it, and if it does not answer the question, say so. The model becomes a reader and a writer, not an oracle.

The result is an answer grounded in text the operator can open and verify. When the corpus is a set of procedures, logs, regulations, or captured documents, that grounding is the entire value. An answer without a source is an opinion.

Where the data goes in a normal RAG deployment

Follow the bytes in a typical cloud RAG product.

The documents are uploaded to the vendor. They are split into chunks and each chunk is sent to an embedding API, which returns a vector. The vectors are stored in a hosted database. At question time the question is embedded by the same API, the nearest chunks are fetched, and the chunks plus the question are sent to a hosted language model. The answer comes back.

Count the egress points: the upload, every embedding call, the vector store, every query, every generation. Each one is a copy of your material on someone else's infrastructure, subject to their retention policy, their logging, their subprocessors, and their breach. "We do not train on your data" is a promise about one use of the copy. It is not the absence of the copy.

For a business, that is a vendor-risk question. For a program handling controlled or classified information, it is disqualifying. The data cannot leave the enclave. Therefore the model has to come to the data.

The three tiers: on-premise, private, air-gapped

Vendors use these words loosely. They are not the same.

On-premise means the software runs on servers you own. Those servers may still have internet access, and the software may still call out for updates, telemetry, or model inference. Better than cloud. Not isolated.

Private or dedicated usually means a vendor runs a single-tenant instance for you in their cloud. Your data still leaves your perimeter; it just has fewer neighbors.

Air-gapped means no network path exists between the system and the outside world. Model weights, index, documents, and interface are all inside. Updates arrive on media. This is the only tier that satisfies a SCIF, a forward operating location without connectivity, or an aircraft.

BABEL and ELLMENT are built to the third tier and tested by unplugging the network. If a feature breaks without a connection, it is a bug.

Building a zero-egress RAG stack

Every component of a RAG pipeline has a cloud default. Each one has to be replaced with a local equivalent that runs on the hardware in the room.

Ingestion

Documents arrive as PDFs, scanned images, office files, chat exports, and message logs, in several scripts. Each is parsed, cleaned, and split into chunks sized to the model's context budget. Chunking is script-aware: Arabic and CJK text are more expensive in tokens and are cut into smaller pieces. Every chunk keeps a provenance record: source file, section, page, and character span. That record is what makes a citation possible later.

Embedding

Each chunk is converted into a vector by a small sentence-embedding model that runs locally. BABEL uses a compact 384-dimensional embedding model that indexes tens of thousands of chunks on a laptop in minutes. The model is a file on disk. No API is called.

Vector index

Vectors go into a local approximate-nearest-neighbor index. For corpora under about ten thousand documents a flat index is exact and fast enough. Larger corpora use a partitioned index that trades a small amount of recall for speed. Either way it is a file next to the documents.

Hybrid retrieval

Semantic search alone fails on the material that matters most. Procedures, logs, and technical documents are full of exact identifiers: system names, procedure numbers, call signs, brevity codes, coordinates. A vector search understands "engine fire on the ground" but can miss "NATOPS 11.3.4" because the embedding smooths over exact strings.

The fix is hybrid retrieval. A dense semantic search and a lexical full-text search run in parallel. Their results are fused and re-ranked. The lexical side, a local full-text index, catches the exact identifiers; the semantic side catches the paraphrases. ELLMENT's retriever is built this way because tactical text is unforgiving of a miss.

Grounded generation

A quantized open-weight language model, served locally, receives the question and the retrieved passages. The prompt constrains it: answer only from the evidence, cite each claim inline with its source, and if the evidence does not support an answer, return "insufficient grounding" instead of guessing. The model runs on the laptop GPU, on Apple Silicon through Metal or on an NVIDIA card, and on the CPU as a fallback.

Citations and evidence view

Every sentence of an answer carries a reference to the chunk it came from, and the operator can expand the exact retrieved text beside the answer. This is not decoration. It is the mechanism by which a human verifies the machine in seconds and the reason a RAG answer can be trusted where a raw model answer cannot.

Abstention and confidence

A model that always answers is a liability. The system computes a confidence score from retrieval strength and answer-evidence agreement, and below a threshold it declines. In the recorded ELLMENT session on this site, the prototype answers a carrier NATOPS question with a page citation and then refuses a fabricated brevity code, because the code is not in the corpus. That refusal is the feature.

Audit

Model version, corpus version, configuration, and every query with its retrieved evidence and answer are logged locally. Results are reproducible, the system can be evaluated against a fixed question set, and an after-action review can see exactly what the operator was shown. The log is a local file with the same access controls as the documents.

Why not just fine-tune the model on our documents

This is the most common alternative proposed, and it is usually wrong for document work.

Fine-tuning changes the model's weights so that it "knows" your material. But knowledge in weights cannot be cited, cannot be updated without retraining, cannot be removed when a document is superseded, and does not stop hallucination; the model still generates fluently past the edge of what it learned. It also requires a training pipeline, GPU time, and a data-preparation effort that must be repeated for every corpus change.

RAG keeps documents as documents. Add one, re-index. Retire one, remove it from the index. Ask a question, get an answer with a page number. The model is a general reader that never needs to be retrained on your data, which also means your data never becomes part of a model artifact that could be copied.

Fine-tuning has a place: adapting a model to a domain's vocabulary and style, or improving translation quality on a language pair. That is a change to how the model reads, not what it has memorized, and it is done once, offline, on curated material.

Automation bias: the human factors problem

A grounded, cited, confidence-scored system still faces a human who is tired, under pressure, and inclined to trust a confident screen. Automation bias is the tendency to accept a machine's output without checking it, and it is the failure mode that matters most in a cockpit or an operations center.

An air-gapped RAG system helps in three ways that a cloud chatbot does not. Citations put the source one click away, which makes checking cheap. Abstention trains the operator that the system will say no, which keeps trust calibrated. And the audit log makes it possible to measure, after the fact, how often operators verified and how often they should have. ELLMENT's Phase I plan includes operator-in-the-loop evaluation with Naval Flight Officers for exactly this reason.

Hardware, honestly

An air-gapped RAG stack for a corpus of a few thousand documents runs on a laptop with 16 GB of RAM. Indexing a large corpus takes minutes. A question takes seconds. A workstation with a modern GPU makes it faster and allows a larger model. None of it requires a server, and none of it requires a network.

That is the whole point. The intelligence is in the room with the documents, and it stays there.

What to ask a vendor

• Unplug the network. Does it still answer?
• Where are the embeddings computed? Where is the index stored? Where does generation run?
• Show me the retrieved passages for this answer.
• Ask it something the corpus does not contain. Does it refuse?
• What is in the audit log, and who can read it?
• Which model is inside, and where did the weights come from? Model provenance is the next page.

BABEL and ELLMENT answer each of these in a live session. The recorded session on the ELLMENT page is real prototype output, unedited.