Running Llama 3 on-prem in an air-gapped lab

When I decided to add LLM-powered root cause analysis to my chaos engineering platform, the obvious path was an API call to OpenAI or Anthropic. But the platform runs in an air-gapped network — no internet route, by design. So I had to run the model locally.

Here's what that actually looked like, and what I learned from it.

Choosing the model

The task was narrow: parse Syslog output from Cisco switches and produce structured RCA reports. I needed something that could run on consumer hardware (a mid-range workstation, not a GPU cluster), handle technical vocabulary, and produce consistent structured output.

Llama 3 8B hit the right balance. Quantised to Q4_K_M with llama.cpp, it ran at roughly 12 tokens/second on CPU — slow enough to be annoying, fast enough to be useful. The 8B parameter count was key: the 70B variant would have required hardware I didn't have.

Getting structured output reliably

The hardest part wasn't running the model. It was getting consistent JSON output from it. Unstructured log parsing was fine — the model understood Cisco Syslog format well. But when I needed it to return a structured object (fault classification, severity, affected interfaces, recommended steps), the output varied in ways that broke downstream processing.

The solution was a combination of strict prompt engineering and a grammar-constrained sampling backend. With llama.cpp's grammar support, you can define a JSON schema and the model's token sampling is constrained to valid outputs matching that schema. Hallucinated field names became impossible. Malformed JSON became impossible.

Constrained generation is the practical solution to LLM reliability in production tooling. Let the model do the reasoning; let the grammar enforce the format.

The confidence problem

Even with structured output, the model sometimes produced plausible but incorrect diagnoses — a real interface name used in the wrong context, a remediation step that made sense for a different fault type. Without a confidence score, there was no way to flag these for human review.

I implemented a rough proxy: ask the model to rate its own certainty on a 1–5 scale alongside the diagnosis. It's not statistically calibrated, but it proved surprisingly useful. Outputs with self-reported certainty of 1–2 correlated strongly with the cases where my manual review found errors. Low-confidence outputs got routed to a review queue rather than auto-committed to the report.

What the isolation actually costs you

Model updates are manual. When Meta releases a new Llama version, I have to download it externally, transfer it in, and re-validate the prompts. That's not a big deal for a dissertation project. In a production air-gapped environment with a proper change management process, it's a real operational consideration.

The same applies to the llama.cpp binary itself, any Python dependencies, and the inference server. Everything has to be treated like infrastructure — versioned, tested before deployment, and rolled back carefully if something breaks.

Where this pattern actually belongs

Running LLMs on-prem in isolated environments isn't a niche academic exercise. HMRC, MOD, NHS, financial services — anywhere data classification or network separation requirements prevent cloud egress. The pattern I implemented in a university lab is directly applicable to real enterprise AI deployments.

The operational overhead is real but manageable. The capability gain — automated analysis of technical telemetry that would otherwise require a skilled engineer to parse manually — is significant enough to justify it.