What building an air-gapped chaos lab taught me

For my final year dissertation I built ACRP — the Automated Chaos Resilience Platform. The brief was: design a chaos engineering environment that runs entirely off-grid, injects real network faults, and produces root cause analysis reports automatically. No cloud. No internet. A rack of Cisco switches, a self-hosted GitLab, and a local Llama 3 8B model doing the heavy lifting on log analysis.

Six months later I had something that worked. Here's what I wish I'd known before I started.

The isolation constraint forces you to understand every dependency

The first week I lost two days chasing a package install that silently failed because pip couldn't reach PyPI. Obvious in retrospect. But that moment set the tone for the entire project — in an air-gapped environment, you can't delegate understanding to a CDN or an API. Every tool, every runtime, every certificate has to be present and accounted for before you start.

Most engineers never have to think about what a Python package actually is at the byte level, or what DNS does when there's no resolver to ask. Air-gapping forces you to. It's uncomfortable at first. Then it becomes a superpower.

Chaos engineering is really just disciplined observation

The common perception of chaos engineering is "deliberately breaking things." That's the visible part. The actual discipline is in the observation loop: you define a steady-state hypothesis, inject a fault, measure deviation, and learn something. The fault injection is almost trivial compared to the instrumentation work.

My SRE control panel for ACRP wasn't impressive because it could trigger link-down events on Cisco interfaces. It was impressive because it could tell you — in real time — what the blast radius of that event was and surface the logs that explained why the system behaved the way it did.

The difference between a chaos experiment and random sabotage is documentation. If you can't reproduce the fault and measure its effect, you learned nothing.

LLMs are genuinely useful for log parsing — with one big caveat

Integrating Llama 3 8B for automated RCA was the most experimental part of the project. The input was raw OOB switch logs (Syslog format, lots of noise). The output was a structured summary: fault classification, probable cause, recommended remediation steps.

It worked well enough to be useful — and that surprised me. Raw switch logs are dense and vendor-specific. A model that can extract "STP topology change on Gi0/1 due to port going down" from forty lines of noise is genuinely time-saving.

The caveat: hallucinations. The model occasionally invented interface names or suggested remediation steps that were plausible but wrong. In a production environment that would be dangerous. In my dissertation lab it was a fascinating problem — it pushed me to build a confidence-scoring layer that flagged low-certainty outputs for human review rather than passing them straight through.

GitLab CI running off a bare-metal box is completely viable

One assumption I had coming in was that CI/CD required internet access — at minimum to pull images, at most to talk to cloud runners. Neither is true. GitLab can be fully self-hosted, runners can pull from a local registry, and pipelines can execute entirely within your air-gapped perimeter.

The pipeline I built ran automated fault injection tests on a schedule, collected metrics, generated reports, and committed them back to the repo. All of it happened on a box with no route to the outside world.

This matters beyond dissertations. Defence, finance, and government environments routinely require this kind of isolation. The skills transfer directly.

What I'd do differently

I'd start with observability. I spent the first month building the fault injection mechanisms and the last two months scrambling to add enough instrumentation to actually understand what the faults were doing. The ratio should have been reversed.

I'd also set up the local model registry before anything else. Getting Llama 3 running locally in an isolated environment takes longer than you expect — quantisation, inference speed, prompt engineering for structured output — and it's a bad dependency to still be debugging in week ten.

The project ended up in a good place. But the path was messier than it needed to be, and most of the mess came from not taking the isolation constraint seriously enough on day one.