The root cause analysis layer of my dissertation platform needed a model behind it. The obvious path would have been an API call to a hosted model. The platform runs air-gapped, by design, so that was never an option — the model had to run locally, on hardware I actually had.
Choosing something that would actually run
The task was narrow: parse Syslog output from Cisco switches and turn it into a structured fault report. I needed something that would run on a normal workstation rather than a GPU cluster. Llama 3 8B, run through llama.cpp, hit that balance — the larger 70B variant would have needed hardware I didn't have.
The hard part wasn't running it
Getting the model running was the easy bit. Getting it to consistently return a structured object — fault classification, affected interface, recommended step — rather than a slightly different shape of answer every time, took real iteration on the prompt and how the request was framed.
Let the model do the reasoning. Don't trust it to also enforce its own output format.
Living with hallucinations
Even with a tighter prompt, the model would occasionally produce a plausible-sounding diagnosis that was simply wrong — a real interface name used in the wrong context, a remediation step that fit a different fault entirely. There's no way around that with an 8B model running locally. The practical fix was treating its output as a first draft for a human to check rather than something auto-applied.
Where this actually matters
Running models on-prem in an isolated network isn't just a university exercise. Government, defence, and financial services environments routinely need this kind of separation from the public internet. Building it once, properly, in a lab is what makes it a real skill rather than a theoretical one.