Building a Synthetic Attack Data Generator, Part 2: Wiring Up Local Narrative Generation

The schema and generator from Part 1 produce a fixed facts dict per alert: technique, actor type, entity, a timeline of what actually happened. The one field still open was narrative, the analyst-style prose an LLM classifier would read. The design constraint was strict: the model paraphrases facts it’s given, it never gets a vote on what those facts are. Getting there took three real bugs, not one.

Bug one: the model name isn’t the model name

First call to LM Studio’s OpenAI-compatible endpoint (localhost:1234/v1/chat/completions) with model: "qwen3-8b" came back with a 400. My own error handling was swallowing the actual reason, raise_for_status() only surfaces the HTTP status, not LM Studio’s error body, so the first fix was making the client print the response text on failure instead of just the status code.

The real answer came from curl http://localhost:1234/v1/models, which lists what’s actually loaded and under what identifier. LM Studio wasn’t serving qwen3-8b, it was serving qwen/qwen3-8b. Small thing, but it’s the kind of small thing that wastes an hour if your error handling doesn’t tell you where to look.

Bug two: Qwen3 thinks before it answers

Model id fixed, next failure was a 30-second read timeout. Qwen3 models reason by default before producing a final answer, and for an 8B model on a Mac Mini’s GPU that reasoning pass alone can exceed a 30-second budget, especially when max_tokens is set low enough that the hidden reasoning tokens consume the whole budget before the model ever reaches the visible answer. This task, turning a handful of fixed facts into two or three sentences of prose, doesn’t need chain-of-thought at all.

Qwen3 exposes a documented switch for this: appending /no_think to the user turn disables the reasoning pass. That, plus a longer timeout as a safety margin, fixed the latency completely. Worth knowing before spending time tuning max_tokens or assuming the hardware is the bottleneck, the model’s default behavior was the actual cause.

Bug three: the one that actually mattered

With the plumbing fixed, the first real response looked fine at a glance. It wasn’t. Given a timeline describing an encoded PowerShell download cradle connecting out to a hosting-provider IP, execution plus a C2-style beacon, the model’s narrative called it an “exfiltration attempt.” Nothing in the facts said anything about data leaving the environment. The model inferred a threat stage that wasn’t there, and got it wrong in the process.

This is the failure mode that actually threatens the project’s design principle. The whole point of the facts-dict architecture is that ground truth stays correct because the model never gets to decide it, only to phrase it. But phrasing that quietly adds an incorrect threat characterization is a label problem wearing a prose problem’s clothes. If that had gone untested into training data, the classifier could learn a false association between “PowerShell plus outbound connection” and “exfiltration” that has nothing to do with what was actually generated.

The fix was a tighter system prompt: explicitly forbid adding any attack stage, objective, or threat characterization not already present in the input facts, and explicitly require the false-positive narratives to name the specific surface pattern the rule matched on (the encoded command flag, in this case) rather than just describing the benign activity in isolation. Re-running the same two test cases afterward, the true positive narrative stopped short at describing the connection and stayed out of the exfiltration/compromise language, and the false positive narrative started explicitly saying things like “flagged due to encoded payload detection, which is a known false positive for benign automation scripts,” which is precisely the reasoning a SOC analyst would write down.

The lesson

None of these three bugs were exotic. Wrong identifier, a model’s default reasoning behavior, and an LLM quietly overstating a conclusion when asked to summarize. But that third one is the one worth testing for deliberately before trusting any LLM-augmented layer of a synthetic dataset, a model correcting for or embellishing facts it’s explicitly supposed to just paraphrase is easy to miss on a skim and expensive to leave in training data.