Building a Synthetic Attack Data Generator, Part 1: Schema and the FP Problem

I’m building a synthetic attack data generator on a Mac Mini M4 with 16GB of unified memory. The end goal is a fine-tuned classifier that triages SOC alerts as true positive or false positive, but that classifier is only as good as the data it’s trained on, and the data is where most of the design work actually lives.

The naive version doesn’t work

The obvious approach: generate malicious logs, generate benign logs, label them, done. That produces a classifier that learns “attack keywords present,” which is a party trick, not triage. Real analyst fatigue comes from alerts that are false positives because they legitimately match the rule, not because they’re obviously unrelated noise. A sysadmin’s -EncodedCommand maintenance script and an attacker’s download cradle both trip the same Sysmon rule. A CI/CD service principal calling iam:ListUsers from an unfamiliar egress IP looks identical, on paper, to the first move of a compromised credential.

So every technique in this generator has to produce two things: a true-positive scenario and a false-positive scenario that trips the same rule, for the same reason, with different underlying intent. That’s most of the actual engineering effort in this project. It roughly doubles the generation work per technique, and I built it that way on purpose.

Three layers, not one

The schema has three layers:

RawLogEvent is a single native-format log record (Sysmon JSON, CloudTrail, Azure Activity Log, GCP Audit Log), tagged with ground truth: is this malicious, which MITRE technique, one line on what’s actually happening. This is what gets replayed into Cribl for pipeline and detection-rule testing.

Session groups related events into one coherent narrative, an attack chain or a benign activity burst, with consistent entities (same user, same host, same cloud principal) across the whole thing.

Alert is the actual training unit for the classifier. It’s produced by running a simulated detection rule against a session, and it carries two parallel representations of the same underlying activity: a structured feature dict for a classical classifier baseline, and a natural-language narrative for an LLM-based classifier. Same session, same label, two input formats, so I can compare approaches on identical ground truth instead of two different datasets.

The alert also carries the label as a field that gets held out at inference time, TP or FP, plus an fp_reason when it’s a false positive, explaining specifically why the benign activity resembles the technique.

Proving it out on two techniques

Before building out all twelve techniques in the starter set, I built the full pipeline end to end on two: PowerShell execution (T1059.001, Sysmon) and Valid Accounts: Cloud Accounts (T1078.004, CloudTrail). Small scope, but it forces every layer of the schema to actually work before I sink time into the other ten.

The PowerShell true positive: an encoded command launched from Explorer, decoding to a download-and-execute cradle, followed within seconds by a network connection to a hosting-provider IP. The false positive: an encoded command launched by the task scheduler, decoding to a completely mundane admin script (patch status export, stale log cleanup), followed by a plain file write. Same detection rule fires on both, -enc present in a powershell.exe command line is the naive rule logic, but the structured features that separate them are real and observable: parent_image, whether a network connection followed, whether a file write followed instead.

I also encode the PowerShell payloads for real rather than faking the string. The generator writes an actual command, encodes it UTF-16LE then base64 the same way -EncodedCommand does, and the alert only ever sees the decoded plaintext in the ground truth. Decoding the generated CommandLine field back to plaintext round-trips correctly, so if you’re testing a Cribl pipeline stage that decodes and inspects encoded PowerShell, it has something real to chew on, not a placeholder.

The cloud side follows the same pattern: a console login from an anomalous source with MFA off, followed by IAM and EC2 discovery calls, versus a login from a new-but-plausible travel location with MFA on, followed only by the account’s routine S3 access. Same rule, same surface signal (new source IP), different follow-on behavior.

What’s next

Ten more techniques to go, using the same TP/FP-pair pattern, then the narrative generation step gets handed to a small local model (Qwen3 8B, running in LM Studio) instead of the templated fallback, conditioned on a fixed facts dict so it can vary the prose without ever getting a vote on what actually happened. Once there’s enough volume, Phase 2 starts: deciding whether the classifier itself should be a fine-tuned LLM or a classical model trained on the structured features, and making that case with actual precision/recall numbers instead of picking the trendier option.