Detection accuracy is reported as area under the ROC curve (AUC), measured on held-out, labeled test data that the recognizers never saw during development. AUC measures the trade-off between detection rate and false-positive rate across all possible thresholds — a score of 1.0 is perfect; 0.5 is random.
Balanced test-set scores flatter real-world performance. In production, genuine violations are rare against a high volume of normal traffic — which means false-positive rate matters more than it appears in a balanced test. We are explicit about this because it is how these systems are misrepresented in the market. Compiled operates as a high-recall triage layer: recognizers flag candidates for analyst review; thresholds are calibrated to your actual traffic before go-live so you tune precision and recall to your tolerance.
A strong score on a controlled test set is necessary but not sufficient. We also validated each recognizer against independent, real-world corpora it was never built against.
The insider-trading recognizer was tested against real SEC and DOJ enforcement communications (positives) versus real Federal Reserve communications and public earnings-call transcripts (legitimate text, negatives). Result: zero false positives on the legitimate corpus. This matters because the failure mode for a compliance tool is not missing a violation — it is generating a queue of false alerts that analysts stop trusting. On real enforcement text, recognition held.
These tests confirm that recognizers generalize beyond the conditions they were built in. They do not guarantee performance on your specific traffic. Your content distribution, language style, and risk population will differ from any test corpus. Calibration on your data before go-live is not optional — it is the step that converts a validated recognizer into a tuned operational tool.
Alert fatigue is the failure mode we engineer against. A system that flags too much trains analysts to ignore it.
Every recognizer change is tested against a frozen, held-out regression suite before it ships. Detection quality is pinned: no change that regresses the score below the established floor reaches your deployment. This is a hard gate, not a review process.
Compiled is not the only organization that has concluded an out-of-band semantic control layer is the right architecture for AI-agent governance. Two independent signals are relevant.
OWASP's 2025 Top 10 for Agentic Applications names the mitigation category that Compiled occupies — an out-of-band semantic inspection layer that monitors agent actions against policy before they execute. This is an independent taxonomy, not a Compiled claim. It reflects where the security research community has landed on what controls are actually needed.
Microsoft Entra Agent ID and AWS Bedrock AgentCore both represent the major infrastructure providers converging on the out-of-band control model: a governance layer that sits outside the agent and evaluates its actions against policy. This is corroboration that the architectural approach is correct, independent of Compiled's specific implementation.
Security vendors overclaim. We would rather lose a prospect by being honest than gain one by overpromising and failing in production.
The air-gapped pilot lets you run Compiled on a sample of your own traffic with zero data egress — no network connection required, no vendor relationship required. You see real verdicts on real content before committing to anything.
The full detection methodology and a benchmark brief — including held-out test design, corpus construction, calibration methodology, and per-category results — are available under NDA for customers in active evaluation.
Request the validation brief — the full methodology, benchmark results, and real-world test data — or start the air-gapped pilot and run it on your own traffic first.