Start with the measurement, not the score

Breach and Attack Simulation platforms are good at producing repeatable tests and easy-to-read results. That can be valuable. But the score only matters if you understand how the test was constructed, what telemetry it can see, where it stops and which parts of the attack surface it never exercises.

‍

If a vendor positions BAS as a replacement for purple teaming, these are the questions to run against the claim.

‍

Agent lineage and attacker realism

1. What does the process lineage of every simulated technique look like? If everything is a child of the vendor’s agent, is your EDR grading the agent instead of the attack?
2. Where does offensive network traffic originate? Could static, well-known vendor infrastructure be blocked on reputation rather than behavior?
3. Do payloads and tradecraft rotate, or are you replaying signatures every major vendor already knows?
4. Do tests validate outcomes? For example, whether credentials could actually be taken — rather than simply confirming a tool detection?

‍

Exclusions and trust

5. ‍‍Does the platform require exclusions, allowlists or trusted-process configuration to run at all? Every one of those can alter how the sensor perceives activity.
6. How do results account for telemetry that an exclusion suppresses? A skipped rule match is neither a real detection nor an honest miss.
7. Do processes spawned under the agent inherit its trust, and how is that controlled for?
8. Can you show me a run with no exclusions at all? If the product cannot function that way, how does it measure organic detection?

‍

Telemetry and left-of-bang detection

9. How is a detection confirmed: a single event in a time window, or correlated evidence across the whole attack chain?
10. Does the platform understand our data schema well enough to query the telemetry surrounding each test?
11. Does it surface left-of-bang events we should alert on even when the final action gets blocked?
12. Does it help us build detections for staging behavior, or only grade the boom?

‍

Defense in depth

13. Can scenarios run with front-line controls in detect-only or disabled mode to exercise the layers behind them?
14. Does execution continue past the first block to validate the full chain end to end?
15. How does the platform emulate an attacker operating from an unmanaged or unenrolled asset?
16. How quickly does new in-the-wild tradecraft land in the content library — days, weeks or quarters?

‍

Identity, SaaS and emerging attack surfaces

17. Can the platform execute identity and SaaS attack paths, or only endpoint techniques?
18. Does it validate detections in SaaS audit logs with the same rigor as endpoint telemetry?
19. What is the roadmap for emerging surfaces such as agentic coding tools and AI integrations?

‍

If the honest answer is “we don’t cover that,” that is useful information. Budget for bespoke testing to fill the gap rather than pretending the gap does not exist.

‍

Resilience and evidence

20. Can you pivot from the score into the evidence underneath it — raw telemetry, detections fired, gaps identified, resilience by attack vector and actor tradecraft, visibility gaps, and the security investments that need attention?

‍

That last question is deliberately bigger than a dashboard metric. Leadership should be able to connect validation results to what the security stack can see, where it fails and whether specific investments are actually improving resilience.

‍

What not to overlook

If the sales pitch calls BAS a replacement for purple teaming, run every question above against it first. What survives is what you are actually buying.

‍

BAS can be an excellent regression layer after controls and attack paths have been validated. Purple teaming is what tells you whether those controls, telemetry and assumptions hold up when the test behaves like an adversary.

‍

For the technical case behind these questions, read Breach and Attack Simulation Isn’t Purple Teaming.

‍

A green heatmap is a promise. Purple teaming done right is proof.‍