Penetration Testing | Red Team Exercises

To meet the penetration testing | red team exercises requirement in NIST SP 800-53 Rev 5 CA-8(2), you must run red team exercises that realistically simulate adversary attempts to compromise your systems, and you must do so under documented rules of engagement that control scope, safety, authorization, and reporting (NIST Special Publication 800-53 Revision 5). Operationally, that means a planned, authorized, and evidenced campaign that produces findings, remediation, and retesting.

Key takeaways:

• Red teaming under CA-8(2) is an assessment activity with formal rules of engagement, not an ad hoc pentest (NIST Special Publication 800-53 Revision 5).
• You need evidence of authorization, scope, execution, results, and remediation tracking, mapped to your system boundary (NIST Special Publication 800-53 Revision 5).
• The fastest path is to standardize a repeatable playbook: ROE template, provider onboarding, logging/safety controls, and a closure workflow.

CA-8(2) sits in the Assessments, Authorization, and Monitoring family and focuses on realism: you are expected to emulate adversary behavior against your environment, not just run a scanner or a checklist-based penetration test (NIST Special Publication 800-53 Revision 5). For a Compliance Officer, CCO, or GRC lead, the practical question is, “What do I need in place so an assessor believes we actually conduct controlled red team exercises, and we can prove it?”

This requirement becomes “real” during audits because red team activity crosses multiple lines: production safety, change control, incident response, third-party management, and legal authorization. The most common breakdown is not technical. It’s governance: unclear rules of engagement (ROE), fuzzy system boundary, poor evidence capture, and no closed-loop remediation. The good news is you can operationalize CA-8(2) quickly by treating red team exercises as a defined program with repeatable artifacts, clear approvals, and a workflow that ties results to fixes and validation.

Use this page as a requirement-level checklist: who is in scope, what to build, what to run, what to retain, and what auditors ask when they suspect “red team” is just marketing language.

Regulatory text

Requirement (verbatim): “Employ red team exercises to simulate attempts by adversaries to compromise organizational systems in accordance with applicable rules of engagement.” (NIST Special Publication 800-53 Revision 5)

Operator interpretation: what you must do

• Employ red team exercises: You need a deliberate activity (internal team or third party) that conducts adversary-emulation tradecraft, not just vulnerability scanning or a standard web app test.
• Simulate attempts by adversaries: The exercise should reflect credible attacker goals (initial access, privilege escalation, lateral movement, data access, persistence), aligned to your environment and threat model.
• In accordance with applicable rules of engagement: You must define and follow ROE that governs what is allowed, where, when, how you control risk, and how you coordinate with defenders and stakeholders (NIST Special Publication 800-53 Revision 5).

Plain-English requirement: what “counts” as meeting CA-8(2)

You meet CA-8(2) when you can show that:

1. You planned a red team exercise against in-scope systems.
2. You authorized it with explicit approvals and safety constraints (ROE).
3. You executed adversary-style tactics within the approved scope.
4. You reported outcomes in a way the business can act on (findings, affected assets, paths taken, evidence).
5. You remediated and validated fixes (or documented risk acceptance) and retained evidence.

If you only have quarterly external pentests that focus on known vulnerabilities, treat that as helpful input, but not sufficient to demonstrate “red team exercises” unless the work truly simulates adversary behavior and is governed by formal ROE (NIST Special Publication 800-53 Revision 5).

Who it applies to (entity + operational context)

Primary in-scope organizations

• Cloud Service Providers operating a system that must align to NIST SP 800-53 controls (NIST Special Publication 800-53 Revision 5).
• Federal Agencies operating organizational systems under the same control baseline expectations (NIST Special Publication 800-53 Revision 5).

Operational contexts where CA-8(2) shows up

• Systems with defined authorization boundaries where you must demonstrate ongoing assessment and monitoring discipline.
• Environments with material exposure (internet-facing services, identity infrastructure, CI/CD, administrative interfaces, high-value data stores).
• Any environment where a red team exercise could create operational risk unless tightly governed (production SaaS, shared cloud accounts, multi-tenant platforms).

What you actually need to do (step-by-step)

1) Define the red team scope against your system boundary

• Start with your authorized system boundary (accounts/subscriptions/projects, VPC/VNETs, endpoints, applications, identity providers).
• Decide what is in scope (prod vs non-prod, corporate network vs cloud control plane, third-party services).
• Decide what is explicitly out of scope (patient/customer production data extraction, denial-of-service, destructive actions).

Deliverable: Scope statement tied to the system boundary used for assessments (NIST Special Publication 800-53 Revision 5).

2) Write Rules of Engagement (ROE) that an auditor can read

Your ROE should be a controlled document approved by accountable leaders. Include:

• Objective(s): what constitutes success (e.g., demonstrate path to sensitive data, domain admin, cloud admin, or business-impact scenario).
• Time window and testing cadence: define when actions may occur and when they must stop.
• Allowed tactics and prohibited actions: include explicit no-go actions (e.g., DoS, ransomware simulation without encryption, destructive payloads).
• Target list and constraints: systems, networks, apps, identity tenants, cloud subscriptions.
• Safety controls: rate limits, data handling constraints, test accounts, kill switch, comms plan.
• Coordination model: fully blind, partially informed, or purple-team style, and what the defenders are told.
• Authorization: named approvers, legal/HR involvement if social engineering is allowed, and escalation contacts.

Deliverable: Signed ROE with approvals and distribution list (NIST Special Publication 800-53 Revision 5).

3) Choose your execution model (internal, third party, or hybrid)

• Internal red team: strong for repeatability and institutional knowledge; confirm independence and reporting lines.
• Third party red team: strong for specialization and fresh perspective; manage access, confidentiality, and evidence retention.
• Hybrid: internal leads with a third party to add capacity or niche skills.

If a third party is involved, treat them as a third party risk and access event: contract scope, confidentiality, access method, offboarding, and evidence handling.

Deliverable: Engagement SOW/contract + access approvals aligned to ROE (NIST Special Publication 800-53 Revision 5).

4) Prepare the environment so the exercise produces useful evidence

• Ensure logging and telemetry are enabled on in-scope systems (identity logs, cloud audit logs, endpoint telemetry, application logs).
• Confirm detection/response workflow: what happens if the red team triggers an alert, and when the team must pause for safety.
• Establish test identities and secrets handling: avoid production credentials reuse; document how credentials are stored and rotated.

Deliverable: Pre-flight checklist with sign-offs from security operations and system owners.

5) Run the red team exercise and capture defensible evidence

During execution, require the team to maintain:

• Activity log: dates, techniques used, targets touched, access gained, and evidence artifacts (screenshots, command outputs, cloud event IDs).
• Chain-of-custody and data handling: what data was accessed, where it was stored, and how it was protected.
• Real-time escalation for safety issues or scope uncertainty.

Deliverable: Execution log and evidence bundle mapped to ROE.

6) Report outcomes in a way you can remediate and retest

Your final report should include:

• Attack narrative: initial access path, privilege escalation, lateral movement, objective achieved or blocked.
• Control gaps: what failed (misconfig, missing MFA, excessive permissions, weak egress controls, monitoring gaps).
• Business impact framing: what could have happened if this were a real adversary.
• Recommended fixes with owners and priority.

Then run a closure workflow:

• Create tickets, assign owners, set due dates.
• Track remediation and exceptions/risk acceptance.
• Retest critical fixes or run a targeted follow-up exercise.

Deliverables: Final report, remediation tracker, retest evidence (NIST Special Publication 800-53 Revision 5).

Required evidence and artifacts to retain

Auditors typically want artifacts that prove planning, authorization, execution, and closure. Keep:

• Rules of Engagement (ROE) with approvals and version control (NIST Special Publication 800-53 Revision 5).
• Scope and boundary mapping (systems/accounts included).
• Engagement documentation (SOW/LOA, third-party access approvals, NDA as applicable).
• Pre-flight checklist (telemetry enabled, contacts, kill switch, test accounts).
• Execution evidence (activity logs, timestamps, key screenshots/output, attack path notes).
• Final report (findings, narrative, impact, recommendations).
• Remediation and verification records (tickets, changes, retest results).
• Lessons learned / after-action review (what you changed in detection, hardening, IR playbooks).

A simple way to make this audit-proof is to store everything in a single “Red Team Exercise” record with attachments and a clear mapping to the in-scope system. Daydream can help by standardizing this as a control workflow: ROE templates, evidence requests, third-party onboarding tasks, and a remediation tracker that stays tied to the requirement.

Common exam/audit questions and hangups

Expect questions like:

• “Show me the rules of engagement and who approved them.” (NIST Special Publication 800-53 Revision 5)
• “How did you decide what was in scope, and how does that map to your system boundary?”
• “Was this a red team exercise or a traditional pentest? Show me the adversary simulation elements.” (NIST Special Publication 800-53 Revision 5)
• “What stops the team from causing harm in production? Where is the kill switch and escalation path?”
• “Show me closure: tickets, fixes, retesting, or accepted risk.”

Hangups that stall audits:

• ROE exists but has no signatures or authority chain.
• Findings are in a PDF but remediation is not tracked.
• Evidence is scattered across email/Slack with no retained record.
• Scope excludes the actual “crown jewels” without a documented rationale.

Frequent implementation mistakes (and how to avoid them)

1. Calling a vulnerability scan a red team exercise.
   Fix: require an attack narrative and proof of adversary-like progression aligned to objectives.

2. ROE is generic and not tied to real systems.
   Fix: include concrete target lists, explicit prohibitions, and named escalation contacts.

3. No defender visibility, so you learn nothing about detection.
   Fix: decide in advance whether you want blind testing or a coordinated purple-team phase, then document it in ROE.

4. Uncontrolled third-party access.
   Fix: issue time-bound accounts, log all access, and complete access removal with evidence.

5. Remediation doesn’t close.
   Fix: treat red team findings like high-priority security defects; track to closure and retest.

Risk implications (why auditors care)

CA-8(2) is about validating that controls work against realistic attacker behavior (NIST Special Publication 800-53 Revision 5). Without red team exercises, you can pass routine control checks while still failing in the ways adversaries exploit: identity takeover, privilege escalation, weak monitoring, mis-scoped segmentation, and fragile operational response. The business impact is simple: weaknesses remain undiscovered until a real incident forces discovery under pressure.

A practical 30/60/90-day execution plan

First 30 days: Stand up governance and repeatable artifacts

• Draft and approve an ROE template with legal/security/operations input (NIST Special Publication 800-53 Revision 5).
• Define in-scope boundary mapping for the upcoming exercise.
• Choose execution model (internal/third party/hybrid) and document provider onboarding requirements.
• Build an evidence binder structure (folders/records) so artifacts land in one place.

Days 31–60: Prepare and execute

• Run pre-flight checks: logging, test accounts, comms plan, escalation path.
• Execute the red team exercise per ROE and collect evidence throughout (NIST Special Publication 800-53 Revision 5).
• Deliver a final report with an attack narrative and actionable findings.

Days 61–90: Close the loop and make it operational

• Convert findings into tickets with owners; track remediation and risk acceptance.
• Retest key fixes and attach verification evidence.
• Run an after-action review to update hardening standards, detections, and IR playbooks.
• Set a recurring governance rhythm: ROE refresh, scope review, and evidence retention checks.

Frequently Asked Questions

Do we need a third party to meet the red team exercises requirement?

No. CA-8(2) requires you to employ red team exercises under rules of engagement, but it does not require a third party (NIST Special Publication 800-53 Revision 5). If you use internal teams, document independence and keep evidence of authorization and results.

Can a standard annual penetration test satisfy CA-8(2)?

It can contribute, but only if it actually simulates adversary attempts and is governed by explicit rules of engagement (NIST Special Publication 800-53 Revision 5). Many “pentests” are vulnerability-focused and lack the adversary-emulation narrative auditors expect under CA-8(2).

What should the rules of engagement include at minimum?

Minimum: objectives, scope, time window, allowed/prohibited actions, safety controls, escalation path, and formal approvals (NIST Special Publication 800-53 Revision 5). If social engineering is in scope, add explicit legal/HR authorization and boundaries.

How do we prove the exercise happened if the work is sensitive?

Keep a controlled evidence package: ROE, execution logs, limited screenshots/output, and a report that avoids sensitive data while still showing attack paths and impacted assets. Auditors generally accept redacted evidence if it remains verifiable and tied to scope.

Do we have to run the red team exercise in production?

CA-8(2) does not mandate production, but the exercise must simulate credible adversary compromise attempts against organizational systems (NIST Special Publication 800-53 Revision 5). If you avoid production, document the rationale and show how the chosen environment meaningfully represents production controls and paths.

How do we operationalize remediation tracking for red team findings?

Treat findings like security defects: ticket them, assign owners, track status, and attach retest evidence or risk acceptance. A GRC workflow tool like Daydream helps by keeping ROE, evidence, and remediation in one control record that is easy to audit.