Penetration Testing

FedRAMP Moderate expects you to define which systems/components need penetration testing and how often, then execute those tests and retain evidence that the scope, method, findings, and remediation are controlled and repeatable. The requirement is simple on paper but fails in practice when “frequency” and “systems” aren’t formally defined and tied to risk and change. (NIST Special Publication 800-53 Revision 5)

Key takeaways:

• Define penetration test scope and frequency explicitly, then get them approved and repeatable. (NIST Special Publication 800-53 Revision 5)
• Treat pen testing as a lifecycle control: plan → authorize → test → report → fix → retest → attest. (NIST Special Publication 800-53 Revision 5)
• Keep audit-ready artifacts: rules of engagement, scope, results, remediation tickets, and retest proof. (NIST Special Publication 800-53 Revision 5)

“Penetration Testing” in NIST SP 800-53 Rev. 5 CA-8 is a requirement to run real-world attack simulations against defined systems at a defined cadence. For a Compliance Officer, CCO, or GRC lead supporting FedRAMP Moderate, the operational challenge is not deciding whether to test, but making the control defensible: you must show you chose the right targets, test often enough for your risk, used qualified testers, and actually closed findings. (NIST Special Publication 800-53 Revision 5)

In audits, CA-8 commonly turns into a documentation and governance exam. Assessors will look for a clear statement of (1) what is in scope, (2) what “frequency” means in your environment, and (3) how test results flow into remediation and retesting. If your testing is ad hoc, if third-party components are excluded without a rationale, or if you cannot show that high-risk findings were fixed and verified, you will struggle to demonstrate control effectiveness. (NIST Special Publication 800-53 Revision 5)

This page gives requirement-level implementation guidance you can execute quickly: scoping, scheduling, rules of engagement, evidence to retain, and the common failure modes that cause CA-8 findings.

Regulatory text

Requirement (excerpt): “Conduct penetration testing at an organization-defined frequency on organization-defined systems or system components.” (NIST Special Publication 800-53 Revision 5)

What this means for an operator

• You must define the cadence (your “organization-defined frequency”). “Annually” is common in many programs, but CA-8 does not prescribe a number; you must document your choice and defend it. (NIST Special Publication 800-53 Revision 5)
• You must define the targets (your “organization-defined systems or system components”). This should be a concrete inventory-based scope, not a vague “production environment.” (NIST Special Publication 800-53 Revision 5)
• You must conduct the testing, not just plan for it, and be able to prove it happened with the stated scope and frequency. (NIST Special Publication 800-53 Revision 5)

Plain-English interpretation (FedRAMP Moderate)

For FedRAMP Moderate, CA-8 is satisfied when you can show a repeatable penetration testing program that:

1. identifies which parts of the system boundary must be tested,
2. runs tests on a documented schedule (and after meaningful changes, if that’s how you define frequency), and
3. drives findings into remediation with evidence of fix verification. (NIST Special Publication 800-53 Revision 5)

Who it applies to

Entity types

• Cloud Service Providers supporting FedRAMP Moderate authorizations. (NIST Special Publication 800-53 Revision 5)
• Federal Agencies operating information systems aligned to NIST SP 800-53 Rev. 5. (NIST Special Publication 800-53 Revision 5)

Operational contexts where CA-8 is routinely examined

• FedRAMP assessments and continuous monitoring activities for in-scope cloud services. (NIST Special Publication 800-53 Revision 5)
• Major architecture shifts (identity redesign, network segmentation changes, new external endpoints) where your “frequency” definition includes event-based triggers. (NIST Special Publication 800-53 Revision 5)
• High-risk system components: internet-facing apps/APIs, admin consoles, CI/CD pathways, identity providers, and cloud control plane configurations (when in your authorized boundary). (NIST Special Publication 800-53 Revision 5)

What you actually need to do (step-by-step)

1) Define scope in a way an assessor can test

Create a written Penetration Test Scope Statement that maps to your system inventory and authorization boundary:

• In-scope environments (prod, staging, specific enclaves)
• In-scope attack surfaces (web apps, APIs, VPNs, SSO, cloud management plane where applicable)
• In-scope system components (critical microservices, containers, serverless functions, IAM roles/policies if testable)
• Explicit exclusions and the reason (for example, “owned/operated by a third party; covered by separate attestation and compensating monitoring”), with approval. (NIST Special Publication 800-53 Revision 5)

Operator tip: If you can’t point to a boundary diagram and asset list and say “these are the things we test,” your CA-8 scope is not mature.

2) Define “organization-defined frequency” with decision logic

Write a Penetration Testing Standard that sets:

• Baseline cadence (time-based)
• Event-based triggers (change-based), if you choose to include them in “frequency”
• Conditions for out-of-cycle testing (critical vulnerability exposure, new external endpoint, major authentication change)
• Who approves deviations and how they are documented. (NIST Special Publication 800-53 Revision 5)

Keep it defensible: tie it to risk, architecture complexity, and change rate. CA-8 gives you discretion, but assessors will expect your definition to be consistent with your environment and followed in practice. (NIST Special Publication 800-53 Revision 5)

3) Set rules of engagement (ROE) and safety controls

Produce a Rules of Engagement document for each test (or a master ROE with test-specific addendum):

• Allowed techniques (auth vs unauth testing, social engineering allowed or prohibited, DDoS prohibited unless explicitly approved)
• Testing windows and stop conditions
• Target lists and IP ranges
• Data handling requirements (no customer data exfiltration; evidence handling; secure transfer)
• Contacts and escalation paths (SOC, system owners, third-party hosting contacts)
• Required written authorization to test. (NIST Special Publication 800-53 Revision 5)

4) Select qualified testers and manage independence

CA-8 does not dictate who performs the test, but you should document:

• Tester qualifications and experience relevant to your stack
• Independence model (internal red team with separation of duties, or external firm)
• Conflict management and approval. (NIST Special Publication 800-53 Revision 5)

Practical reality: If engineering “tests themselves” with no separation and no formal methodology, the work may be treated as vulnerability scanning, not penetration testing, during review.

5) Execute the test and preserve raw evidence

During execution, ensure you collect:

• Test plan and methodology used
• Proofs-of-concept (sanitized), screenshots, logs, command output as appropriate
• Clear mapping from finding → affected asset → exploit path → impact. (NIST Special Publication 800-53 Revision 5)

If you use a third party, require deliverables that include scope confirmation, dates performed, and a finding severity approach that your teams can act on.

6) Triage findings and drive remediation with SLAs you control

Operationalize a Pen Test Findings Workflow:

• Intake: create tickets in your system of record (Jira/ServiceNow/etc.)
• Ownership: assign each finding to a system owner with a security partner
• Decision: fix, mitigate, accept risk (with formal approval), or document false positive
• Retest: verify closure for material findings and document evidence. (NIST Special Publication 800-53 Revision 5)

CA-8 is about conducting tests, but auditors will quickly pivot to whether the program improves security outcomes. Unremediated findings without documented risk decisions are a common control failure pattern. (NIST Special Publication 800-53 Revision 5)

7) Report, attest completion, and feed continuous improvement

Create a Penetration Test Report package that includes:

• Executive summary for governance
• Full technical details for engineering
• Scope statement and dates
• Summary of remediation status and retest results
• Lessons learned and program improvements (scope changes, new test cases). (NIST Special Publication 800-53 Revision 5)

8) Track third-party dependencies inside your boundary

Where critical components are delivered by a third party (managed identity, WAF, CI/CD service, logging pipeline), document:

• Whether the component is in the authorized boundary
• What pen test coverage is feasible
• What compensating assurance exists (contractual right-to-audit, third-party test reports you can obtain, strong monitoring)
• How you test your integration points (misconfigurations, authZ boundaries, exposed APIs). (NIST Special Publication 800-53 Revision 5)

If you manage third-party due diligence in Daydream, link the dependency record to your CA-8 scope decision and attach the relevant assurance artifacts so assessors can see the end-to-end rationale in one place.

Required evidence and artifacts to retain

Keep these in a controlled repository with versioning and access controls:

1. Penetration Testing Policy/Standard defining frequency and scope approach. (NIST Special Publication 800-53 Revision 5)
2. System/component scope list mapped to inventory and boundary. (NIST Special Publication 800-53 Revision 5)
3. Rules of Engagement + written authorization to test. (NIST Special Publication 800-53 Revision 5)
4. Test plan and methodology summary (internal or third-party). (NIST Special Publication 800-53 Revision 5)
5. Final report with findings, impacted assets, and evidence. (NIST Special Publication 800-53 Revision 5)
6. Remediation tickets with ownership, decisions, and closure notes. (NIST Special Publication 800-53 Revision 5)
7. Retest evidence (tester attestation or internal validation) for key findings. (NIST Special Publication 800-53 Revision 5)
8. Exception/risk acceptance approvals for deferred items, with compensating controls. (NIST Special Publication 800-53 Revision 5)

Common exam/audit questions and hangups

Assessors tend to ask:

• “Show me your organization-defined frequency. Where is it documented, and where is the proof you met it?” (NIST Special Publication 800-53 Revision 5)
• “Which systems/components are in scope, and how did you decide?” (NIST Special Publication 800-53 Revision 5)
• “Do your tests cover internet-facing entry points and privileged paths?” (NIST Special Publication 800-53 Revision 5)
• “How do you ensure testing is authorized and safe in production?” (NIST Special Publication 800-53 Revision 5)
• “What happened to the highest-risk findings from the last test?” (NIST Special Publication 800-53 Revision 5)

Hangups that trigger deeper scrutiny:

• Testing reports that don’t match the authorized boundary list. (NIST Special Publication 800-53 Revision 5)
• Findings closed without evidence or without retest. (NIST Special Publication 800-53 Revision 5)
• Pen test activity confused with automated vulnerability scanning outputs. (NIST Special Publication 800-53 Revision 5)

Frequent implementation mistakes (and how to avoid them)

Mistake	Why it fails CA-8	Fix
“Frequency” is tribal knowledge	You can’t prove compliance with an undefined cadence	Publish a standard that defines cadence and triggers, then track execution against it. (NIST Special Publication 800-53 Revision 5)
Scope is “prod” with no asset list	Assessors can’t confirm coverage	Tie scope to inventory and boundary artifacts; list apps/APIs/components explicitly. (NIST Special Publication 800-53 Revision 5)
No written authorization / ROE	Operational and legal risk, plus weak governance	Require ROE + authorization for each engagement; keep it with the report. (NIST Special Publication 800-53 Revision 5)
Fixes aren’t tracked to closure	Control looks performative	Enforce ticketing, ownership, and retest evidence for material findings. (NIST Special Publication 800-53 Revision 5)
Third-party components ignored	Hidden exposure; scope decisions look arbitrary	Document boundary decisions and third-party assurance; test integrations you control. (NIST Special Publication 800-53 Revision 5)

Risk implications (why operators treat this as high-stakes)

Penetration testing is one of the few controls that demonstrates how a real attacker can chain weaknesses across identity, configuration, and application logic. Under FedRAMP Moderate, weak CA-8 execution increases the chance that systemic issues persist across releases, and it raises assessment risk because the control’s evidence is concrete and easy to challenge: either you tested the defined scope on the defined cadence, or you did not. (NIST Special Publication 800-53 Revision 5)

Practical execution plan (30/60/90)

Because CA-8 does not prescribe exact deadlines, use phases that drive you to a defensible steady state without committing to unsupported timing claims. (NIST Special Publication 800-53 Revision 5)

First 30 days (stabilize governance)

• Publish the penetration testing standard defining scope method and frequency. (NIST Special Publication 800-53 Revision 5)
• Produce the initial in-scope system/component list aligned to your authorization boundary. (NIST Special Publication 800-53 Revision 5)
• Create ROE and authorization templates, plus a report template that forces scope confirmation and dates. (NIST Special Publication 800-53 Revision 5)
• Stand up the findings workflow in your ticketing system (required fields: asset, severity, owner, due date, decision, retest evidence). (NIST Special Publication 800-53 Revision 5)

Days 31–60 (execute and prove)

• Run a penetration test against the highest-risk entry points in your defined scope (often external interfaces and privileged paths). (NIST Special Publication 800-53 Revision 5)
• Triage findings and open remediation tickets with named owners. (NIST Special Publication 800-53 Revision 5)
• Build the audit binder: standard, scope list, ROE, authorization, report, and ticket export. (NIST Special Publication 800-53 Revision 5)

Days 61–90 (close the loop)

• Retest material fixes and collect closure evidence. (NIST Special Publication 800-53 Revision 5)
• Run a governance review: what was out of scope, what should be added next cycle, and what triggers out-of-cycle testing. (NIST Special Publication 800-53 Revision 5)
• If you manage third parties, link critical third-party dependencies to your scope decisions and store their assurance artifacts alongside CA-8 evidence in Daydream for faster assessments and cleaner traceability. (NIST Special Publication 800-53 Revision 5)

Frequently Asked Questions

What counts as “penetration testing” versus vulnerability scanning?

CA-8 calls for penetration testing, which is attacker-style validation, not just automated scanning outputs. In practice you should be able to show exploit paths, impact, and how defenses held or failed, with a formal scope and ROE. (NIST Special Publication 800-53 Revision 5)

Do we have to test every system in the boundary every time?

CA-8 allows you to define which systems/components are tested and how often, so you can scope based on risk and architecture. Document the rationale and keep it consistent with your inventory and boundary artifacts. (NIST Special Publication 800-53 Revision 5)

How do we define “organization-defined frequency” without getting second-guessed?

Write the frequency into an approved standard and tie it to your change rate and risk profile. Then prove execution against that definition with dated test reports and completion attestations. (NIST Special Publication 800-53 Revision 5)

Can an internal team perform the penetration test?

CA-8 does not prohibit internal testing, but auditors will expect clear authorization, a defined methodology, and credible independence controls such as separation of duties and governance oversight. Keep evidence of qualifications and approvals. (NIST Special Publication 800-53 Revision 5)

What evidence is most likely to be requested in an assessment?

Expect requests for your documented frequency and scope definitions, ROE/authorization, the final report, and proof that findings were remediated or formally accepted with approvals. Retest evidence for key issues is a common follow-up. (NIST Special Publication 800-53 Revision 5)

How should we handle penetration testing for third-party-managed components?

Document whether the component is in your authorized boundary and what testing is feasible, then collect third-party assurance artifacts and test your integration points. Track the dependency and evidence in your third-party risk workflow so the rationale is easy to produce during review. (NIST Special Publication 800-53 Revision 5)