SA-15(1): Quality Metrics

SA-15(1): Quality Metrics requires you to contractually require the system (or component/service) developer to define, collect, and report objective software quality metrics, then use those metrics to accept, reject, and remediate deliverables. To operationalize it fast, set a minimum metric set, bake it into SOW/SDLC gates, and retain repeatable evidence that metrics were produced, reviewed, and acted on. ¹

Key takeaways:

• You must make quality measurable and enforce it through developer obligations, not informal expectations. ¹
• Auditors look for two things: defined metrics and proof you used them to make decisions. ²
• Evidence has to be recurring ¹, not a one-time dashboard screenshot. ²

SA-15 sits in the System and Services Acquisition family, which means the control intent is procurement-and-development discipline: you don’t “hope” software quality is good, you require it from whoever builds the system, component, or service. SA-15(1): Quality Metrics narrows that into a measurable obligation: the developer must produce quality metrics, and you must be able to show that those metrics exist, are reviewed, and drive action when thresholds are not met. ¹

For a Compliance Officer, CCO, or GRC lead, the fastest path is to translate “quality metrics” into (1) a defined metric catalog with thresholds, (2) contract language that compels measurement and reporting, and (3) SDLC/acceptance gates tied to those thresholds. This requirement becomes high-friction only when teams treat it as a generic “quality” statement rather than a procurement-enforceable, testable condition of delivery. ²

This page gives requirement-level implementation guidance you can drop into third-party development, internal engineering, or mixed delivery models (COTS + custom). It is written to help you walk into an assessment and show a clean line from requirement, to procedure, to recurring evidence. ²

sa-15(1): quality metrics requirement — plain-English meaning

SA-15(1) expects you to require the developer (internal dev team or third-party developer) to produce objective quality metrics for the system/component/service. Those metrics must be specific enough that you can evaluate deliverables, spot quality regressions, and trigger remediation before production release. ¹

Operationally, that means:

• You define what “quality” means in measurable terms for your environment.
• The developer measures it consistently and reports it on a schedule tied to delivery.
• You review the results and document decisions (acceptance, conditional acceptance with remediation, rejection, exception/risk acceptance). ²

Regulatory text

Excerpt: “Require the developer of the system, system component, or system service to:” ¹

Operator interpretation: The requirement is framed as a procurement/engineering management obligation. Your control design should show (1) the mechanism you use to impose the requirement on the developer (contract clauses, SOW language, internal SDLC policy), and (2) the operational workflow that proves the developer actually supplies quality metrics and you act on them. ²

Because the excerpt provided is incomplete, keep your implementation conservative: focus on the enforceable outcome the enhancement name implies—quality metrics are required and used—and document your chosen metric set and review cadence as part of your system development and acquisition governance. ²

Who it applies to

Entity scope

• Federal information systems and programs implementing NIST SP 800-53 controls. ²
• Contractors and other third parties building or operating systems that handle federal data and inherit 800-53 obligations through contract. ²

Operational scope

• Custom software development (internal teams, outsourced development, staff augmentation).
• System components and services delivered through CI/CD (microservices, APIs, infrastructure-as-code, container images).
• Major configuration changes and version upgrades where quality can regress (patches, platform migrations). ²

What you actually need to do (step-by-step)

1) Name the control owner and decision rights

Assign an owner who can enforce gates (often AppSec/Engineering Excellence with GRC oversight). Define who can:

• set metric thresholds,
• approve exceptions,
• block releases based on quality failures. ²

Practical tip: If the control owner cannot block acceptance, your metrics become “reporting only,” which auditors frequently treat as weak operating effectiveness.

2) Define a minimum quality metric catalog (with thresholds)

Create a short, auditable baseline that fits most systems. Common categories you can standardize:

• Defect quality: defect density trends, escaped defects, severity mix.
• Test quality: unit/integration test pass rate, coverage expectations, flaky test rate.
• Security quality (security-as-quality): SAST/DAST findings by severity, dependency vulnerability counts by severity, remediation SLAs.
• Reliability quality: build stability, change failure rate indicators, rollback frequency (if you track it).
• Maintainability: code complexity thresholds, duplication thresholds, lint/static rule compliance.

Write thresholds as “release criteria” (pass/fail or conditional). Document your rationale and allow system-specific tailoring with approval. ²

3) Put the requirement into enforceable developer obligations

For third-party developers, add contract/SOW language that:

• requires the metric set and reporting format,
• requires delivery of raw evidence (tool exports) on request,
• ties acceptance and payment milestones to meeting thresholds,
• requires remediation and re-test when thresholds are missed,
• requires toolchain transparency (what scanners/test frameworks were used and versions). ²

For internal dev teams, implement the same obligations via SDLC policy, engineering standards, and release management procedures. ²

4) Implement “quality gates” in the SDLC and acceptance workflow

Hardwire metrics into your delivery process:

• PR/MR checks (linting, SAST, unit tests)
• CI build gates (test pass, coverage, policy checks)
• Release gates (vulnerability threshold, open defect threshold, exception approval)
• Post-release monitoring gates where appropriate (for services) ²

Your evidence should show the gate exists and is used, not just configured.

5) Establish a repeatable review and exception process

Create a lightweight routine:

• Developer submits a quality metrics pack per release (or on a defined cadence).
• Reviewer signs off: accept, accept with remediation plan, reject, or request exception.
• Exceptions require documented risk acceptance with compensating controls and an expiry condition. ²

6) Make it assessable: map the requirement to procedures and recurring artifacts

Maintain a control record that ties SA-15(1) to:

• control owner,
• implementation procedure,
• recurring evidence artifacts.
  This is the difference between “we do this” and “we can prove this.” ¹

Daydream fits naturally here as the system to map SA-15(1) to an owner, a standard operating procedure, and a predictable evidence checklist that refreshes each release cycle, so the control stays audit-ready without manual chasing.

Required evidence and artifacts to retain

Keep evidence that proves definition, collection, review, and action:

Definition artifacts

• Quality metrics standard (metric catalog + thresholds)
• Tailoring records per system (if thresholds vary)
• SDLC/release policy referencing quality gates ²

Developer obligation artifacts

• Contract clauses / SOW language / internal engineering standard
• Acceptance criteria language tied to metrics
• Third-party reporting templates ²

Operational evidence (recurring)

• CI/CD gate results (build logs, pipeline run summaries, policy check outputs)
• Test reports, coverage reports, and defect summaries
• Security scan outputs (or exports) and remediation tickets
• Release/Change approval records showing metrics review
• Exception approvals with expiry and remediation plan ²

Retention tip: Store artifacts by release version/build ID to support traceability from a deployed version back to the quality metrics used for acceptance.

Common exam/audit questions and hangups

Auditors and assessors commonly ask:

• “Show me your defined quality metrics and thresholds for this system.” ²
• “Where is the developer required to provide these metrics?” (contract/SOW/policy) ²
• “Pick a recent release. Show the metric output and the approval decision.” ²
• “What happens when a threshold is not met? Show an example.” ²
• “How do you prevent teams from bypassing gates?” ²

Hangups that cause findings:

• Metrics exist but have no thresholds, so they can’t function as acceptance criteria.
• Thresholds exist but exceptions are informal (Slack/email) with no expiry or owner.
• Evidence is point-in-time, not tied to specific releases. ²

Frequent implementation mistakes (and how to avoid them)

Mistake	Why it fails	Fix
“Quality metrics” defined as subjective narrative	Not testable; doesn’t support acceptance decisions	Require tool-generated, repeatable measures plus defined thresholds 2
Metrics tracked, but no one signs off	No proof of governance	Add a release approval step that references metrics outputs 2
Security treated separately from quality	Leaves blind spots in delivery acceptance	Include security scan results in the quality metric catalog 2
Third-party dev says “trust our process”	No enforceable obligation	Put metrics, formats, and acceptance consequences in the contract/SOW 2
Exceptions become permanent	Risk accumulates silently	Require expiry, owner, and remediation plan for every exception 2

Enforcement context and risk implications

No public enforcement cases were provided in the source data for SA-15(1). Practically, the risk is assessment failure (control not implemented or not operating) and downstream security/availability exposure when low-quality code reaches production without measurable checks. Your strongest defense is a clean evidence trail: defined metrics, recurring measurement, and documented decisions tied to releases. ²

Practical 30/60/90-day execution plan

First 30 days (stand up the control)

• Assign control owner and backup; document decision rights for release blocking. ²
• Publish a baseline quality metric catalog with initial thresholds and an exception workflow. ²
• Update one active dev engagement (or one internal product) to include metric reporting and an approval record.

Day 31–60 (make it repeatable)

• Implement CI/CD quality gates for the baseline metrics on the pilot system. ²
• Add contract/SOW boilerplate for third-party developers and a review checklist for procurement and project kickoff. ²
• Start storing evidence by release/build identifier in a central repository; define naming conventions.

Day 61–90 (scale and audit-proof)

• Expand to additional systems and third-party development relationships based on criticality. ²
• Run an internal “mini-assessment”: select a release, trace metrics to approval to deployment to remediation tickets. ²
• Operationalize continuous evidence mapping (owner, procedure, recurring artifacts) in Daydream so the control stays current as teams and toolchains change. ¹

Frequently Asked Questions

What counts as a “quality metric” for SA-15(1)?

A quality metric is a defined, objective measure that can be produced repeatedly (usually from tools) and compared to a threshold for acceptance decisions. Keep a baseline set, then tailor per system with documented approval. ²

Does SA-15(1) apply to internal development teams or only third parties?

It applies to “the developer,” which can be internal engineering or a third-party developer, depending on your delivery model. The key is that the development function is required to provide metrics and you can show governance over them. ²

We buy SaaS. How do we meet SA-15(1) if we can’t see their CI pipelines?

Treat the SaaS provider as the developer of the service and require quality reporting through contractual terms, attestations, or agreed deliverable reports. If the provider cannot provide meaningful metrics, document the gap and decide whether to accept the risk or select another provider. ²

Are security scan results “quality metrics” under SA-15(1)?

They can be, and many programs include them in a quality metric catalog because exploitable defects are quality failures in operational terms. Define which scans are required and how findings affect release acceptance. ²

What evidence is strongest in an assessment?

Release-by-release artifacts that show metrics output, a documented review, and an acceptance decision (or exception) tied to a specific build/version. Tool exports plus approval records beat screenshots and narrative summaries. ²

How do we prevent teams from bypassing quality gates in emergencies?

Allow an emergency exception path, but require documented approval, compensating controls, and a follow-up remediation deadline captured as tickets. Assessors mainly care that bypass is controlled and evidenced. ²

Footnotes

1. NIST SP 800-53 Rev. 5 OSCAL JSON

2. NIST SP 800-53 Rev. 5