Processes for responsible AI system design and development

ISO/IEC 42001 Annex A Control A.6.1.3 requires you to define and run documented, repeatable processes that make “responsible AI” real during design and development, not after release. Operationally, you need lifecycle gates, risk-based design controls, testing protocols, and traceable evidence that teams followed them for each AI system. ¹

Key takeaways:

• Build a governed SDLC-style process for AI with defined roles, stage gates, and “stop/go” criteria. ¹
• Tie design and development work to explicit risk controls (data, model, evaluation, human oversight, deployment constraints) and keep end-to-end traceability. ¹
• Retain artifacts that prove the process ran: requirements, risk assessments, test results, approvals, and change records per system release. ¹

“Processes for responsible AI system design and development” is a deceptively short requirement that auditors interpret as: show me your AI engineering playbook, show me that teams follow it, and show me the evidence for each system and release. Control A.6.1.3 sits in ISO/IEC 42001 as a management-system requirement, so the expectation is not a one-off checklist. You need operational processes that are embedded into how products are built and changed. ¹

For a Compliance Officer, CCO, or GRC lead, the fastest way to operationalize A.6.1.3 is to treat it like an AI-specific SDLC with governance hooks: intake, design requirements, data and model controls, evaluation, approvals, release, monitoring, and change management. Then require teams (including third parties where relevant) to produce standard artifacts at each gate. ¹

This page provides requirement-level implementation guidance: who it applies to, what to build, how to run it, what evidence to retain, and where audits usually stall.

Regulatory text

Requirement: “The organization shall establish and implement processes for responsible AI system design and development.” (Annex A, Control A.6.1.3) ¹

Operator interpretation: You must (1) define documented processes that translate “responsible AI” expectations into engineering activities and decision points, and (2) demonstrate those processes operate in practice across AI initiatives, including changes after initial release. Auditors will look for consistency, accountability, and traceability from requirements through deployment. ¹

Plain-English meaning (what this requires)

You need a repeatable way to design and build AI systems that:

• Identifies and manages risks early (before production). ¹
• Sets clear constraints and acceptance criteria for data, models, and outputs. ¹
• Produces evidence that decisions were made deliberately, approved by accountable owners, and revisited when systems change. ¹

In practice, “responsible AI” is not a slogan. It becomes:

• Documented requirements (what the system may and may not do).
• Controlled inputs (data provenance and quality expectations).
• Defined evaluation (what “good enough” means, plus safety/fairness/robustness checks appropriate to your context).
• Human oversight and fallback paths where needed.
• Change control for retraining, model swaps, prompt changes, and feature updates. ¹

Who it applies to

ISO/IEC 42001 states applicability broadly to organizations involved with AI, including:

• AI providers building AI capabilities into products/services.
• AI users deploying AI in internal operations (HR, fraud, customer support, IT operations, content workflows).
• Organizations that manage AI development directly or through third parties. ¹

Operational contexts that fall under this control:

• Custom model development and training.
• Fine-tuning or adaptation of a foundation model.
• Use of third-party AI APIs where you still design the system behavior (prompting, tool use, retrieval augmentation, guardrails).
• Any material change that affects outputs or risk posture (new datasets, new model versions, changed decision thresholds, new use cases). ¹

What you actually need to do (step-by-step)

Treat this as building an “AI development lifecycle” that plugs into existing SDLC, security, privacy, and model risk practices. The steps below are ordered to get you audit-ready quickly while still being usable by engineering teams.

1) Define scope and a standard AI system record

Create a lightweight AI System Profile template and require it at intake. Minimum fields:

• Intended purpose and prohibited uses.
• Users/impacted parties and operational environment.
• High-level architecture (data sources, model type/provider, downstream systems).
• Decision impact level (advisory vs automated action).
• Human oversight model (who reviews, when, and how overrides work). ¹

Why auditors care: This profile becomes your anchor for traceability and helps prove you run a consistent process. ¹

2) Establish lifecycle gates with accountable approvers

Define stage gates and who can approve progress. A practical set:

• Gate A: Use-case approval (business owner + compliance/privacy/security review as needed)
• Gate B: Design approval (architecture + risk controls selected)
• Gate C: Pre-release approval (testing completed, residual risk accepted)
• Gate D: Change approval (for retraining, model/provider swap, major prompt or policy changes)
• Gate E: Retirement approval (decommission plan, data/model disposal rules) ¹

Include “stop/go” criteria per gate. Keep them risk-based: higher impact use cases require stronger evidence and more approvals. ¹

3) Convert “responsible AI” into engineering requirements

Publish a Responsible AI Design Standard that engineering can implement. It should cover:

• Data requirements (source legitimacy, quality checks, data minimization where applicable, labeling rules).
• Model requirements (reproducibility expectations, versioning, dependency control).
• Safety requirements (content restrictions, refusal behavior, secure-by-design principles for AI endpoints).
• Output requirements (explainability expectations where applicable, confidence signaling, escalation paths).
• Human oversight requirements (review workflows, override logging, training for reviewers). ¹

This is where many programs fail: they publish ethics principles but not buildable requirements. Your standard must translate into test cases and acceptance criteria. ¹

4) Implement risk assessment embedded in design

Require a documented AI Risk Assessment at design time. It should be tied to the system profile and include:

• Foreseeable misuse and abuse cases.
• Impact analysis for errors (operational, legal/regulatory, customer harm, safety).
• Third-party risks (model provider terms, data transfer, subcontractors).
• Control selection and residual risk acceptance decision. ¹

Operational tip: Don’t let risk assessment become a separate GRC “paper exercise.” Make it the artifact that drives design decisions (guardrails, monitoring, human-in-the-loop). ¹

5) Standardize testing: functional, safety, and robustness

Define a minimum test suite that scales with system risk. Examples of categories to require:

• Functional tests (requirements met, known limitations documented).
• Data tests (drift checks, bias/imbalance flags where relevant, leakage checks).
• Model behavior tests (stress tests, adversarial prompts if using generative AI, regression tests across model versions).
• Security tests (prompt injection defenses where relevant, access control tests, logging/telemetry validation).
• Human oversight tests (review queues, escalation paths, override works as designed). ¹

Keep the test plan tied to acceptance criteria in the design standard. Retain results per release. ¹

6) Control changes after release

Your process must cover the reality that AI changes frequently. Implement:

• Model and prompt/version control (what changed, why, who approved).
• Retraining governance (data change approvals, training run records, evaluation comparisons).
• Release notes for AI behavior changes that could affect users or operations.
• Rollback plans and kill switches where appropriate. ¹

7) Extend the process to third parties

If a third party provides the model, data, or major components, your process still must work. Require:

• Due diligence inputs (documentation, testing evidence, known limitations).
• Contractual requirements for change notice, incident notice, and evaluation support.
• Internal validation of third-party model behavior in your use case (don’t accept generic claims). ¹

Where Daydream fits: Daydream can centralize third-party due diligence workflows, intake questionnaires, and evidence collection for AI providers, then link those artifacts to your AI system records so audits don’t stall on missing supplier proof.

Required evidence and artifacts to retain

Auditors typically ask for “show me” evidence per AI system and per major release. Maintain:

• AI System Profile (current version + history).
• Responsible AI Design Standard (policy/standard) and any engineering procedures.
• AI Risk Assessment, with control mapping and residual risk sign-off.
• Data documentation (sources, approvals, quality checks, labeling guidance).
• Model documentation (versioning, training runs if applicable, configuration, dependencies).
• Test plan and test results (including regression tests and sign-offs).
• Gate approvals (who approved, date/time, conditions, exceptions).
• Change logs (model swaps, retraining, prompt/guardrail changes).
• Monitoring and incident records tied back to requirements (what triggered, what changed). ¹

Common exam/audit questions and hangups

Expect these questions and prepare crisp evidence:

• “Show the process document. Where is it required in your SDLC?” ¹
• “Pick one AI system. Walk me from intake to release with artifacts and approvals.” ¹
• “How do you decide what testing is required?” (Risk-based criteria and defined baselines.) ¹
• “How do you control changes to model behavior after launch?” ¹
• “How do third-party models fit into your process?” ¹

Hangup: Teams often cannot produce artifacts for older releases or emergency changes. Fix this with a simple release checklist and a single evidence repository. ¹

Frequent implementation mistakes (and how to avoid them)

1. Principles without procedures.
   Fix: Convert principles into required fields, gate checklists, and testable requirements. ¹

2. No clear ownership.
   Fix: Assign a business owner, technical owner, and risk approver per system; define who can accept residual risk. ¹

3. Treating generative AI as “just another SaaS.”
   Fix: Add prompt/guardrail versioning, jailbreak testing, and output monitoring; document intended/prohibited uses. ¹

4. Weak change control for retraining and model upgrades.
   Fix: Require re-approval triggers (dataset change, model version change, new use case, expanded user base). ¹

5. Evidence scattered across tools.
   Fix: Define one “system of record” for AI governance artifacts, even if engineering work stays in Jira/Git. Daydream can serve as the evidence spine for third-party and internal approvals.

Enforcement context and risk implications

ISO/IEC 42001 is a standard, not a regulator. Your practical risk is: if you cannot prove disciplined design and development processes, you increase the chance of unsafe outputs, operational incidents, customer harm, and failed audits against your own commitments or customer requirements. Control A.6.1.3 is commonly used by customers and assessors as a proxy for whether “responsible AI” is real inside engineering. ¹

A practical 30/60/90-day execution plan (phased)

First phase (immediate)

• Inventory AI systems in scope and create the AI System Profile template.
• Define lifecycle gates and approvers; publish a one-page gate checklist.
• Stand up a centralized evidence location and naming convention for artifacts. ¹

Second phase (near-term)

• Publish the Responsible AI Design Standard with minimum required tests and acceptance criteria.
• Pilot the process on one high-visibility AI system and run it end-to-end, including change control.
• Update third-party intake to request model documentation, change notice terms, and testing evidence. ¹

Third phase (operationalize)

• Roll the process into SDLC tooling (Jira templates, PR checks, release approvals).
• Train engineering, product, and review teams on gates, artifacts, and escalation paths.
• Establish a recurring governance rhythm to review exceptions, incidents, and post-release changes across systems. ¹

Frequently Asked Questions

Does A.6.1.3 require a separate “Responsible AI SDLC,” or can we extend our existing SDLC?

Extending your existing SDLC is usually the fastest path, as long as you add AI-specific gates, testing, and evidence. The key is showing a defined process and proof it runs for AI work. ¹

We only use third-party AI APIs. Do we still need design and development processes?

Yes. You still design the system behavior, configure guardrails, select data sources, and decide where outputs go. Your process should cover selection, validation in your use case, and ongoing change control. ¹

What’s the minimum evidence to pass an audit for one AI system?

Auditors typically expect a system record, risk assessment, test results tied to acceptance criteria, and documented approvals at defined gates. If you cannot recreate decisions and changes, you will struggle to demonstrate implementation. ¹

How do we handle rapid model updates without paralyzing releases?

Define change triggers that require full re-approval versus lightweight review, and standardize regression tests. Keep versioned release notes so approvals track what changed and why. ¹

Who should be the accountable approver for “responsible AI” risk?

Assign a business owner who accepts operational outcomes, plus a designated risk approver who can accept residual risk under your governance model. Document the decision rights in your gate process. ¹

How should we document “ethical considerations” without creating vague paperwork?

Tie ethical considerations to specific, testable requirements for the use case, then record trade-offs and residual risk decisions. Keep it concrete: intended use, prohibited use, known limitations, and controls. ¹

Footnotes

1. ISO/IEC 42001:2023 Artificial intelligence — Management system