MAP-3.4: Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant technical standards and certifications – are defined, assessed, and documented.

To meet MAP-3.4, you must run a documented proficiency program for everyone who builds, operates, monitors, or relies on AI outputs, covering AI performance, trustworthiness, and any relevant technical standards or certifications. Define role-based competencies, assess them before access and periodically, and retain auditable evidence of training, testing, and authorization decisions. ¹

Key takeaways:

• Define role-based competency requirements tied to each AI system’s risks, not generic “AI training.” ¹
• Assess proficiency (knowledge checks, practical exercises, sign-offs), then document access/authorization decisions based on results. ¹
• Keep evidence that stands up in an exam: training content, attendance, assessments, remediation, and certification status where applicable. ¹

MAP-3.4 is an operational requirement disguised as a “training” statement: you need a repeatable way to prove that operators and practitioners are competent to run an AI system safely, interpret its performance, and recognize when trustworthiness is degrading. That means role-based skill definitions (what someone must know), assessments (how you confirm they know it), and documentation (what you retain to prove it happened). ¹

This requirement applies equally to internal teams (data science, ML engineering, product, security, IT operations, customer support) and to third parties who meaningfully influence model performance or decisions (managed service providers, AI platform providers, labeling firms, model evaluators). Your auditors and regulators will focus on whether your program is specific to the AI system’s context and risk, and whether access to operate or override the system is controlled by demonstrated proficiency rather than job title. ¹

Operationalizing MAP-3.4 quickly comes down to three moves: build a competency matrix by role, run a lightweight but defensible assessment process, and attach the outputs to identity/access management, change management, and incident response so it becomes “how you run AI,” not a one-off course. ²

Regulatory text

NIST excerpt (MAP-3.4): “Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant technical standards and certifications – are defined, assessed, and documented.” ¹

What the operator must do in practice:
You must (1) define what “proficient” means for each role that can affect AI outcomes, (2) assess whether people meet that bar before they perform safety- or risk-relevant work, and (3) document both the process and the results. Where technical standards or certifications are relevant to the AI system’s use case, you must specify them and track completion/validity as part of proficiency evidence. ¹

Plain-English interpretation

MAP-3.4 requires a role-based competency and authorization program for AI. “Operator” includes people who run or supervise the AI in production (monitoring, thresholds, rollbacks, overrides). “Practitioner” includes people who design, build, evaluate, tune, validate, or materially change the system (data, model, prompts, RAG, guardrails, pipelines). “Performance and trustworthiness” means they can interpret metrics and signals that indicate risk: drift, error patterns, bias or harmful outputs, security misuse, and control failures, based on your own risk assessment and deployment context. ¹

Who it applies to (entity and operational context)

Applies to: Organizations developing or deploying AI systems. ¹

Operational contexts where MAP-3.4 becomes exam-relevant fast:

• High-impact decisioning (eligibility, pricing, underwriting, hiring, healthcare triage, fraud actions) where incorrect outputs create legal, financial, or safety risk.
• Human-in-the-loop workflows where staff must interpret model confidence, handle edge cases, and escalate correctly.
• Model monitoring / MLOps where operators decide when to rollback, retrain, disable features, or change thresholds.
• Third party-supported AI where external teams maintain models, data pipelines, or evaluation tooling.

Typical in-scope roles (map them to your org chart):

• Model developers (ML engineers, data scientists)
• Evaluators/validators (model risk, QA, red team)
• Production operators (SRE, MLOps, IT ops)
• Product owners and risk owners (business accountable owners)
• Customer operations and reviewers (case workers, call center leads)
• Security and privacy (who approve controls and respond to incidents)
• Third party operators with admin access or evaluation duties

What you actually need to do (step-by-step)

Step 1: Inventory AI systems and identify “proficiency-impacting” roles

Start with your AI inventory (even a spreadsheet) and tag each system by:

• Use case and decision impact
• Who can change model behavior (code, prompts, rules, data)
• Who can override outputs or approve exceptions
• Who monitors and responds to incidents
  Then produce a short list of roles per system. ¹

Deliverable: “AI System Proficiency Scope” table: system → roles → why role is in scope → required proficiency tier.

Step 2: Define proficiency requirements by role (competency matrix)

Create a matrix that converts MAP-3.4 into requirements people can meet. Keep it tight and job-relevant.

Example competency categories (edit to match your risk profile):

• System purpose & boundaries: intended use, prohibited uses, known failure modes
• Performance interpretation: what metrics are tracked, how to read dashboards, what “good” looks like for this system
• Trustworthiness factors: bias/harm signals relevant to domain, explainability expectations, robustness expectations, security misuse patterns
• Controls & operating procedures: escalation paths, override rules, logging obligations, change control requirements
• Data handling: data quality checks, labeling guidance, privacy/security constraints
• Standards/certifications: any required internal certifications or external credentials you choose to adopt for risk roles (define applicability and equivalencies)

Your matrix should specify: competency statement, target role(s), required level (basic/advanced/admin), assessment method, and renewal trigger (e.g., “after material model change,” “after incident,” “annual”). ¹

Deliverable: “AI Proficiency Competency Matrix” approved by the AI risk owner and the control owner (often GRC, Model Risk, or Security).

Step 3: Build the assessment process (not just training)

Auditors will ask how you verify proficiency. Use at least two assessment types for higher-risk roles:

• Knowledge check: short quiz on policies, boundaries, and escalation rules
• Practical exercise: interpret a monitoring dashboard, triage example outputs, perform an incident tabletop, or execute a rollback runbook step
• Peer/manager sign-off: attestation that the person can perform the role
• Access gating: only grant production permissions after passing

Define pass/fail criteria, remediation steps, and retest rules. If you choose not to fail people (common in early programs), require remediation and document that remediation occurred before granting access. ¹

Deliverables: Assessment templates, scoring rubrics, sign-off form, remediation plan template.

Step 4: Tie proficiency to IAM, SDLC, and change management

MAP-3.4 becomes real when proficiency changes what people can do.

• IAM: map proficiency tiers to access groups (read-only dashboards vs admin overrides vs model deployment)
• SDLC/MLOps: require completion of role-based proficiency before approving model releases, prompt changes, or monitoring rule edits
• Incident response: include “qualified responder” requirements for AI incidents; track who is on-call and whether they are current

This alignment is what turns “training records” into “operational control.” ¹

Deliverable: RACI plus control mapping: MAP-3.4 → policy → procedure → control owner → systems of record (LMS, GRC, IAM) → evidence cadence. ¹

Step 5: Document the program and run it on a cadence

Write a short procedure that states:

• In-scope roles and systems
• Required competencies and how they are maintained
• Assessment methods and frequency triggers
• How exceptions are handled (temporary access, emergency changes)
• Evidence retention expectations
  Then run onboarding and recurring cycles, and report completion and exceptions to your AI governance forum. ²

Required evidence and artifacts to retain

Keep artifacts in a system auditors can access without heroics (LMS + GRC ticketing + IAM exports).

Minimum evidence set ³:

• Proficiency policy/procedure for AI operators/practitioners ¹
• Role-to-competency matrix with version history and approvals ¹
• Training materials (slides, runbooks, labs, tabletop scenarios) and change log
• Assessment artifacts: quizzes, lab exercises, scoring rubrics, results
• Attendance/completion records (including third party participants where feasible)
• Access gating evidence: IAM group membership rules, approval tickets, access reviews tied to proficiency
• Exceptions register: who got temporary access, why, compensating controls, expiration
• Remediation documentation: assigned training, retest outcomes, manager sign-off
• Certifications tracking (if you require any): certificate copies, validity dates, equivalency decisions ¹

Common exam/audit questions and hangups

What auditors ask most often:

1. “Who is an operator/practitioner for this system, and how did you decide?” ¹
2. “Show me how proficiency is assessed, not just trained.” ¹
3. “Does passing proficiency change access or responsibilities?” ¹
4. “How do you keep proficiency current after model changes or incidents?” ¹
5. “How do you cover third parties who run parts of the AI lifecycle?” ¹

Hangups that stall programs:

• Waiting for a perfect enterprise AI policy before defining role-based competencies.
• Treating proficiency as “everybody takes an AI ethics course,” which does not show operational readiness for a specific system.
• No system of record. If results live in email, you will fail the “documented” part under pressure.

Frequent implementation mistakes and how to avoid them

Mistake	Why it fails MAP-3.4	Fix
One generic AI training for all staff	Doesn’t show role-specific proficiency with performance and trustworthiness	Create tiers by role (viewer, operator, admin, developer) and assess differently 1
No practical assessment	You can’t show people can interpret metrics or follow runbooks	Add a short lab/tabletop for high-risk roles; store results 1
Proficiency not tied to access	Training becomes “FYI,” not a control	Gate admin/override/deploy permissions on completion and sign-off 1
No trigger after changes	Teams become “certified” on an old model	Reassess after material model/prompt/data/monitoring changes 1
Ignoring third parties	External operators can cause the same failures	Add contract requirements and collect attestations/completion evidence 1

Enforcement context and risk implications

No public enforcement cases were provided in the source catalog for MAP-3.4. From a risk standpoint, weak proficiency controls tend to surface as operational incidents: misinterpreted monitoring, delayed escalation, improper overrides, undocumented changes, and preventable harmful outputs. Those incidents then create regulatory, contractual, and litigation exposure depending on your sector and use case. Your best defense is a clean evidence trail that shows you defined proficiency expectations, verified them, and restricted sensitive actions to qualified personnel. ¹

Practical 30/60/90-day execution plan

Use this as an execution sequence; adjust scope based on which AI systems are highest risk. ¹

First 30 days (stabilize and scope)

• Name a control owner and backups for MAP-3.4, and define systems of record for evidence. ¹
• Pick the first AI system(s) to operationalize based on impact and production criticality. ¹
• Draft the role inventory and initial competency matrix for those systems. ¹
• Publish a short interim procedure: who needs proficiency, how it’s assessed, and how exceptions work. ¹

Days 31–60 (assess and gate)

• Build assessments (quiz + practical exercise) and scoring rubrics for in-scope roles. ¹
• Run assessments for current operators/practitioners; document remediation for gaps. ¹
• Implement access gating for the most sensitive permissions (deploy, rollback, override, threshold changes). ¹
• Stand up an exceptions register and begin tracking temporary access. ¹

Days 61–90 (operationalize and expand)

• Integrate proficiency checks into change management for model/prompt/data updates. ¹
• Add proficiency requirements to third party onboarding and contract renewals where third parties operate or maintain AI. ¹
• Expand to additional AI systems using the same templates; report status to governance. ²
• If you use Daydream for GRC workflows, map MAP-3.4 to a control, assign an owner, and schedule recurring evidence collection so audits pull from one place. ¹

Frequently Asked Questions

Does MAP-3.4 require formal external certifications for AI staff?

MAP-3.4 requires you to define “relevant technical standards and certifications” where they matter to your system and roles. If external certifications are not relevant, document your rationale and rely on internal competency definitions and assessments. ¹

Who counts as an “operator” versus a “practitioner”?

Operators run or supervise the AI in production and make live decisions like overrides, escalations, and rollbacks. Practitioners design, build, validate, or materially change the system, including data, prompts, models, and monitoring rules. ¹

We use a third party model/API. Are we still responsible for MAP-3.4?

Yes for your organization’s operators and practitioners, and for any third party personnel performing in-scope operational tasks for you. Your contract should require appropriate proficiency evidence or attestations, and you should retain what you can for audit. ¹

What’s the minimum “assessment” that will satisfy an auditor?

A tracked knowledge check plus a documented sign-off may be acceptable for lower-risk roles, but higher-risk roles usually need a practical exercise tied to your runbooks and monitoring. The key is written criteria, recorded results, and clear authorization decisions based on outcomes. ¹

How do we keep proficiency current when models change frequently?

Define reassessment triggers tied to “material changes” and incidents, and embed them into change management tickets. Update the competency matrix and require affected roles to complete delta training or a shortened reassessment. ¹

How should we document exceptions for emergency access?

Use a time-bound exception with an approver, a reason, compensating controls (extra logging or peer review), and a documented removal date. Retain the ticket and proof access was removed, then schedule remediation for the individual. ¹

Footnotes

1. NIST AI RMF Core

2. NIST AI RMF Core; Source: NIST AI RMF program page

3. AI system or shared program, as applicable