Why Traditional T&E Doesn't Transfer to AI Systems
Developmental test and evaluation exists to characterize system performance against requirements before operational testing. For hardware-dominant systems — an aircraft, a radar, a munition — the test regime is built around repeatability. The same input produces the same output; deviation is a defect; a test matrix captures the performance envelope. Test events are discrete, schedulable, and produce data that maps directly to system requirements.
AI and machine learning components break these assumptions at multiple layers.
Distribution shift is a systemic failure mode, not a defect. A model trained on one operational context will degrade when its deployment environment changes — different lighting, different imagery resolution, different communication latencies, different adversary behavior. That degradation typically does not appear as a hard failure; it appears as a quiet reduction in accuracy that only becomes visible in operational data. Traditional DT&E has no standard method for stress-testing a system against environments it hasn't seen. A test matrix covering the training distribution provides no coverage of the distribution it will actually encounter.
Adversarial inputs create vulnerabilities without visible signatures. Adversarial attacks on machine learning systems can cause systematic misclassification using inputs that are indistinguishable from legitimate ones to a human observer. An imagery classifier can be manipulated to consistently miss a target category through modifications that are invisible at normal resolution. Standard developmental testing does not include adversarial probing as a required test case; that omission leaves a class of failure modes uncharacterized before a system enters service.
Post-deployment learning changes the system the evaluators tested. Several AI programs — particularly those using reinforcement learning or continuous training pipelines — update model weights on operational data. The system that entered OT&E is not the system fielded six months later. Traditional T&E evaluates a fixed configuration; programs with continuous learning components need continuous evaluation, a concept the T&E instruction set did not accommodate when most current TEMPs were written.
What CDAO's April 2024 Frameworks Established
The Chief Digital and AI Office published four test and evaluation frameworks in April 2024 — covering Model T&E, Human Systems Integration T&E, Systems Integration T&E, and Operational T&E — as an initial step toward a community of practice around AI T&E. These frameworks are not binding policy; they do not amend DoDI 5000.89 or the Services' T&E manuals. What they do is define a structured vocabulary and logic for addressing the failure modes that traditional T&E misses.
The Model T&E framework organizes evaluation around six areas: performance, testing methods, data, AI models, context, and documentation. On robustness specifically, the framework calls for out-of-distribution testing and adversarial input evaluation — explicitly naming the failure modes that traditional developmental testing does not address. On documentation, it requires that programs record model lineage, training data provenance, and versioning — establishing a baseline that continuous-learning programs need before they can trace a performance change to a cause.
The Systems Integration T&E framework addresses a different and underexamined problem: AI components interact with host systems and networks in ways that compound their individual behaviors. A model that performs well in isolation may produce unstable behavior when integrated with a command node that operates under different latency assumptions, or when its outputs feed an automated decision system without a human review step. The framework requires that integration testing characterize emergent behaviors — interactions between components that were not present in any component's individual test. Program offices that test their AI model in isolation and assume integration adds no new risks have an uncharacterized failure surface at the seam.
The Human Systems Integration T&E framework addresses trust calibration: operators who over-trust AI outputs defer to the system when they should override it; operators who under-trust it create workarounds that negate the system's value. Neither outcome surfaces in a test matrix that evaluates only model accuracy. The framework requires T&E to measure whether operators understand what the AI is doing, recognize when it is wrong, and exercise appropriate override judgment. This requires human-in-the-loop testing with realistic task complexity — a test type with significant cost and schedule implications that TEMPs written before this guidance did not budget for.
The Operational T&E considerations address performance in the deployed environment rather than the test range, including whether the system degrades gracefully under degraded connectivity, whether it produces auditable outputs that support after-action review, and whether AI-driven decisions are traceable to a responsible human authority.
What MYSTIC DEPOT Reveals About the Remaining Gap
In March 2026, the Defense Innovation Unit and the Office of the Director of National Intelligence jointly released a solicitation called MYSTIC DEPOT for a vendor-agnostic AI evaluation harness. The solicitation's requirements provide a precise statement of what the April 2024 CDAO frameworks identified as necessary but did not solve at the operational level.
MYSTIC DEPOT requires the harness to support human-in-the-loop evaluation, agentic behavior testing, and automated adversarial red-teaming — including execution of adversarial prompts and attack patterns against target AI systems. It requires the harness to work across unclassified, secret, and top secret cloud environments and in air-gapped configurations, addressing the classification barrier that has historically prevented consistent evaluation methodology from crossing security domains. It includes explicit requirements for operational stress testing, simulating network degradation in reproducible conditions. And it requires "gaming resistance" evaluation — testing whether AI outputs are stable under attempts to systematically manipulate them toward a desired result.
No comparable government-owned harness exists at the program level. Programs conducting AI T&E today are building their own methodologies, with fragmented tooling and no cross-program comparability. MYSTIC DEPOT, if delivered, provides the infrastructure for what CDAO's frameworks described as required. Until it is available and integrated into the T&E ecosystem, program offices face the practical question of how to demonstrate AI T&E compliance with guidance that assumes tools and methods that aren't yet deployed.
Practical Guidance: Updating Your TEMP Now
The TEMP is the governing document for a program's T&E strategy; it specifies what will be tested, at what fidelity, by whom, and in what sequence. Programs with AI components should treat the following as decisions that need to be reflected in the TEMP before the next milestone review, regardless of whether formal DoDI amendments have been published:
1. Map AI components to the CDAO framework categories. Identify which of the four frameworks apply to each AI component — a supervised classification model in a sensor system has different T&E considerations than a reinforcement learning agent in a logistics optimizer. Document which frameworks apply and why; this creates the audit trail that milestone reviews will increasingly require.
2. Add out-of-distribution and adversarial test cases to the developmental test matrix. Define the operational environments the program anticipates and build test cases that deliberately fall outside the training distribution. Engage with the program's T&E community and the CDAO T&E frameworks on methodology. This does not require MYSTIC DEPOT; it requires test design that explicitly addresses distribution shift.
3. Include human-in-the-loop events in the TEMP schedule and budget. HSI T&E for AI requires operators, realistic task scenarios, and measurement of over-trust and under-trust indicators. These events take time and are expensive to schedule close to a milestone. Building them into the TEMP early is structurally different from retrofitting them during an OT&E acceleration.
4. Define a configuration management approach for continuous-learning components. If the AI system updates model weights post-deployment, the TEMP needs to specify at what point a model update constitutes a change that requires re-evaluation and what the lightweight re-evaluation standard looks like. The absence of this definition creates ambiguity about what the fielded system is.
5. Build a model card for every AI component in the system. Model lineage, training data sources, version history, and known failure conditions should be documented before integration testing, not reconstructed afterward. The CDAO Model T&E framework makes this a documentation expectation; it is also the minimum foundation for any future adversarial evaluation.
A credible alternative approach is to wait for the DoDI update and MYSTIC DEPOT delivery before revising TEMP structure. The risk is that milestone reviews are incorporating AI T&E questions now, and programs that have not yet incorporated the April 2024 frameworks will face those questions without a documented approach.
The CDAO frameworks also address a prior-to-integration question that milestone reviewers are beginning to ask: what does a program office actually know about how its AI component behaves outside its training envelope? Programs that can answer that with test data are in a different position than programs that rely on contractor assertions.
Spartan X
The gap between CDAO's guidance and what most program offices can currently execute against it is where technical advisory matters most. Spartan X's work in AI verification and assurance — including multi-model evaluation and adversarial stress testing — is directly relevant to programs building T&E approaches for AI-enabled systems. The structured question of what a model does under out-of-distribution inputs, adversarial pressure, and operational stress is the same question Spartan X's technical teams approach in AI program support and independent verification work.
Sources and Further Reading
- CDAO AI T&E Frameworks, April 2024 — ai.mil
- CDAO T&E Strategy and Frameworks overview — ai.mil blog
- DIU MYSTIC DEPOT solicitation — diu.mil
- DefenseScoop, "AI system testing, DoD, intelligence agencies" — defensescoop.com
- Defense News, "Pentagon seeks system to ensure AI models work as planned," March 12, 2026 — defensenews.com
- CNAS, "Safe and Effective: Advancing DoD Test and Evaluation for AI and Autonomous Systems," March 2025 — cnas.org
