Mission Models Need Mission Specifications: Buying Defense AI That Can Be Tested

Separate the document dispute from the acquisition lesson

The Intercept's September 8 reporting described documents obtained through Freedom of Information Act litigation and disputed language concerning OpenAI “Mission Models.” That reporting is the source of the controversy, not independent proof here of an executed contract requiring unrestricted compliance. The difference between draft terms and a signed instrument matters whenever a claim is made about what the government actually bought. The Intercept's reported contract dispute

OpenAI's own account of its defense agreement describes a cloud deployment with retained safety controls and restrictions. That statement provides the supplier's position; it does not by itself resolve every question about the released document. Taken together, the accounts warrant a careful discussion of requirements rather than a confident declaration that an executed contract eliminated safeguards. OpenAI's agreement statement

The practical question remains important: what should a mission-specific model be expected to do, and what evidence would demonstrate that it does so reliably?

Credentials establish identity, not unlimited authority

Commercial and government deployments can differ in user population, data sensitivity, approved purpose, and oversight. Those differences can justify different application controls and evaluation cases. They do not mean all commercial products lack authentication, or that every request from an authenticated government user is lawful and appropriate.

An authorized logistics analyst asking for a summary of approved maintenance guidance presents a different use case from someone asking the same application to disclose information outside their permissions. The interface may look identical. The system still needs to distinguish permitted work from work outside its scope.

A poorly calibrated refusal can interrupt legitimate work. A confident but inaccurate answer can be equally disruptive, and a disclosure beyond the user's authority can be more consequential. A sound specification considers all of these outcomes instead of optimizing a single response rate.

Define the failure modes before choosing the metric

A buyer should distinguish at least five kinds of failure:

• Incorrect refusal: the system declines a permitted request that it should handle.
• Incorrect compliance: it fulfills a request outside the approved scope or permissions.
• Incorrect answer: it responds but misstates the evidence, calculation, or applicable guidance.
• Unsupported certainty: it presents an inference or missing information as an established fact.
• Unreviewable action: it changes a record or initiates a workflow without the required authorization or usable audit evidence.

These categories need separate measures. A model can improve its refusal rate while becoming less accurate or less protective of restricted information. Conversely, a highly restrictive system can look safe in a narrow evaluation while being unusable for the task the government purchased.

The ARMOR research benchmark provides a useful example of bounded measurement: it evaluates doctrinal knowledge and false refusals using 519 multiple-choice questions. That is evidence about performance on a defined test, not certification that a model can safely carry out an operational workflow. A benchmark result should be reported with its task, dataset, and limitations. ARMOR research paper

Turn a mission description into acceptance evidence

A practical acquisition package can make “mission suitability” concrete without attempting to predict every possible prompt.

1. Describe the approved work. State which users, records, and decisions the application supports. Separate drafting or analysis from authority to change a system of record.
2. Identify boundaries. Define the information and actions that remain unavailable even to otherwise authorized users. Include how the system should handle ambiguous requests.
3. Build representative cases. Use permitted tasks, out-of-scope requests, incomplete source material, conflicting guidance, and access-control scenarios. Protect sensitive evaluation data.
4. Set acceptable outcomes. Specify the required evidence, accuracy criteria, escalation behavior, and acceptable reasons for declining a request.
5. Assign review authority. Name who accepts the application, who can change its scope, and which decisions require human approval.
6. Retest meaningful changes. Model updates, retrieval sources, permissions, tools, and workflow changes can alter behavior even when the product name stays the same.

For example, a procurement-document assistant should identify the source of an applicable requirement, distinguish a draft recommendation from an approved decision, and flag missing information. Its acceptance test should include those obligations. Counting how many answers it produces would miss most of the work that determines whether it is useful.

Audit records should support review without collecting sensitive content indiscriminately. Define what must be recorded, who may inspect it, how long it is retained, and how records themselves are protected. The aim is accountable operation with an appropriate evidence trail.

Policy direction is not completed implementation

NSPM-11, issued June 5, 2026, directs national-security AI assurance and accountability measures, including standardized testing, evaluation, verification, and validation methodologies and an update to the autonomy-in-weapons directive. The memorandum establishes policy direction and deadlines; it does not prove every implementing method or revised directive was already issued. NSPM-11

Acquisition teams should map applicable policy to the particular use case. A document-analysis assistant, a financial decision-support application, and an autonomous weapon do not acquire identical obligations merely because each uses AI. The responsible program needs to identify which rules apply and translate them into requirements that can be inspected and tested.

Deployment is moving quickly. CDAO's September 8 update describes Gemini Enterprise general availability and beta launches of ChatGPT Mil and Grok for Government with an Authority to Operate. That establishes platform status in the department's account, not blanket approval of every use case or every output. CDAO enterprise update

The phrase “mission model” will be useful only to the extent that the acquisition package gives it a specific meaning. Providers should be able to explain the approved function, demonstrate its boundaries, and show how performance remains acceptable as the system changes. That is a stronger basis for adoption than a label or a low refusal percentage.

Sources

• The Intercept: reported FOIA contract dispute
• OpenAI: account of its defense agreement
• ARMOR benchmark research
• White House NSPM-11
• CDAO September 8 enterprise update

Spartan X's AI consulting and cybersecurity expertise supports this translation from mission intent to an accountable system: useful requirements, realistic evaluation, and clear responsibility for the decisions an application supports.