Test semantic correctness

A schema-valid answer has the right form. A policy-compliant answer obeys your rules. Neither alone proves that the model understood the input correctly. Evaluate the interpretation as well as the constraints.

Test the decisions your application depends on

Use representative inputs with independently reviewed expected answers. Include ambiguous wording, missing facts, uncommon categories and boundary cases. For extraction, compare actual values—not only their types. For policy tasks, include valid, repairable and impossible cases.

Keep development examples separate from a held-out evaluation set. Improve descriptions using the development set, then measure whether the change helps on unseen inputs.

Record enough to reproduce a result

Save the input, schema or immutable reference, supplied state, output or error, and client elapsed time. For policies, also record the policy definition or digest, retry budget, feedback setting and reported retry count. Keep credentials and sensitive claims out of general-purpose logs.

Track semantic accuracy, compliant-success rate, failure rate and latency separately. A stricter policy can prevent invalid answers while increasing failures. Excluding failed or slow requests makes the comparison misleading.

Improve the guidance before increasing the budget

Use descriptions to explain evidence, label boundaries, missing-information behavior and relevant constraints. A policy can reject an answer, but it cannot teach all the knowledge needed to solve an unfamiliar task. A larger search budget helps only when useful alternatives can be found.

Evidence in these docs

Outputs labelled illustrative explain the contract; they are not promises about a future model response. The source files and clients have local checks. All sixteen published examples passed a live check through Struct in Python, TypeScript and cURL, including state, permissions, totals, dates, label choices and the text-count example. Successful outputs were checked independently against their schemas and policies. This small verification set is not an accuracy benchmark or a hosted-service latency guarantee.

Use those examples as starting points, then evaluate your own inputs. See latency for measurement boundaries and errors for handling unsuccessful requests.