Understand limits and latency
Measure elapsed time around the full HTTP request. That is the latency your users experience.
Separate the timing boundaries
| Measurement | Includes |
|---|---|
| Registration | Validation and schema compilation before publishing a version |
| First execution | Request work plus any remaining initialization and cache setup |
| Warm execution | An execution after the schema has been used |
| Client wall time | Network, gateway, admission, execution and response transfer |
Use a client timer for the full round trip. Compare the same input sizes, schemas, policy settings and concurrency.
Successful resolve responses also include a Server-Timing header. Durations use milliseconds:
| Metric | Measures |
|---|---|
gateway | Gateway processing up to sending the response headers |
runtime | The gateway's call to Struct, including transport between those services |
total | Struct's total request processing time, when supplied |
inference | Model execution time, when supplied |
queue | Time waiting for compilation or execution capacity, when supplied |
The measurements overlap: do not add gateway, runtime and total, or add inference to total. Total processing also includes work outside inference and queues. Missing metrics mean unavailable measurements, not zero. These server timings exclude the network round trip from your application. Some failed requests also include timing; handle its absence and ignore unrecognized metric names.
Reduce avoidable work
Register repeated schemas ahead of traffic. Send relevant input and state, give open strings and arrays useful bounds, and ask only for fields your application needs. Registration reduces repeated compilation; it does not promise warm model state.
A resolve request body, including any inline schema, state or policy, can be at most 8 MiB. Schema, policy and alias request bodies can be at most 4 MiB. A larger body returns 413.
Account for policy search
Measure successful and failed/exhausted requests separately, retaining all calls in your overall latency calculation. Compare policy-free requests with policy-bound requests and different retry budgets. Policy checks, additional attempts and optional feedback can increase latency for requests with policies. A larger retry budget is not a throughput or success guarantee.
Plan for concurrency
Each account can start 10 resolves a second, run 8 at once and use 1,000,000 tokens a minute. A resolve over a limit returns 429; queue requests in your application to stay within them. Requests from all accounts share service capacity; when it is full, a request returns 529 CAPACITY_BUSY. See rate limits.
Measure throughput and p50/p90 latency together at the input sizes your application expects. An idle single-request measurement does not predict latency under load. Handle capacity and deadline errors with a bounded retry strategy. Unsupported or excessively complex schemas fail compilation; increasing a client timeout does not expand schema support.