Model choice and model use: an internal evaluation case
Synthetic engineering case from our own evaluation stand. Internal engineering evidence — not customer traction, not a savings guarantee.
What happened
An early evaluation on our own stand allowed automatic routing. Some requests were served by a model other than the one named by the test arm, while the runner estimated cost using the requested model's price. The gateway recorded the routing decision, the served model and the usage — so the divergence was detectable after the fact.
What changed
The comparison was rerun with pinned routes and a serving-model check: the gateway was switched to passthrough routing and every response's serving model was verified against the requested route. The earlier mixed-route results are not a valid model-to-model comparison.
What the result shows
Requested model, served model and price attribution must be checked
together. Provider-reported model identity (the model field in the
response) is useful evidence — it is how the mismatch was detected — but it
is not an independent attestation of model weights or physical compute
location.
What it does not show
This development smoke does not establish equivalent model quality, customer savings, production reliability or a reconciled provider bill. Some scenarios were rerun after server errors. A full cost comparison must state how those attempts are included.
Local and subscription costs
No per-token charge does not mean no operating cost. Hardware, power, administration and subscription allocation are outside an estimate that only uses provider token prices.
Method summary
| Field | Value |
|---|---|
| Run ID | 20260914T233306Z (pinned re-run; earlier mixed-route run 20260914T165059Z corrected) |
| Executed | 2026-09-14, UTC timestamps in records |
| Split | Public development seed set (smoke-selection.json), 18 cases × 2 arms; not a holdout |
| Task/runner/verifier version | Runner run-smoke.py @ repo HEAD 0f2f152+; verifiers in-repo; temperature 0, max_tokens 2500 |
| Requested/served identity | Requested = test arm; served = model field echoed by the provider response, cross-checked per call |
| Routing mode | Pinned routes (gateway passthrough); serving-model guard flagged any mismatch (0 flagged) |
| Attempts / rerun rule | No in-episode retries; four transport-error episodes were rerun once and are labeled as reruns |
| Price source / date | Nebius console price snapshot 2026-09-08 (per 1M tokens); zero-price arms are local/subscription routes |
| Cost coverage | Provider token usage only; hardware, power, administration and subscription allocation excluded |
| Latency population / method | Wall time per completed request at the caller, one hop through the gateway; p50/p95 over per-arm samples |
| Scenario inventory | A01, A02, A06, A07, A09 (agent, base+variant), D01–D03, D05 (coding, base+variant) |
Fields marked Not established from the available record: no provider-side attestation of serving hardware; no reconciliation against the provider console balance beyond the runner's journal.