Field guide / 9 min
Keep model choice at the edge
Route by tested capability instead of embedding a fashionable name.
The OpenAI model catalog changes over time. Product code should ask for a tested capability profile—quality, latency, modalities, tool support, and budget—then resolve that profile through configuration and evaluations.
Working principles
Names are deployment data
Keep active model identifiers in configuration or a registry. Historical fixtures may remain pinned for reproducibility.
Capabilities need tests
Documentation narrows candidates; your fixtures establish whether a candidate works for your prompts, tools, schemas, and latency target.
Preserve routing roles
Do not collapse every workload onto one model. Separate quality-sensitive, latency-sensitive, realtime, and background roles.
Field procedure
- 01
Define capability profiles
For each workload, record required modalities, tools, structured output, context, latency, and budget.
- 02
Create a model registry
Map profile names to deployable model identifiers and keep the mapping outside feature code.
- 03
Build representative fixtures
Include normal, edge, adversarial, refusal, and tool-use cases with observable acceptance rules.
- 04
Run a shadow comparison
Compare candidate and current deployments on quality, schema success, tool outcomes, latency, and cost before switching.
- 05
Roll out reversibly
Use a staged percentage, monitor product-level outcomes, and retain a fast rollback path.
PASS / FAIL
Acceptance checklist
- No feature branch embeds a volatile model name.
- Profiles state hard and soft requirements.
- Evaluation data includes tool and schema behavior.
- Rollout and rollback are configuration changes.
WATCH / REJECT
Failure patterns
- Choosing solely from a leaderboard or marketing label.
- Updating examples and production defaults without behavioral tests.
- Using one global model setting for unrelated workloads.