Beyond the receipt
The GPU bill leaves out most of the project
Fine-tuning can be cheap enough to fit on a team lead's card and expensive enough to waste a quarter. The difference sits outside the GPU rental.
- Training run
- $500
- reported GPU cost
- Training data
- 177,767
- simulated episodes
- Validation
- 200
- stratified episodes
- Decision
- 5 gates
- before training
On July 27, 2026, FermiSense reported that it trained an unnamed 9B open model for catalog review using two RTX PRO 6000 GPUs, 1,000 optimizer steps, and about three and a half days. The run cost roughly $500. On its benchmark, the specialist scored 0.626, ahead of every frontier configuration the team tested, while costing a reported $0.50 per 1,000 listings. The result prompted a useful Hacker News discussion about how much of the real project that figure leaves out.
Treat those numbers as a signal rather than a transferable quote. The same engineering report says the team built 177,767 simulated review episodes from the Amazon Berkeley Objects dataset, added controlled policy cases, exposed tools, encoded different penalties for missed violations and false alarms, and created a scorer. The training invoice covered the final optimization run. It did not cover the system that made training possible.
A useful budget starts when the baseline is designed. Someone must define the task, collect representative inputs, settle disputed labels, implement tools, build a scorer, and reserve data that training cannot touch. The model still needs an inference service, observability, security review, a fallback path, and an owner.
The linked PRIME-RL framework shows why the GPU line is incomplete even when training code is open. Its operating surface includes inference, evaluations, checkpointing, observability, model-specific kernels, and multi-node deployment. A small run avoids some distributed work, but it still needs a working environment and reliable evaluations.
Encoded judgment
The scorer becomes product policy
Reinforcement learning improves the behavior that receives reward. In the catalog experiment, a missed policy violation carried seven times the penalty of a false alarm. That ratio is a product decision expressed as code: it determines which mistakes the model learns to fear and how much manual review the system may create.
A weak scorer can reward the wrong behavior with great precision. A 2026 study of models gaming verifiers found that RL-trained models could pass an incomplete rule checker by enumerating instance-level labels instead of learning the intended general rule. The benchmark improved while the underlying capability did not.
Before training, ask domain owners to review three artifacts:
- A written definition of success, abstention, and each costly failure mode.
- A scorer tested against obvious shortcuts and adversarial examples.
- A locked test set drawn from a later period or a source excluded from training.
Subjective work raises the price of this step. Bridgewater AIA Labs and Thinking Machines reported that models trained on non-expert labels still performed poorly because many labels were wrong. They routed contested examples to investment experts, cleaned the training data, and evaluated the result on a held-out test set. Expert attention, rather than optimizer time, was the scarce input.
Complete economics
Put every recurring cost in the model
Use a cost ledger before approving a pilot. An omitted category means the estimate is unfinished.
- Baseline
- Prompt, retrieval, rules, and frontier-model evaluations on the same test set.
- Data
- Collection, licensing, privacy review, labeling, adjudication, and versioning.
- Environment
- Tool simulators, sandboxes, scorer implementation, and shortcut tests.
- Training
- Failed runs, ablations, GPU time, experiment tracking, and engineer time.
- Serving
- Accelerators, batching, autoscaling, storage, network, and fallback calls.
- Operations
- Quality monitoring, incident response, retraining, and model upgrades.
Use cost per accepted task, since a cheap answer that is escalated, corrected, or silently wrong is not a saving. Include reviewer time in both paths. Run the comparison at the quality threshold the product needs, not at each model's most flattering operating point.
Scale can make the arithmetic compelling. Shopify has described a production catalog system doing 40 million multimodal calls and about 16 billion tokens per day. The same report describes Triton serving, batching, routing, quantization, caching, human review, and active retraining. Those are cost lines, not decorative architecture.
Go or no-go
Require five gates before training
A narrow specialist deserves a pilot when all five gates pass.
- The task repeats at material volume
Monthly savings can repay the fixed build cost inside an acceptable period.
- Success can be scored
The team can encode correct outputs, costly errors, and abstention.
- The task boundary is stable
Training carries durable procedure while current facts remain in tools or retrieval.
- Production-shaped data exists
Important slices and rare failures have explicit provenance and separation.
- A team owns the model after launch
Named owners can monitor, reproduce, roll back, refresh, and retire the release.
Failure at any gate points to a cheaper next experiment. Tighten the workflow with deterministic code, improve retrieval, collect labels behind a frontier-model baseline, or keep a human approval step. Fine-tuning becomes an option after the missing prerequisite exists.
Production evidence
Shadow traffic decides whether the pilot ships
Offline scores earn a candidate a shadow test. Run the specialist beside the current path without letting it affect users, then compare both on the same live requests.
- Weighted error by critical slice
- Abstention and escalation
- Latency and capacity
- Cost per accepted task
- System disagreement
- Rollback threshold
Review a sample of agreements as well as disagreements; two systems can confidently share the same blind spot. Set rollback and retraining thresholds before anyone sees an encouraging chart.
This operating loop continues after launch. Google's current production ML guidance calls for monitoring serving-data drift, prediction drift, quality, latency, quota, and test-data skew, with deployment and rollback ready when a model degrades.
Approve the fine-tune when measured savings survive the data, verifier, serving, and ownership costs. Until then, the $500 receipt proves that one training run was cheap.