Open weights are the easy part
Open weights let you obtain a model. Production ownership begins with memory, concurrency, networking, evaluation, and failure recovery.
Kimi K3 makes that distinction hard to ignore. Moonshot AI describes a 2.8-trillion-parameter mixture-of-experts model with a one-million-token context window. It uses 16 of its 896 experts for each token and stores weights in MXFP4 format. Sparse activation reduces the computation needed for each token, but the full collection of experts still has to live somewhere in the serving system.
A rough lower bound explains the scale. At four bits per parameter, 2.8 trillion parameters account for 1.4 trillion bytes, or about 1.4 TB in decimal units, for weights alone. That arithmetic excludes scales and metadata, the runtime, KV cache, temporary memory, redundancy, and any overhead from the actual checkpoint format. Moonshot's Kimi K3 launch post recommends a supernode configuration with at least 64 accelerators.
This is a distributed-systems purchase wearing a model-selection hat. Before assigning a platform team to it, pass five engineering gates.
Gate 1: Prove the model on your work
Start with a representative evaluation set from the jobs your product performs. For a coding assistant, that may include repository navigation, multi-file edits, test repair, tool calls, and review feedback. For an operations agent, it may cover incident evidence, runbook selection, and safe command planning. Score the result that reaches a user, not a model's answer in isolation.
Run every candidate through the same tools, prompt policy, reasoning budget, and context-management rules. Kimi's published benchmark table uses different evaluation harnesses for different models. That can be reasonable when each model needs its recommended setup, but it also means headline scores cannot replace an evaluation in your stack.
The harness is part of the product. Moonshot's launch post documents that K3 can become unstable when a client fails to carry its earlier reasoning history forward or switches models during a session. Its model guide also notes that changing the model invalidates the context cache. If an existing gateway strips reasoning state, rewrites tool messages, or silently falls back to another provider, you are testing a different system from the one in the benchmark.
Set a gate in advance: the model must clear your quality and safety thresholds with the exact serving path you intend to operate. If it fails, cheaper tokens will not rescue the product.
Gate 2: Model memory and concurrency together
One million tokens is a capacity ceiling, not a sensible default allocation for every request. Long contexts expand the KV cache, reduce the number of simultaneous sessions a cluster can carry, and can turn a traffic spike into a queue. The right capacity plan begins with the distribution of real requests: input length, output length, concurrent sessions, cache reuse, and latency targets.
Build a load test with at least these measurements:
time to first token at p50 and p95;
output tokens per second at p50 and p95;
successful concurrent sessions at the target context sizes;
KV-cache use and cache-hit rate;
behavior when a worker disappears or the queue reaches its limit.
The vLLM scaling guide recommends checking its reported KV-cache capacity and maximum concurrency, then adding GPUs or nodes when the measured throughput is insufficient. It supports tensor parallelism within a node and combined tensor and pipeline parallelism across nodes. Those options can make a model fit, but they also add communication and more failure boundaries.
Do not buy a cluster from a spreadsheet that assumes every request uses the maximum context. Do not benchmark with tiny prompts if production agents carry hours of tool history. Test the workload you expect, plus the ugly hour when several customers arrive together.
Gate 3: Compare whole-system cost
API cost is variable and visible. Self-hosted cost is partly fixed and scattered across several budgets. Put both in the same monthly model.
As of July 27, 2026, Moonshot lists Kimi K3 API prices of $0.30 per million cache-hit input tokens, $3.00 per million cache-miss input tokens, and $15.00 per million output tokens. The API side of the comparison is straightforward:
Monthly API cost = cache-hit input MTok × $0.30 + cache-miss input MTok × $3.00 + output MTok × $15.00
Use token and cache data from a real pilot. Vendor-reported cache behavior is useful for forming a hypothesis, but your prompt prefixes, tenant isolation, deployment changes, and model switching determine the cache rate you will actually receive.
For self-hosting, include accelerators or lease fees, CPU and RAM, high-speed networking, storage, power, idle capacity, redundancy, observability, platform labor, and the capacity held for peaks. Amortize setup and upgrade work. Price a recovery environment too; a cluster that is cheap only while every worker is healthy is not production capacity.
There is no honest universal break-even token count. Utilization decides whether expensive hardware earns its keep. Stable, high-volume demand can favor owned capacity. Bursty or uncertain demand often rewards the API's ability to shrink back to zero.
Gate 4: Write the security boundary down
Self-hosting can be the right answer for an air-gapped environment, strict data residency, or a requirement to control retention and telemetry. The weights do not grant those properties by themselves. They come from the network, access controls, logs, backups, deployment process, and people who operate the system.
K3 uses a custom published license. It generally permits use and modification, but it requires a separate agreement for certain Model-as-a-Service businesses above $20 million in aggregate revenue over a consecutive 12-month period. It also requires attribution for some commercial products above 100 million monthly active users or $20 million in monthly revenue. The license states exceptions for internal use and use through official products or certified inference partners. Have counsel map your deployment to the actual terms.
Multi-node inference deserves particular attention. vLLM's documentation says its inter-node traffic is unencrypted and should run on a private network. That network must also deliver enough bandwidth for distributed inference; a slow or misconfigured link can undermine both performance and availability.
Trace one sensitive prompt through the entire design. Record where it is accepted, cached, transmitted, logged, inspected, and deleted. Then do the same for generated code and tool output. Compare that boundary with the contractual and technical controls of the API or managed service. “It runs in our cloud account” is a location, not a threat model.
Gate 5: Make somebody own the bad days
A 64-accelerator deployment needs more than an inference command. It needs health checks, admission control, autoscaling or queue limits, rolling upgrades, rollback, capacity alerts, model-loading procedures, and a plan for failed workers. The environment must stay consistent across nodes. The interconnect and collective-communication path need monitoring. A model release becomes an infrastructure change with a blast radius.
Name the owning team and run two exercises before launch: remove a worker under load, then roll back a model or runtime upgrade while requests are active. Measure error rate, recovery time, and whether in-flight sessions remain valid. If nobody has the time or authority to keep that path healthy, self-hosting has already failed the operations gate.
Choose the least ownership that meets the requirement
Use the API while proving product quality or serving demand that is still unpredictable. Choose managed or dedicated inference when stronger isolation and predictable capacity matter but owning a large distributed cluster would distract the team. Self-host when hard sovereignty requirements, custom serving work, or sustained utilization make the operational burden worthwhile.
Kimi K3 may be an excellent model for the job. The useful engineering decision is the amount of its production system your company has a reason to own.