Open model weights have been public for a long time, but the software that turns those weights into cheap, fast tokens has stayed inside the labs. On 20 October 2026, DeepSeek published its own production inference engine, DeepInfer, along with a deployment Q&A that explains how it serves DeepSeek-V4.1-Flash at the scale of 155 trillion tokens a day on a single cluster.
AG Tech Consulting's advice for clients this week: if you run, or plan to run, open-weight models on your own GPUs, have your platform team read the DeepInfer deployment Q&A before you buy more hardware or sign another per-token contract. Use it as a reference design to check your own serving stack (prefill/decode split, expert parallelism, KV cache retention, speculative decoding) and benchmark it on your own traffic. Before any code goes near production, get a written decision from legal and security on DeepSeek-derived software, especially if you sell to US federal agencies.
What DeepSeek released on 20 October
DeepInfer is described in its README as "the high-performance inference framework that powers DeepSeek's large-scale production services." It is on GitHub under the Apache License 2.0, and for now it supports one model: DeepSeek-V4.1-Flash, loading the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face by default.
It runs on two hardware targets: NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. The build needs Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2, selected with DEEPINFER_ACCELERATOR=nvidia or ascend. Hopper and older NVIDIA architectures are not on the list.
Alongside it, DeepSeek released deepseek-recipe, a Rust library with Python bindings that converts Chat Completions, Responses and Messages API requests into model tokens the same way DeepSeek's official API does.
In April 2025, DeepSeek said its engine, an early fork of vLLM, was too tightly coupled to its cluster tools to release. The new README still credits vLLM as its original foundation, with inspiration from SGLang and TensorRT-LLM.
What is in the box, and what is not
The release is an offline inference engine: batches of prepared requests, not a live service. It ships with DeepSeek's kernel libraries (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect) plus custom TileLang kernels, and with attention data parallelism, balanced context parallelism, and MoE expert parallelism with an expert-parallel load balancer (EPLB).
It also includes DSpark speculative decoding, SWA bounded replay, a zero-overhead asynchronous batch scheduler, structured outputs (regex, JSON schema or EBNF), batch-invariant inference and deterministic sampling. The last two work only on NVIDIA.
Three things a complete service needs are missing: a KV cache system, vision-encoder/prefill/decode disaggregation (the README says "most required engine functionality is already implemented"), and load balancing across data-parallel replicas. DeepSeek says that code is tightly coupled to its internal infrastructure and "not yet fully open-sourced," but the Q&A and a Zhihu blog post describe how it works.
How the production system is built
The Q&A states three design goals: run on accelerators from several vendors; put prefill on ordinary 8-GPU machines and decode on 64-128-GPU supernodes; and keep KV cache for more than 72 hours so repeated context does not have to be prefilled again.
The result is a disaggregated vision-encoder / prefill / decode (E/P/D) architecture with dedicated storage nodes. Each stage scales on its own, so time-to-first-token and time-between-tokens can be tuned separately.
Prefill and decode on separate machines
Prefill runs at small expert parallelism (EP2-8 / DP2-8, the smallest that HBM allows). Decode runs "Wide-EP" at EP64-128, the largest the available GPUs allow. Tensor and context parallelism are not used for now.
The P:D GPU ratio is roughly 1:1, so DeepSeek describes the smallest efficient unit as 64 GPUs of prefill plus 64 GPUs of decode. Splitting P and D across data centers is possible because V4.1-Flash compresses its KV cache heavily; DeepSeek budgets about 400 Gbps of interconnect per 8 GPUs and uses adjacent data centers.
Wide-EP and expert load balancing
Wide-EP cuts per-step memory traffic in the MoE layers. Hot experts get replicas on redundant expert slots: 4 per GPU in prefill, re-assigned every 1,000 steps, and 1 per GPU in decode, re-assigned every 10,000 steps.
The prefill balancer buckets requests by context length (under 16K, 16-64K, 64-256K, 256K and above) and gives each step a time budget, predicted by a fitted quadratic cost model. DeepSeek says its load balancing reaches roughly 80% of a perfectly balanced offline benchmark.
A KV cache that lives on SSD for days
The block KV cache sits on DeepSeek's 3FS distributed file system, about 100 PiB of SSD, with 10+2 erasure coding: a read goes to all 11 nodes in a group and any 10 replies are enough. Because the store is shared, any prefill machine can hit any prefix, so there is no prefix-affinity routing.
Retention is 3 days for agent traffic and 7 days for chatbot traffic; DeepSeek says a TTL of about 3 days would cost chatbot traffic 0.5%-1% of its cache hit rate. Swap-in to HBM uses GPUDirect RDMA rather than the CUDA copy engine, so it does not consume GPU compute units.
Two techniques worth copying: SWA bounded replay and DSpark
SWA bounded replay. When the sliding-window attention state is missing, DeepInfer recomputes only the most recent 128 tokens instead of the whole window, so recomputation cost becomes constant regardless of context length and layer count. In the decoder layers, recomputing the last 128 prompt tokens instead of 2,560 tokens through 20 layers is, per DeepSeek, "nearly halving total prefill computation."
DSpark. A draft model proposes 5 tokens, and a Confidence Head predicts how likely each one is to be accepted. Verification compute goes only to the most promising prefix. DeepSeek reports about 30% fewer draft tokens verified per step (5.00 down to 3.2-4.4) with accepted length almost unchanged (3.29 vs 3.20), worth 4-7% more throughput at its production operating point of about 60 concurrent sequences per rank.
Neither gain is dramatic on its own. The point is that serving cost is the sum of many such engineering choices, and DeepSeek has now published its list.
NVIDIA SM100 and Huawei Ascend 950 side by side
Huawei Ascend gets the same billing as NVIDIA from day one. The Ascend path follows a 30 September release that brought TileLang kernels to Huawei chips, and Bloomberg reported in September that DeepSeek plans to deploy at least 160,000 Ascend 950DT accelerators.
Read the benchmarks carefully. They were measured on 4 SM100 GPUs for prefill and 64 for decode, all on NVIDIA, and they are DeepSeek's own figures with no independent testing yet. DeepSeek also says the open FlashMLA lacks one hand-tuned SASS optimization, making its decode kernel 5%-10% slower than the internal version, with end-to-end results at most 5% apart.
FP4 is used heavily: the main KV cache is per-16 e4m3 (which DeepSeek likens to NVFP4 without the global scale), and the indexer KV and routed MoE expert weights use MXFP4.
The procurement and security question
DeepInfer landed the same week the US Department of Homeland Security published Exclusion and Removal Order #2026-01 in the Federal Register on 20 October. Issued under the Federal Acquisition Supply Chain Security Act, it removes DeepSeek products, and "third-party products and services developed using DeepSeek implementations," from federal civilian agency IT.
The order does not define "DeepSeek implementation," and law firm Haynes Boone notes it does not separate DeepSeek-hosted from self-hosted deployments or fine-tuned versions. Under FAR 52.204-30, contractors may not provide or use covered products under an applicable FASCSA order unless a waiver is granted.
Self-hosting keeps prompts off DeepSeek's servers, but it brings new risks to plan for:
1. Scope creep in regulation. An undefined "implementation" could be read to cover an engine, a tokenizer library or a fork. Keep a software bill of materials that shows exactly which DeepSeek-origin code runs where.
2. A shared, long-lived KV cache. A cache that any prefill node can hit and that keeps context for days is also a store of your users' prompts. If you copy this design, decide how tenants are separated, who can read the storage tier, and how deletion requests reach cached context.
3. Prompt-format injection. deepseek-recipe encodes strings such as <｜User｜> in user input or tool results as ordinary tokens, "preventing text from injecting control tokens." Any home-grown serving layer must do the same, or untrusted text can spoof roles and tool calls.
4. Fast-moving native code. Custom CUDA, TileLang and Ascend kernels are a large new code base on your inference hosts. Treat them like any other third-party dependency: pin versions, build from reviewed source and watch advisories.
A checklist for CTOs evaluating self-hosted inference
Measure your current cost per million tokens and GPU utilisation before you compare anything with DeepSeek's figures. Check hardware fit: DeepInfer targets NVIDIA SM100 and Ascend 950 only, and DeepSeek's efficient unit is 64 prefill plus 64 decode GPUs. Decide who builds the missing pieces: KV cache storage, P/D disaggregation and replica load balancing. Run DeepSeek's own sweep script (examples/sweep/run_sweep.sh) and the eval/ Pass@1 sanity checks on your hardware, then replay your own traffic. Compare against what you run today (vLLM, SGLang, TensorRT-LLM) on the same model and hardware; the vLLM project says it will keep improving its V4.1-Flash support with ideas from DeepInfer. If you need reproducible outputs for audit, note that batch-invariant inference and deterministic sampling are NVIDIA-only. Set a KV cache retention policy that matches your data rules, not just your hit-rate target. Use deepseek-recipe or an equivalent that blocks special-token injection from user input and tool results. Get a written legal view on the DHS FASCSA order if you sell to US federal agencies or their contractors. Pin DeepInfer by commit, build from source in a controlled pipeline, and keep an SBOM for every inference host.
The bottom line
Until now, "open model" usually meant open weights and closed economics. DeepInfer and its Q&A close part of that gap: the serving engine and the design of the system around it are now public, even if the cache and load-balancing code are not.
Treat it as a reference architecture first and a dependency second. Benchmark it on your own traffic, and settle the procurement question before your engineers fall in love with it.
Sources
https://github.com/deepseek-ai/DeepInfer https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 https://siliconangle.com/2026/10/20/deepseek-open-sources-deepinfer-the-inference-framework-behind-its-production-services/ https://thenewstack.io/deepseek-deepinfer-huawei-nvidia/ https://thenextweb.com/news/deepseek-deepinfer-open-source-inference-engine https://www.scmp.com/tech/big-tech/article/3371592/deepseek-open-sources-new-tool-slash-ai-costs-fight-against-closed-source-rivals https://realpython.com/ai-news-weekly-2026-10-23/ https://www.federalregister.gov/documents/2026/10/20/2026-21447/notice-of-order-regarding-exclusion-and-removal-of-deepseek-products-from-federal-civilian-executive https://www.akingump.com/en/insights/ai-law-and-regulation-tracker/dhs-orders-exclusion-and-removal-of-deepseek-products-from-federal-civilian-agency-systems https://www.haynesboone.com/news/alerts/dhs-issues-deepseek-exclusion-and-removal-order.pdf
Also on our website: https://agtechconsulting.ai/insights/your-open-model-was-half-stack-deepseek-opened-engine.html
In this story: DeepSeek AI , NVIDIA , Huawei , vLLM , SGLang , Hugging Face , Red Hat , Together AI
Want help benchmarking self-hosted inference against your current AI bill, or designing a serving stack your security team can sign off? Email avinash@agtechconsulting.ai, visit agtechconsulting.ai, or book a 30-minute call: https://calendly.com/avinash-dubey-30-min-catchup/meeting-with-ag-tech
Originally published on LinkedIn in the Your CTO Advisor newsletter: read and comment on LinkedIn →
