DeepSeek DeepInfer: the Apache 2.0 inference engine behind V4.1 Flash runs on Nvidia Blackwell and Huawei Ascend, with key serving parts held back

DeepSeek DeepInfer is the software DeepSeek says it uses to serve its models to the public, released as open source under the Apache 2.0 license on October 20, 2026. It is an inference engine, not a model: it takes the weights of DeepSeek-V4.1-Flash, an MIT-licensed model among the open models we track, and serves them from a cluster. The GitHub post drew about 2,300 points and 880 comments on Hacker News. The code is real, but narrow. It supports one model on two kinds of hardware, and DeepSeek kept back the parts that tie a production service together.

This story is based on DeepSeek's DeepInfer repository, README and deployment Q&A, its 2025 note on open-sourcing its inference engine, and reporting by SiliconANGLE and SemiAnalysis, checked October 23, 2026.

The short version

• What: DeepSeek's in-house inference framework, now on GitHub as deepseek-ai/DeepInfer under Apache 2.0.
• Models: DeepSeek-V4.1-Flash only; the default checkpoint is the October 17 "1017" update.
• Hardware: Nvidia SM100 (Blackwell data-center) GPUs and Huawei Ascend 950 NPUs. No other hardware is listed.
• Left out: the KV-cache system, splitting the vision-encoder, prefill and decode stages across machines, and load balancing between replicas.
• DeepSeek's numbers (vendor-reported): 155 trillion tokens a day on one cluster, $38 per GPU-hour at off-peak prices.

What is DeepInfer?

The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services." Its only supported model is DeepSeek-V4.1-Flash. This release, the README says, "provides offline inference with the core features needed for serving at scale."

Its main parts:

• Kernels: DeepSeek's FlashMLA (attention), DeepGEMM (matrix math), DeepEP (expert communication), TileKernels, DeepSelect and custom TileLang kernels.
• Parallelism: attention data parallelism, context parallelism, MoE expert parallelism (EP) and expert load balancing (EPLB).
• DSpark speculative decoding: a draft model proposes five tokens and a "confidence head" decides how many to verify.
• Other features: asynchronous scheduling, structured outputs, and, on Nvidia only, deterministic sampling.

What did DeepSeek leave out?

The README says a complete service also needs "a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas," and that most engine functions for these already exist. DeepSeek's deployment Q&A says the KV-cache and load-balancing components are "tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced." A DeepSeek post on Zhihu describes their design.

The open kernels aren't quite DeepSeek's internal versions either. The Q&A says the public FlashMLA lacks one hand-tuned optimization, making its decode kernel 5%–10% slower, and that end-to-end results are "at most 5%" apart.

Can I run DeepInfer?

Only on data-center hardware. The prerequisites are Linux, GCC 13 or newer, Python 3.12, and CUDA 13 (Nvidia) or CANN 9.2 (Ascend). The single-node example uses eight accelerators. The V4.1 Flash weights alone take about 510 GB on Hugging Face, per our V4.1 Flash 1017 story.

DeepSeek's own sizing is far larger. Its Q&A says prefill runs on "ordinary 8-GPU machines," decode on "64–128-GPU supernodes," and that the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs. The README's performance charts were measured on 4 SM100 GPUs for prefill and 64 for decode. It plots throughput curves but prints no throughput figures.

Last verified: October 23, 2026

What performance does DeepSeek claim?

All of these are vendor-reported in the deployment Q&A:

• One cluster serves 155 trillion tokens a day, worth US$38 per GPU-hour at off-peak ("valley") prices. The document doesn't name the GPU model or say how tokens are counted.
• Recomputing only the last 128 tokens of a prompt "nearly halv[es] total prefill compute."
• DSpark's variable-length verification adds an estimated 4%–7% throughput at about 60 concurrent sequences per rank. Its table shows +5.4% at 60 and −1.5% at 16.

SiliconANGLE noted on October 20 that all of the published benchmarks ran on Nvidia chips and that the figures are DeepSeek's own. We found no independent DeepInfer benchmark as of October 23.

How is DeepInfer different from vLLM and SGLang?

vLLM and SGLang are general-purpose open-source engines used to serve many model families. DeepInfer serves one. The README says DeepInfer "was originally built primarily on vLLM," with ideas from SGLang and TensorRT-LLM. In a 2025 note, DeepSeek said its engine was "based on an early fork of vLLM" and decided against releasing it, citing its ties to internal infrastructure and limited capacity to maintain it. It chose to contribute to existing projects instead. DeepSeek hasn't published a head-to-head comparison with either engine.

What does it mean for Huawei Ascend?

DeepInfer runs on Huawei's Ascend 950 as well as Nvidia hardware. SemiAnalysis, in a paywalled October 21 report, says DeepSeek reworked FlashMLA's KV-cache layout for Ascend: "Earlier FlashMLA KV-cache layouts stored quantized values and their scaling factors in separate regions. The new implementation places them together." Bloomberg reported in September that DeepSeek plans to deploy at least 160,000 Ascend 950DT accelerators, according to SiliconANGLE.

The release came a day after the US Department of Homeland Security filed its order removing DeepSeek products from federal civilian agencies. Whether the order's "third-party products" language reaches self-hosted DeepSeek software is unclear, as our story on the DHS order explains. DeepInfer also followed DeepSeek's Search and Fetch APIs, launched on October 17.

Is DeepInfer the same as DeepInfra?

No. Search engines often autocorrect "deepinfer" to DeepInfra, a separate company that hosts open models through an API at deepinfra.com. DeepInfer is DeepSeek's software, at github.com/deepseek-ai/DeepInfer.

What's confirmed, reported and uncertain

• Confirmed (repository): Apache 2.0, V4.1 Flash only, SM100 and Ascend 950 support, feature list, what's left out, prerequisites.
• Vendor-reported: 155 trillion tokens a day, $38 per GPU-hour, the DSpark gains, the 5%–10% FlashMLA gap.
• Reported: SemiAnalysis on Ascend kernel changes; Bloomberg (via SiliconANGLE) on Ascend 950DT deployment plans.
• Uncertain: whether DeepSeek will release the KV-cache and load-balancing code; support for other models or Hopper GPUs; independent performance results.