Engineers at Baseten have open-sourced DeepSeek Ultrafast, an inference engine tailored for DeepSeek V4.1 Flash. In demonstration benchmarks, the runtime achieved generation speeds of 2,060 tokens per second across an eight-way cluster of Nvidia B300 accelerators. The release highlights how significantly large language model latency and throughput depend on software infrastructure, runtime memory management, and hardware orchestration.
The DeepSeek Ultrafast repository went public on GitHub in November 2026.
An essential clarification: the engine was developed and published by Baseten Labs, not by the original creators of the DeepSeek model family. It is an optimized serving architecture built specifically to run the pre-existing DeepSeek V4.1 Flash checkpoint at maximum hardware utilization.
Benchmark Performance and Latency Numbers
The developers published verified throughput numbers across two high-end Nvidia server configurations:
| Metric | 4 × Nvidia GB300 | 8 × Nvidia B300 |
|---|---|---|
| Generation Throughput | 1,740 tokens/s | 2,060 tokens/s |
| Decoding Step Latency | 3.30 ms | 2.71 ms |
| Average Accepted Tokens per Step | 5.6 | 5.7 |
| 1M Token Context Processing Time | 13.2 s | 11.7 s |
| Input Prefill Bandwidth | 76,000 tokens/s | 85,000 tokens/s |
To benchmark throughput under consistent conditions, the authors ran a specific programming prompt: generating clean HTML and JavaScript code for a playable Tetris game without extraneous conversational filler.
Reasoning mode was intentionally disabled during the public replication benchmark.
Baseten also documented the throughput variance across repeated test runs: roughly 1,700 to 1,770 tokens/s on four GB300 chips and 1,980 to 2,170 tokens/s on eight B300 accelerators.
Thus, 2,060 tokens/s represents an observed measurement on dedicated cluster hardware rather than an absolute guarantee for DeepSeek V4.1 Flash under arbitrary workloads.
To put this in perspective: streaming a 2,000-token coding response at 2,060 tokens/s takes roughly a single second. In live production environments, total perceived response time will also factor in input prefill duration, network round trips, and concurrent load scheduling.
Architectural Optimizations Behind the Speed
DeepSeek Ultrafast combines three architectural techniques to push inference throughput to its theoretical limit.
1. Speculative Decoding
A key metric in the benchmark report is the average number of accepted speculative tokens per forward step.
Standard autoregressive decoding requires the model to compute one token at a time sequentially. Speculative decoding uses a smaller draft mechanism to predict candidate continuations, allowing the main model to verify several tokens simultaneously in a single forward pass.
In the eight-GPU B300 configuration, the engine verified an average of 5.7 tokens per step.
This acceptance rate explains why decoding latency drops to just 2.71 ms per step.
2. Aggressive KV Cache Reuse
When serving multi-turn agentic conversations, the runtime caches intermediate key-value projections from previously processed turns.
Without persistent KV caching, servers would have to recompute preceding dialogue history on every follow-up turn, driving up latency and server compute bills.
DeepSeek Ultrafast persists KV tensors in high-speed memory across requests and supports cluster-wide sharing.
3. Cross-Node RDMA Cache Migration
The team integrates Redis to track which physical worker node holds the active KV cache for a given session.
If a subsequent request is routed to a different physical machine in the cluster, the necessary cache pages migrate over low-latency RDMA links, avoiding a complete re-read of the conversation context.
According to Baseten, DeepSeek's native Multi-Head Latent Attention (MLA) format achieves high KV compression, keeping payload sizes small enough to make real-time network transfers practical on every turn.
This setup grants engineering teams flexibility in balancing cluster traffic.
Handling 1 Million Tokens of Context
The engine supports context windows up to 1 million tokens.
In demonstration tests, ingesting a full 1M-token prompt took 11.7 seconds on eight B300 accelerators.
Here it is vital to distinguish two operational metrics:
- Prefill: Processing the existing input prompt tokens.
- Decode: Sequentially emitting newly generated output tokens.
The measured bandwidth of 85,000 tokens/s applies exclusively to input prefill processing. It does not imply that the model streams novel output at that speed.
Why Serving Architecture Matters for the AI Industry
Much of the competitive discourse around frontier models has centered on total parameter counts, architectural changes, and academic benchmarks.
However, inference efficiency is rapidly becoming an equally critical competitive battleground.
A single model checkpoint can exhibit dramatically different response latency, concurrent throughput, and operating costs depending on its serving runtime, kernel optimizations, and GPU cluster topology.
Open-source serving frameworks such as vLLM and SGLang tackle similar challenges with continuous batching, PagedAttention, and speculative decoding.
DeepSeek Ultrafast represents a specialized, hardware-optimized implementation focused squarely on one model architecture and cutting-edge Nvidia server platforms.
Implications for Developers and Agent Workflows
Extreme generation speeds are particularly transformative for agentic coding workflows.
Autonomous coding agents routinely generate files, run unit test suites, parse error traces, and regenerate patches over dozens of iterative loops. Compressing generation times from minutes to seconds fundamentally alters the viability of running autonomous software development agents.
However, the complete economic profile remains to be quantified.
Determining cost-per-task efficiency requires factoring in cluster rental rates, steady-state utilization, queue lengths, and token consumption patterns.
Furthermore, direct comparisons between DeepSeek Ultrafast and commercial hosted APIs are only meaningful when measured across identical hardware configurations and service-level agreements.
Availability and Deployment Considerations
Baseten Labs has published the code, installation guides, and reproduction scripts on GitHub, allowing engineering teams to audit the implementation and independently reproduce results.
Nonetheless, these published numbers currently reflect benchmarks conducted by Baseten's internal team. Independent head-to-head comparisons against the latest builds of vLLM and SGLang under identical cluster conditions have yet to be published.
Deployment requirements also present a practical boundary. Reproducing the demonstration requires modern enterprise-grade Nvidia accelerators, CUDA 13, and high-bandwidth interconnects; the setup is not intended for consumer workstations.
DeepSeek V4.1 Flash Fast is available as a managed endpoint via Baseten Model APIs, while the GitHub repository allows teams to self-host the stack with an OpenAI-compatible interface.
Conclusion: Clocking 2,060 tokens per second highlights the immense headroom available through specialized inference engineering. For teams deploying large models in production, it underscores how deeply infrastructure choices shape end-user latency and operational margins.
Sources: