Chasing the Batch-1 Latency Floor: Ling-3.0-flash at 0.78 ms TPOT on 4× B200

Most inference benchmarks reward batching. Real users do not experience a batch. They experience the next token.

That distinction changes the optimization problem. At batch 1, there is no concurrency to amortize launch overhead, no batch to fill pipeline bubbles, and little arithmetic intensity for a clever tile to hide behind. Every microsecond on the decode critical path is visible to an interactive agent, coding assistant, or real-time application.

That is the problem we set out to solve for Ling-3.0-flash on 4× NVIDIA B200 GPUs (SM100), using SGLang. We started with the model’s NEXTN speculative-decoding path, then brought the same systems work to DSpark, a confidence-scheduled speculative decoder.

Here is what that looked like on a fixed 8,192-token input / 1,024-token output synthetic workload, with --max-concurrency 1 (one request in flight):

Batch-1 latency results on 4× B200

The two left-hand columns are earlier, shorter campaign measurements, so the 3.33 ms → 1.53 ms NEXTN improvement is best read as an engineering trajectory—not a controlled A/B. The two right-hand columns are directly comparable: both use the same 1,000-request benchmark with --max-concurrency 1 on the same machine. On that controlled comparison, DSpark reduces mean TPOT by 1.9×.

The headline numbers matter, but the lesson matters more: batch-1 performance is a whole-stack problem. The model, runtime, CPU launch path, CUDA Graph replay, kernel dependencies, and speculative-decoding policy all meet on the same critical path.

Why Ling-3.0-flash is a good batch-1 test case

Ling-3.0-flash is a 42-layer hybrid-attention MoE model: 35 KDA linear-attention layers and 7 MLA full-attention layers, with 512 routed experts plus one shared expert. It runs with tensor parallelism across four GPUs (TP4) in bf16, and each rank holds roughly 63 GB of weights.

Ling-3.0-flash combines 35 KDA linear-attention layers with 7 MLA full-attention layers and a 512-expert MoE.

The hybrid design is important. In the final 8K profile, MLA attention accounts for only about 244 µs of a decode step. Long context is not the main challenge here. The hard part is more familiar: cold MoE-weight reads, dense weight bandwidth, hundreds of small kernels, and every handoff between the host and GPU.

So this is not a story about finding one slow attention kernel. It is about finding the brief moments when an otherwise fast stack stops moving—and getting rid of them.

One decode step is a relay race

For NEXTN, one decode step involves three CUDA Graph replays:

A lightweight draft model proposes a fixed token chain.
The 42-layer target verifies that chain in one forward pass.
An extend graph repairs the draft-side cache and produces the next seed.

Three CUDA Graph replays form one speculative decode step. Fixed shapes and persistent device buffers keep accept-dependent values on the GPU.

The key detail is that the CPU does not need to know how many tokens were accepted on every step. Device-resident buffers carry tokens, sequence lengths, probabilities, and hidden states forward by request index. Things that really do need the CPU—EOS, stop strings, and detokenization—take an asynchronous side path and are consumed one step later.

That gives the CPU room to queue the next step while the GPU works on the current one. It only works, though, if the host does not accidentally force everything back into lockstep.

The biggest win: let the host get ahead

Our first traces showed the GPU busy for only about two-thirds of each batch-1 step. That sounds like a kernel problem. In this case, it was mostly a host problem.

Under speculative decoding, SGLang’s scheduler queried sequence lengths while preparing the next batch. A KDA attention backend inherited a default that declared CPU-side sequence lengths necessary. That declaration caused a device-to-host copy and synchronization in every step. The copy itself was small; the synchronization was not. It waited for the prior verify graph to finish, resetting the host’s run-ahead depth to zero.

Removing a single inherited CPU-sequence-length requirement changed the host from lockstep execution to a deep, asynchronous launch queue.

KDA’s metadata path never read the CPU mirror, so declaring needs_cpu_seq_lens = False removed a wait we did not need. That recovered roughly 485 µs of median host stall per step and let the CPU queue the next graph sequence while the GPU was still busy.

Here is the practical lesson: before tuning a batch-1 kernel, look for blocking reads on the host path. A tiny D2H read can turn CPU work that should be invisible into a very visible GPU bubble.

It also changes how traces should be read. A small all-reduce that appears to take hundreds of microseconds may not be slow collective work; it may be absorbing cross-rank skew while another rank is late to launch. Aligning ranks in time is essential before assigning a kernel the blame.

Then we tightened the seams

Once the CPU was no longer pinned to GPU progress, the seams between graph replays became the next target. Those seams rebuild step-specific attention metadata—KV indices, block tables, and recurrent-state slots. The operations are tiny, but at batch 1 their dispatch cost adds up.

We reduced this overhead in two ways:

Fused chains that update cache locations and state indices while preserving the side effects required by later captured kernels.
Captured pure device-side metadata refills into a small CUDA Graph, keyed by batch size and forward mode.

We kept the capture boundary deliberately conservative. Device kernels that write persistent buffers are safe to capture; a path that invokes a mutable planning API is not. A FlashInfer-style plan(), for example, can rebuild internal state. Capturing it would buy a little latency at the cost of a stale-plan bug.

We then applied Programmatic Dependent Launch (PDL) across the MoE, router, KDA, and all-reduce chains. PDL allows a consumer kernel to begin producer-independent work before its producer has retired. The consumer still waits before reading producer output, but it can prefetch its own weights or set up its prologue during the producer’s execution.

PDL turns some launch and cold-weight latency into overlap. The driver determines how much of that potential overlap materializes under real resource pressure.

The key implementation rule is simple: put producer-independent loads before the dependency wait, and ensure every read of producer output remains after it. CUDA documents PDL as a scheduling opportunity rather than a deterministic overlap guarantee; the actual result depends on driver scheduling and resource availability. NVIDIA’s PDL guide is a useful reference for these semantics.

Three kernel changes that were worth the complexity

After that, the remaining work was mostly memory bandwidth and a long chain of small kernels. We made three targeted changes, with numerical correctness treated as a hard constraint rather than an afterthought.

A single-CTA moe_align for the decode shape. The generic MoE alignment path required two launches because it depended on a grid-wide scan. At small decode shapes, a pair-axis construction computes a stable per-bucket rank and count in one launch. We dispatch it only when the problem is small enough (numel ≤ 64); the pairwise approach scales quadratically and would spill badly at larger shapes.
SwiGLU fused into the MoE up-GEMM epilogue. Interleaving the gate and up-projection columns per expert lets the GEMM epilogue calculate silu(gate) × up directly. This removes a standalone activation launch and its intermediate write/read. Exactness required reproducing the existing fast-math behavior and avoiding an unintended extra bf16 rounding.
A KDA chain-verify retune. For the fused KDA verify kernel at the relevant six-token shape, choosing BV=4 on B200 reduced per-call time by 19%. It improved latency without changing the K-dimension reduction order, so it remained bitwise identical to the baseline.

The largest single gain came from a simple observation: precision is also a bandwidth knob. Moving the router gate and vocab-parallel lm_head from fp32 to bf16 delivered about 10%. At batch 1, dtype is not just a memory-footprint decision; it sits directly on the critical path.

DSpark: spend verification where it pays back

NEXTN uses a fixed draft-and-verify pattern. DSpark changes the question from “how much should we verify?” to “which tokens are worth verifying?”

Instead of always verifying a fixed token chain, DSpark drafts a block and creates a ragged verification plan for each request. Its planner uses calibrated draft confidence, the cumulative probability that a proposal survives to each position, and an offline-measured steps-per-second cost curve. The goal is expected tokens per second—not simply the largest possible draft.

DSpark uses calibrated confidence and a measured cost curve to decide how much of each draft block to verify.

That yields a much higher accept length: 9.95 in this benchmark versus 3.25 for tuned NEXTN. We also use a 1,319-example GSM8K run as an end-to-end regression guard against a no-spec baseline of the same target model; both must clear 0.90. It is a guardrail, not a proof of recurrent-state correctness. That confidence comes from the stage-then-commit design and byte-exact greedy tests.

The first DSpark trace uncovered the same kind of problem in a different place. FlashInfer planning received device tensors and internally performed blocking .to("cpu") operations for its index arrays. Each one pinned the host behind an in-flight draft graph. We instead built the plans from sequence lengths the host already knew, eliminating those D2H reads while keeping the captured graph’s persistent buffers valid.

With that pin removed, the CPU could enqueue draft replay, verification preparation, and verify replay back-to-back. The result on the 1,000-request run was 1,120 tok/s mean output throughput and 0.78 ms mean TPOT.

Accept length is workload-specific. All quoted results use SGLang’s random dataset with a fixed 8,192-token input and 1,024-token output. Treat them as a reproducible measurement for this deployment shape, not as a universal result for every prompt distribution. SGLang’s bench_serving documentation also makes an important distinction: TPOT excludes time to first token, while output throughput includes total benchmark wall time.

What is left when the host gets out of the way

In the final NEXTN profile, the host is hidden. Weight bandwidth, collectives, dense GEMMs, and small-kernel latency now dominate the next step.

Once host run-ahead was back, the conversation became a GPU one. MoE grouped GEMM and dense GEMM together account for about 2.1 ms of near-pure weight bandwidth in the final NEXTN profile. Small kernels and all-reduce still matter. MLA attention, by contrast, is only 244 µs at 8K context.

That leaves a fairly clear roadmap:

FP8 weights are the next large lever. Reducing bytes moved for bandwidth-bound work has a larger expected payoff than chasing benchmark noise.
Router fusion has a limit. Folding all work into one kernel can reduce parallelism too far; PDL has proven to be a better stopping point for this path.
Host environment tuning still matters. CPU core pinning and garbage-collection tuning protect the run-ahead slack that keeps GPU execution continuous.

The takeaway

The most useful outcome is not a single throughput number. It is a way to treat batch-1 inference as a first-class deployment target.

Start by separating host-induced idle from GPU-induced idle. Calibrate profiled traces against unprofiled runs before drawing conclusions about CPU timing. Measure cold-weight behavior, not just hot microbenchmarks. Keep a strict correctness gate, and make every change independently reversible. Then treat the system like the relay it is: model architecture, speculative-decoding policy, host preparation, graph replay, kernel dependencies, and GPU bandwidth all need to pass work on without a pause.

For interactive AI, that pause is the product.

We are continuing to share and upstream applicable parts of this work through the SGLang ecosystem. We welcome collaboration with serving-platform, hardware, and application teams building real-time agents and interactive AI products.

Reproduction notes

The figures above were measured with SGLang on 4× NVIDIA B200 GPUs, using TP4 and bf16 weights. TP4 means all four GPUs work together on each request. The benchmark uses --max-concurrency 1, meaning one request is in flight at a time. Each run uses 1,000 prompts from SGLang’s random dataset, with 8,192-token inputs and 1,024-token outputs. The final NEXTN and DSpark runs were collected on the same machine.

For DSpark, fit the confidence-calibration artifact and SPS cost table for the target deployment. Without them, the scheduler falls back to more conservative defaults.

Acknowledgments

This project was a joint effort between the Ant Ling Infra Team and RadixArk. We’d also like to thank DeepInfra for serving Ling-3.0-flash on SGLang.