I compared autoregressive decoding with two-token MTP in a controlled vLLM deployment and observed a 1.91x to 2.19x increase in output throughput. The dominant MTP GEMM was not faster than the dominant AR GEMV. Instead, MTP made greater token progress while requiring substantially fewer recurring GPU executions per generated token.
I followed this result through runtime acceptance telemetry, PyTorch Profiler, NVIDIA Nsight Systems, and Nsight Compute. I am sharing the investigation both as a performance report and to request feedback on the interpretation of the PyTorch runtime hierarchy and its relationship to recurring CUDA Graph executions.
Interesting write-up. Could you clarify whether the two-token MTP and AR runs used identical CUDA Graph capture sizes and batch sizes? I’m wondering how much of the throughput gain comes from fewer graph launches per token versus the acceptance rate at higher concurrency.
Thanks for the question. This is an important distinction. I did not manually configure the CUDA Graph capture sizes ( cudagraph_capture_sizes and max_cudagraph_capture_size). Both AR and MTP used vLLM’s default graph-capture configuration under the same serving setup. The profiler and Nsight captures used one active request, so the observed decode batch size was 1 in both cases.
So, my interpretation here is limited to batch size 1. I did not test different batch sizes or higher concurrency, so I cannot separate those effects from the acceptance-driven reduction in recurring GPU executions. I agree that this would be a useful follow-up experiment.