# Investigation: Why Two-Token MTP Outpaced Autoregressive Decoding Without a Faster Dominant Kernel

**URL:** <https://dev-discuss.pytorch.org/t/investigation-why-two-token-mtp-outpaced-autoregressive-decoding-without-a-faster-dominant-kernel/3451>\
**Category:** performance\
**Created:** [September 29, 2026, 11:02am UTC](https://dev-discuss.pytorch.org/t/investigation-why-two-token-mtp-outpaced-autoregressive-decoding-without-a-faster-dominant-kernel/3451 "2026-09-29T11:02:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![suwesh](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/suwesh/32/3345_2.png) [@suwesh](https://dev-discuss.pytorch.org/u/suwesh)\
**Post date:** [September 29, 2026, 11:02am UTC](https://dev-discuss.pytorch.org/t/investigation-why-two-token-mtp-outpaced-autoregressive-decoding-without-a-faster-dominant-kernel/3451/1 "2026-09-29T11:02:53Z")

</div>

I compared autoregressive decoding with two-token MTP in a controlled vLLM deployment and observed a 1.91x to 2.19x increase in output throughput. The dominant MTP GEMM was not faster than the dominant AR GEMV. Instead, MTP made greater token progress while requiring substantially fewer recurring GPU executions per generated token.

I followed this result through runtime acceptance telemetry, PyTorch Profiler, NVIDIA Nsight Systems, and Nsight Compute. I am sharing the investigation both as a performance report and to request feedback on the interpretation of the PyTorch runtime hierarchy and its relationship to recurring CUDA Graph executions.

**Full preprint:**

> **[Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction...](https://arxiv.org/abs/2609.35188)**
>
> Autoregressive large language model inference repeatedly invokes the target model to generate one token at a time, making generation sensitive to GPU memory movement and sequential execution. This study evaluates two-token multi-token prediction...

---

<div class="post-metadata">

**Author:** ![colorfulcms](https://avatars.discourse-cdn.com/v4/letter/c/8491ac/32.png) [@colorfulcms](https://dev-discuss.pytorch.org/u/colorfulcms)\
**Post date:** [October 25, 2026, 12:49pm UTC](https://dev-discuss.pytorch.org/t/investigation-why-two-token-mtp-outpaced-autoregressive-decoding-without-a-faster-dominant-kernel/3451/2 "2026-10-25T12:49:36Z")

</div>

Interesting write-up. Could you clarify whether the two-token MTP and AR runs used identical CUDA Graph capture sizes and batch sizes? I’m wondering how much of the throughput gain comes from fewer graph launches per token versus the acceptance rate at higher concurrency.