# \[RFC-0036\] Zero-GC 64-Byte Cache-Aligned Flat Arena for Speculative Decoding Verification **URL:** \ **Category:** compiler\ **Created:** [September 21, 2026, 4:26pm UTC](https://dev-discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/3441 "2026-09-21T16:26:22Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![markbgilbert](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/markbgilbert/32/3343_2.png) [@markbgilbert](https://dev-discuss.pytorch.org/u/markbgilbert)\ **Post date:** [September 21, 2026, 4:26pm UTC](https://dev-discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/3441/1 "2026-09-21T16:26:22Z")
We have submitted **[RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena Runtime](https://github.com/pytorch/rfcs/pull/110)** to address host-side CPU overhead and allocator contention in high-throughput LLM speculative decoding and batch token ingestion. ### Empirical Progression: From Prototype to Native Silicon Following our initial prototype discussion, we compiled the 64-byte aligned flat arena directly to native C/AVX2 on physical AMD Zen 5 silicon, dropping execution time for 1 Billion operations from 644 ms down to ~200 ms (5 Billion ops/sec, sub-0.3 ns latency) with zero runtime heap allocations. | Metric | Phase 1: Prototype (V8 JavaScript) | Phase 2: Refined Production (Native C / AVX2) | Hardware Advantage | | --- | --- | --- | --- | | **Runtime Execution** | Node.js v20.x (V8 JIT Engine) | **Pure C (Clang 18.0.2, `-O3 -mavx2`)** | **Zero runtime / Pure Silicon** | | **Memory Structure** | V8 Heap TypedArray / In-Place | **64-Byte Aligned Struct (`QueueOrder64`)** | **Hardware L1 Cache Resident** | | **Workload** | 1,000,000,000 Operations (1B) | **1,000,000,000 Operations (1B)** | 100% Workload Parity | | **Execution Duration** | 643.80 ms (~0.64 s) | **200.00 ms to 276.79 ms (~0.20 s)** | **3.25x FASTER (Zero JIT / Zero GC)** | | **Throughput** | 1.55 Billion ops/sec | **3.61 to 5.00 Billion ops/sec** | **+2.06B to +3.45B ops/sec higher** | | **Amortized Latency** | 0.644 ns / op | **0.200 to 0.277 ns / op** | **Sub-0.3 nanosecond execution** | | **Dynamic Allocations** | 14 KB heap delta | **0 bytes dynamic heap allocation** | **100% Deterministic Flat Buffer** | ### 60-Minute Dual-OS Soak Receipts (Windows 11 vs. Ubuntu 24.04 LTS) We also completed sustained 60-minute Dual-OS enterprise soak tests across Windows 11 and Ubuntu 24.04 LTS (up to 29,711 steps and 486M tokens processed), proving 0.00 MB PyTorch VRAM drift across 2,300+ consecutive steps. The test mathematically separates OS driver page table quantization (5x 1.00 MB WDDM paging jumps vs. 9x 0.25 MB glibc ptmalloc sub-arenas) from application memory stability. On Karpathy Micro-GPT (10.69M parameters, Blackwell RTX 5060), host batch token ingestion dropped from 997.70 us (PyTorch DataLoader) down to 7.32 us (136.3x speedup) with an 82.5% reduction in host RAM. We welcome feedback from the PyTorch Core, `torch.compile` (Inductor), and ExecuTorch maintainers. - Full RFC and discussion: [RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena (Aventine Labs LLC) by markbgilbert · Pull Request #110 · pytorch/rfcs · GitHub](https://github.com/pytorch/rfcs/pull/110) - Standalone reproduction repository: [GitHub - markbgilbert/aegis-zero-gc-benchmark: Empirical 1-Billion Operation Zero-GC Flat Arena Benchmark (Native C11 AVX2 + C++17 C10 Operator) · GitHub](https://github.com/markbgilbert/aegis-zero-gc-benchmark) ---
**Author:** ![discourse\_ai\_spam](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@discourse\_ai\_spam](https://dev-discuss.pytorch.org/u/discourse_ai_spam)\ **Post date:** [September 21, 2026, 4:26pm UTC](https://dev-discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/3441/2 "2026-09-21T16:26:26Z")
---
**Author:** ![system](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/1X/6cd7da56682d360e2c6006ff3e31eb250c5a8675.png) [@system](https://dev-discuss.pytorch.org/u/system)\ **Post date:** [September 23, 2026, 1:16pm UTC](https://dev-discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/3441/3 "2026-09-23T13:16:58Z")