We have submitted RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena Runtime to address host-side CPU overhead and allocator contention in high-throughput LLM speculative decoding and batch token ingestion.
Empirical Progression: From Prototype to Native Silicon
Following our initial prototype discussion, we compiled the 64-byte aligned flat arena directly to native C/AVX2 on physical AMD Zen 5 silicon, dropping execution time for 1 Billion operations from 644 ms down to ~200 ms (5 Billion ops/sec, sub-0.3 ns latency) with zero runtime heap allocations.
| Metric | Phase 1: Prototype (V8 JavaScript) | Phase 2: Refined Production (Native C / AVX2) | Hardware Advantage |
|---|---|---|---|
| Runtime Execution | Node.js v20.x (V8 JIT Engine) | Pure C (Clang 18.0.2, -O3 -mavx2) |
Zero runtime / Pure Silicon |
| Memory Structure | V8 Heap TypedArray / In-Place | 64-Byte Aligned Struct (QueueOrder64) |
Hardware L1 Cache Resident |
| Workload | 1,000,000,000 Operations (1B) | 1,000,000,000 Operations (1B) | 100% Workload Parity |
| Execution Duration | 643.80 ms (~0.64 s) | 200.00 ms to 276.79 ms (~0.20 s) | 3.25x FASTER (Zero JIT / Zero GC) |
| Throughput | 1.55 Billion ops/sec | 3.61 to 5.00 Billion ops/sec | +2.06B to +3.45B ops/sec higher |
| Amortized Latency | 0.644 ns / op | 0.200 to 0.277 ns / op | Sub-0.3 nanosecond execution |
| Dynamic Allocations | 14 KB heap delta | 0 bytes dynamic heap allocation | 100% Deterministic Flat Buffer |
60-Minute Dual-OS Soak Receipts (Windows 11 vs. Ubuntu 24.04 LTS)
We also completed sustained 60-minute Dual-OS enterprise soak tests across Windows 11 and Ubuntu 24.04 LTS (up to 29,711 steps and 486M tokens processed), proving 0.00 MB PyTorch VRAM drift across 2,300+ consecutive steps. The test mathematically separates OS driver page table quantization (5x 1.00 MB WDDM paging jumps vs. 9x 0.25 MB glibc ptmalloc sub-arenas) from application memory stability.
On Karpathy Micro-GPT (10.69M parameters, Blackwell RTX 5060), host batch token ingestion dropped from 997.70 us (PyTorch DataLoader) down to 7.32 us (136.3x speedup) with an 82.5% reduction in host RAM.
We welcome feedback from the PyTorch Core, torch.compile (Inductor), and ExecuTorch maintainers.
- Full RFC and discussion: RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena (Aventine Labs LLC) by markbgilbert · Pull Request #110 · pytorch/rfcs · GitHub
- Standalone reproduction repository: GitHub - markbgilbert/aegis-zero-gc-benchmark: Empirical 1-Billion Operation Zero-GC Flat Arena Benchmark (Native C11 AVX2 + C++17 C10 Operator) · GitHub