NDPool: Correctness-Preserving Shared Execution for Efficient LLM Inference on CXL-NDP Systems

Abstract

The inference performance of Large Language Models (LLMs) is increasingly constrained by memory capacity and bandwidth rather than raw computation, particularly for long-context workloads. CXL-based Near-Data Processing (CXL-NDP) has emerged as a promising solution for expanding memory capacity, but existing systems suffer from severe resource under-utilization due to exclusive device allocation under hybrid parallel execution. To minimize per-request latency, prior designs dedicate multiple NDP devices to tensor parallelism for FC layers, while non-FC layers are executed only on a single master device. As a result, most auxiliary devices remain idle for a large fraction of execution time, fundamentally limiting overall system throughput. Safe device sharing is challenging because out-of-order peer-to-peer communication can break request isolation and computational correctness at the shared CXL receive path. To address this limitation, we propose NDPool, a correctness-preserving shared execution framework that dynamically reuses idle TP devices across concurrent requests for FC-layer execution. NDPool combines ordered receive control for safe data delivery with adaptive scheduling to reduce execution bubbles and maximize shared-device utilization. Experimental results show that, under equivalent service-level objective (SLO) constraints, NDPool improves system throughput by up to 3.7 & times; and by 1.5 & times; on a geometric mean basis over the state-of-the-art CXL-NDP hybrid parallel design.