Fixed-Point Implementation Analysis of MLSE Receiver DSP for High-Speed Wireline Transceivers

Abstract

This paper analyzes the fixed-point (FXP) implementation of the maximum-likelihood sequence estimation (MLSE) engine for high-speed PAM-4 wireline transceivers (TRXs). Quantization effects in the ADC/FFE output, branchmetric (BM), and path-metric (PM) computations are systematically evaluated through end-to-end BER simulations at 118Gb/s over a 28.5dB-loss channel. Comprehensive precision sweeps show that insufficient quantization at early DSP stages, particularly in the FFE output and BM computation, causes irreversible BER degradation even when later stages maintain high precision. The analysis identifies the dominant precision bottlenecks and provides quantitative design guidelines for balancing BER performance and hardware efficiency in MLSE receiver DSP implementations for future high-speed wireline transceivers. © 2026 IEEE.