Ted Hirokawa
January 31, 2025 06:38
NVIDIA’s NCCL 2.23 release introduces new scaling algorithms, acceleration initialization and profiler Russian APIs to optimize GPUs and multi -node communication between AI and HPC applications.
The latest release of NVIDIA Collective Communications Library (NCCL) 2.23 introduces improvement products that aims to optimize GPUs and multi -node communications essential for artificial intelligence (AI) and high -performance computing (HPC) applications. According to NVIDIA, this improvement is designed to improve the efficiency and scalability of parallel computing.
Highlights and functions release
NCCL 2.23 release shows some major innovations.
- Parallel Tree Tree (PAT) Algorithm: A new algorithm for reducescatter and allgather tasks that provide log scaling to improve the performance of small and medium -sized messages.
- Acceleration initialization: Improving performance by using networking in bands for boot strap communication, new performance, new
ncclCommInitRankScalableAPI. - Intra Node User Buffer Registration: Reduces memory sub -system pressure and improves communication overlap to provide performance improvement.
- New Profiler Plugin API: It provides API hooks that measure microfolves NCCL performance and improve diagnostic function.
PAT algorithm and initialization improvement
Inspired by the Bruck algorithm, the PAT algorithm minimizes buffering demands, enabling efficient communication of various network sizes. This improvement is particularly advantageous for large language model education, where pipelines and tensor parallel processing are important.
that ncclCommInitRankScalable The API allows multiple unique IDs to facilitate expandable initialization, relieving bottlenecks related to all communication patterns in large -scale tasks.
Intra node user buffer registration
NCCL 2.23 supports intra node user buffer registration, supporting data transmission optimization through NVLINK and PCIe. This feature uses a registered user buffer that is automatically registered during the CUDA graph capture to reduce overhead and improve performance.
Profiler Plugin API
The new profiler plug -in API is increasing demand for monitoring tools for each domain in a wide range of GPU clusters. This API helps to detect performance abnormalities and optimize resource allocation by enabling profiling of NCCL events.
conclusion
NVIDIA’s NCCL 2.23 promises to enhance utilities in AI and HPC domains by greatly improving the performance and scalability of GPU communication by introducing these advanced features. For more information about these updates, visit the official NVIDIA blog.
Image Source: Shutter Stock