MemVerge and Micron Boost NVIDIA GPU Utilization with CXL® Memory

The joint solution increases GPU utilization by 77% and more than doubles the speed of OPT-66B batch inference.

San Jose, CA – March 18, 2024 – MemVerge®, a leader in AI-first Big Memory Software, has joined forces with Micron to unveil a groundbreaking solution that leverages intelligent tiering of CXL memory, boosting the performance of large language models (LLMs) by offloading from GPU memory to CXL memory. This innovative collaboration is being showcased in Micron booth #1030 at GTC, where attendees can witness firsthand the transformative impact of tiered memory on AI workloads.

Charles Fan, CEO and Co-founder of MemVerge, emphasized the critical importance of overcoming the bottleneck of GPU memory capacity. “Scaling LLM performance cost-effectively means keeping the GPUs fed with data,” stated Fan. “Our demo at GTC demonstrates that pools of tiered memory not only drive performance higher but also maximize the utilization of precious GPU resources.”

The demonstration, conducted by engineers from MemVerge and Micron featured a FlexGen high-throughput generation engine and OPT-66B large language model running on a Supermicro Petascale Server equipped with an AMD Genoa CPU, Nvidia A10 GPU, Micron DDR5-4800 DIMMs, CZ120 CXL memory modules, and MemVerge Memory Machine™ X intelligent tiering software.

The results of the demonstration were impressive. The FlexGen benchmark, utilizing tiered memory, completed tasks in less than half the time compared to conventional NVMe storage methods. Simultaneously, GPU utilization soared from 51.8% to 91.8%, thanks to the transparent management of data tiering across GPU, CPU and CXL memory facilitated by MemVerge Memory Machine X software.

This collaboration between MemVerge, Micron, and Supermicro marks a significant milestone in advancing the capabilities of AI workloads, enabling organizations to achieve unprecedented levels of performance, efficiency, and time-to-insight. By harnessing the power of CXL memory and intelligent tiering, businesses can unlock new opportunities for innovation and accelerate their journey towards AI-driven success.

“Through our collaboration with MemVerge, Micron is able to demonstrate the substantial benefits of CXL memory modules to improve effective GPU throughput for AI applications resulting in faster time to insights for customers. Micron’s innovations across the memory portfolio provide compute with the necessary memory capacity and bandwidth to scale AI use cases from cloud to the edge,” said Raj Narasimhan, senior vice president and general manager of Micron’s Compute and Networking Business Unit.