SlipStream cuts high-bandwidth flash latency overheads in GPU systems

November 17, 2026

SlipStream cuts high-bandwidth flash latency overheads in GPU systems

Image: ChatGPT

Researchers from Georgia Tech, NVIDIA Research, and Stanford University have presented SlipStream, a software-hardware co-designed prefetcher aimed at making High-Bandwidth Flash usable in GPUgraphics processing unitA GPU is a parallel graphics processor widely used for AI acceleration. systems with lower overheads. The work addresses two main barriers: the roughly 20 μs read latency of HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. and the buffering cost needed to hide it.

HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. replaces the DRAM1. dynamic random access memoryMemory technology or system for temporarily or permanently storing digital information.2. dynamic random-access memoryDRAM is a volatile memory technology dies in HBM1. human body modelESD test model that simulates a person discharging static electricity into a component.2. High Bandwidth MemoryHBM is a stacked DRAM architecture designed to provide very high memory bandwidth. with NAND1. Not ANDA logic gate that outputs 0 when all inputs are 1, and outputs 1 in all other cases2. negative-ANDNAND denotes a logic operation and is also the name of a widely used flash memory type. flash and can deliver up to 16x the capacity at comparable read bandwidth. That capacity comes with a latency penalty of nearly 20x versus HBM1. human body modelESD test model that simulates a person discharging static electricity into a component.2. High Bandwidth MemoryHBM is a stacked DRAM architecture designed to provide very high memory bandwidth., and on-demand accesses can slow inference by as much as 12.8x. Conventional cache-line prefetchers would need about 20x more in-flight tracking state to cover that delay, while tensor-streaming approaches such as H3tensor-streaming architectureA tensor-streaming architecture organizes data movement for tensor processing and relies on buffering along the access order. require hundreds of megabytes of buffering because they follow virtual-address order rather than kernel-demand order.

SlipStream changes prefetching into a demand-paced producer-consumer pipeline. In software, it profiles the repeatable weight-page access order of data-independent LLM1. local linear models2. Large Language ModelThe model class referenced in the long-context serving demo. kernels and inserts a staging warp that replays that order with non-blocking STAGE1. staging instructionA staging instruction triggers the early movement of data within a memory hierarchy.2. non-blocking staging instructionA non-blocking staging instruction initiates data staging without immediately blocking execution. instructions. In hardware, the staging warp acts as the producer and the work warps act as consumers.

A central idea is to retire each STAGE1. staging instructionA staging instruction triggers the early movement of data within a memory hierarchy.2. non-blocking staging instructionA non-blocking staging instruction initiates data staging without immediately blocking execution. tracking entry only when demand first reaches the page, not when the page returns from HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems.. That keeps staging aligned with actual consumption and limits how far ahead prefetching can run. The design tracks 4 KB pages instead of cache lines and allocates buffer slots only when pages return, which decouples buffer size from outstanding traffic.

Across seven dense models ranging from 16 GB to 800 GB, SlipStream with a 40 MBmachine batch staging buffer reaches 92.4 % of the performance of an ideal HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. system with HBM1. human body modelESD test model that simulates a person discharging static electricity into a component.2. High Bandwidth MemoryHBM is a stacked DRAM architecture designed to provide very high memory bandwidth.-like latency. By comparison, H3tensor-streaming architectureA tensor-streaming architecture organizes data movement for tensor processing and relies on buffering along the access order. reaches 43.1 % even with a 320 MBmachine batch buffer.

The buffer sizing is independent of both HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. latency and workload. A 16 MBmachine batch buffer, 20x smaller than H3tensor-streaming architectureA tensor-streaming architecture organizes data movement for tensor processing and relies on buffering along the access order.'s, still performs within 4 % of the 40 MBmachine batch configuration. SlipStream also supports Mixture-of-Experts models and reaches 91.2 % of ideal performance with the 40 MBmachine batch buffer.

The paper is titled SlipStream: Enabling HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. for GPUgraphics processing unitA GPU is a parallel graphics processor widely used for AI acceleration. Systems by Mitigating the Latency and SRAM1. static random access memoryMemory technology or system for temporarily or permanently storing digital information.2. static random-access memorySRAM is used here as a test vehicle for embedded memory performance. Overheads and appears as an arXiv preprint from November 2026.

Source: Semiconductor Engineering

More background: How does it work?

← All posts