Researchers from Huawei, ETH Zürich and HUST have described FLINT, a system that uses High Bandwidth Flash as a near-accelerator capacity tier for large language model inference. The work targets systems where inference is limited less by compute throughput than by on-package memory capacity, especially single-accelerator and small-node setups.
The paper positions HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. as an emerging 3D-stacked NANDNot ANDA logic gate that outputs 0 when all inputs are 1, and outputs 1 in all other cases flash technology with multi-terabyte capacity, suitable for storing model weights that do not fit in HBM1. human body modelESD test model that simulates a person discharging static electricity into a component.2. High Bandwidth MemoryHBM is a stacked DRAM architecture designed to provide very high memory bandwidth.. At the same time, the authors identify three obstacles in earlier HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. proposals: coarse static prefetching for weights, flash-management work that remains on the critical inference path, and a lack of specialization of flash management for workload behavior.
FLINT addresses these issues with three mechanisms. A hardware burst-buffer controller dynamically coalesces and pipelines HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. reads so existing NANDNot ANDA logic gate that outputs 0 when all inputs are 1, and outputs 1 in all other cases flash buffers can be used while maintaining high bandwidth. A phantom-plane refresh mechanism moves refresh-related operations out of the read foreground through low-cost resource duplication, removing refresh from the critical path. In addition, a read-only FTL replaces SSD1. Solid State Drivenon-volatile electronic storage medium, usually a hard disk drive in computers2. Solid-State DriveA non-volatile storage device used in the rack demo to hold full KV caches.-style support for arbitrary writes with a compact table that maps logical weight bursts to physical HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. locations.
The result is an HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. substrate that is designed specifically around LLM1. local linear models2. Large Language ModelThe model class referenced in the long-context serving demo. inference access patterns rather than general SSD1. Solid State Drivenon-volatile electronic storage medium, usually a hard disk drive in computers2. Solid-State DriveA non-volatile storage device used in the rack demo to hold full KV caches. behavior. By combining HBF1. High Bandwidth FlashA NAND-based concept that combines storage capacity with higher bandwidth.2. High-Bandwidth FlashA memory concept under development that is intended to relieve bandwidth and capacity bottlenecks in AI inference systems. with HBM1. human body modelESD test model that simulates a person discharging static electricity into a component.2. High Bandwidth MemoryHBM is a stacked DRAM architecture designed to provide very high memory bandwidth. in a capacity-scalable memory hierarchy, FLINT focuses on keeping flash-management overhead away from the accelerator-visible inference path while preserving the bandwidth needed to serve weight reads.