AIartificial intelligenceAI stands for artificial intelligence inference is pushing memory architecture back into the spotlight as systems move from training in data centers to deployment at the edge. The central goal is to reduce how far data must travel between memory and compute, because that movement costs power, time, and bandwidth.
The shift is especially relevant for workloads that stream sensor data in physical AIartificial intelligenceAI stands for artificial intelligence, as well as for battery-electric vehicles and drones, where every watt saved on data handling can be spent elsewhere. In these systems, the key problem is no longer just adding capacity, but deciding what data must be kept, what can be discarded, and what has to be acted on immediately.
Different inference stages place different demands on the memory subsystem. Low-batch decode can be a good fit for near-memory approaches, while large-batch decode and prefill can become constrained by activation traffic and bandwidth. That means the execution flow has to be split into parts that are storage-sensitive, bandwidth-sensitive, or batching-sensitive.
Several architectural options are emerging around this idea, including processing-near-memory (PNMProcessing-Near-MemoryAn architecture that places compute close to memory to reduce data movement.), processing-in-memory (PIM1. Processing-In-MemoryAn approach that performs some computation close to or inside memory.2. processor-in-memoryPIM describes computing directly inside memory to reduce data movement.), and in-memory computing (IMC). In one formulation, custom high-bandwidth memory places compute in the same package, while LPDDR6-PIM1. Processing-In-MemoryAn approach that performs some computation close to or inside memory.2. processor-in-memoryPIM describes computing directly inside memory to reduce data movement. places logic alongside DRAM1. dynamic random access memoryMemory technology or system for temporarily or permanently storing digital information.2. dynamic random-access memoryDRAM is a volatile memory technology banks. The article also notes compute-in-memory (CIM1. computer-integrated manufacturing2. compute-in-memoryCiM reduces data-movement overhead by processing data near or inside memory.) as a closely related term, while also pointing out that IMC can mean something different in enterprise software.
Industry participants describe the same basic trade-off: less data movement and higher energy efficiency, but with the CPUcentral processing unitMain processor that executes program instructions and performs data operations. or GPUgraphics processing unitA GPU is a parallel graphics processor widely used for AI acceleration. no longer doing all of the work. Near-memory processing can offload simple MACmultiply-accumulateMAC is the basic operation used for AI inference and training workloads. operations from the host, while more complex computation still remains with the accelerator. The memory system therefore becomes part of the compute path rather than just a passive store.
That architectural change also shifts pressure onto the software stack. Compilers, operating systems, and system software need to understand which operations can be dispatched to memory and which still have to stay local. The text also highlights the need for standardization, since today’s infrastructure is largely built around processor cores and accelerators doing all of the work inside the SoC.
Thermal and manufacturing issues remain important barriers. Mixing logic and DRAM1. dynamic random access memoryMemory technology or system for temporarily or permanently storing digital information.2. dynamic random-access memoryDRAM is a volatile memory technology process technologies increases complexity and cost, and the software side may be even harder because scheduling, load balancing, coherence, and synchronization become more complicated when compute is coupled to memory access.
Several timeline expectations are still measured in years rather than months. In-memory applications are framed as a three- to five-year prospect, with around 2030 mentioned as a possible point where the approach may become more visible. Samsung’s LPDDR6-PIM1. Processing-In-MemoryAn approach that performs some computation close to or inside memory.2. processor-in-memoryPIM describes computing directly inside memory to reduce data movement. effort is cited as an example of a system aimed at low-power, mobile on-device AIartificial intelligenceAI stands for artificial intelligence, while broader commercialization is still described as early.