Not a HBM Replacement: SanDisk HBF Complements DRAM for AI Inference

1. Introduction

When SanDisk introduced High-Bandwidth Flash (HBF), much of the initial industry discourse framed it as a potential challenger to High-Bandwidth Memory (HBM). Headlines asked whether HBF could "replace" HBM, "disrupt" the DRAM market, or render HBM obsolete. This framing, while attention-grabbing, fundamentally misunderstands both technologies.

The reality is more nuanced — and more interesting. HBF is not a replacement for HBM. It is a complementary memory tier designed to work alongside DRAM, not against it. The two technologies address different aspects of the memory problem: HBM delivers unmatched latency and bandwidth for compute-critical data, while HBF delivers unmatched capacity and cost-efficiency for data that must be "near" the GPU but does not require nanosecond access.

This distinction is especially important for AI inference, the workload that is rapidly becoming the dominant driver of AI infrastructure investment. Inference workloads have a unique memory profile — one that is overwhelmingly read-dominated, capacity-hungry, and latency-tolerant in ways that training is not. It is precisely this profile that makes HBF such a natural complement to DRAM.

This article provides a comprehensive analysis of why HBF should be understood as a complement to HBM, not a replacement — and why this complementary relationship is the key to unlocking more efficient, more scalable AI inference.

2. The Replacement Misconception

The idea that HBF could replace HBM stems from a category error — conflating two memory technologies that operate at fundamentally different points in the performance spectrum. Understanding why this misconception is wrong requires examining the physics and economics of each technology.

2.1 The Latency Chasm

The most fundamental difference between HBM and HBF is latency. HBM, built on DRAM, delivers access latency measured in nanoseconds. HBF, built on NAND flash, delivers latency measured in microseconds. This is a difference of roughly 1000x — a chasm that no amount of architectural cleverness can fully bridge. For workloads that require frequent, small, random memory accesses at extreme speed, HBM is simply irreplaceable.

2.2 Different Problems, Different Solutions

HBM and HBF solve different problems. HBM solves the bandwidth-and-latency problem: how to move data to the compute engine as fast as possible. HBF solves the capacity-and-cost problem: how to keep enormous amounts of data close to the compute engine without breaking the bank. These are complementary, not competing, objectives.

2.3 The False Either/Or Framing

The "replacement" narrative forces a false choice: either HBM or HBF. But the reality of modern system design is that memory hierarchies are additive. Just as the introduction of SSD did not replace DRAM but rather complemented it, HBF will not replace HBM — it will complement it. The question is not "which one?" but "how do they work together?"

3. Distinct Roles in the Memory Hierarchy

To appreciate the complementary relationship, it helps to map each technology to its proper place in the memory hierarchy. Each tier has a distinct role defined by its latency, bandwidth, capacity, and cost characteristics.

3.1 HBM: The Compute-Critical Tier

HBM sits at the top of the off-chip memory hierarchy, directly adjacent to the compute die. Its role is to hold the data that the GPU needs right now — the activations, intermediate results, and frequently-accessed data that must be available at nanosecond latency. HBM's 24–36 GB capacity is deliberately small because its value lies in speed, not volume.

3.2 HBF: The Capacity-Abundant Tier

HBF sits one tier below HBM, providing terabyte-scale capacity (1–4 TB per module) at a fraction of the cost. Its role is to hold the large, read-mostly datasets that AI inference requires — model weights, embeddings, and KV-caches — that are too large for HBM but must remain "near" the GPU to avoid the catastrophic performance penalty of accessing bulk storage.

3.3 The Complete Hierarchy

Together, the tiers form a complete system:

Each tier is optimized for a specific purpose. Removing any tier — or trying to make one tier do another's job — degrades the entire system. HBM and HBF are not interchangeable; they are complementary by design.

4. The AI Inference Workload: A Unique Memory Profile

The complementary value of HBF is most apparent when examining the specific characteristics of AI inference workloads. Inference has a memory profile that is fundamentally different from training — and it is this profile that makes HBF so valuable.

4.1 Read-Dominated Access Patterns

Inference is overwhelmingly read-dominated. Once a model is trained, its weights are fixed and read repeatedly for every inference request. This maps perfectly to NAND flash's strengths — NAND excels at high-speed reads but has limited write endurance. For inference, where writes are rare and reads are constant, HBF's endurance limitations are largely irrelevant.

4.2 Capacity-Hungry, Not Bandwidth-Hungry

Modern inference workloads are increasingly capacity-bound rather than bandwidth-bound. A trillion-parameter model requires ~500 GB to 1 TB of memory for weights alone, plus additional capacity for KV-cache and embeddings. This far exceeds HBM's capacity, but does not necessarily require HBM's bandwidth. The bottleneck is space, not speed.

4.3 The KV-Cache Challenge

Long-context inference generates KV-caches that grow linearly with sequence length. For 1M-token contexts, the KV-cache alone can consume hundreds of gigabytes. This is a pure capacity problem — the data must be stored and accessible, but it does not require nanosecond latency. HBF's terabyte capacity is ideally suited to this challenge.

4.4 Cost Sensitivity

Inference is a cost-sensitive workload. Unlike training, which is a one-time capital expense, inference is an ongoing operational cost that scales with usage. Every dollar saved on memory directly improves the economics of serving AI. HBF's 10–20x cost advantage per bit makes it the natural choice for the capacity tier of inference systems.

5. How HBF Complements DRAM in Practice

The complementary relationship between HBF and DRAM is not theoretical — it manifests in concrete architectural decisions and measurable performance improvements. Here is how the two technologies work together in a real inference system.

5.1 Intelligent Data Partitioning

The key to effective complementarity is intelligent data partitioning. The system places latency-critical data — active activations, current computation state, frequently-accessed small data — in HBM. It places capacity-hungry data — model weights, embeddings, KV-cache — in HBF. This division ensures that each technology is used for what it does best.

5.2 Reducing HBM Pressure

By offloading the bulk of the working set to HBF, the system dramatically reduces the pressure on HBM. This has two benefits: first, it frees HBM capacity for the latency-critical data that truly needs it; second, it reduces the amount of HBM required per accelerator, lowering the bill of materials and easing the HBM supply crunch.

5.3 The Prefetching Bridge

The latency gap between HBM and HBF is bridged through prefetching. Because inference access patterns are highly predictable (the model reads weights in a known sequence), the system can prefetch data from HBF into HBM before it is needed. This hides the HBF latency and allows the GPU to operate as if all data were in HBM — at a fraction of the cost.

5.4 The Whole Greater Than the Sum

The result is a system that is greater than the sum of its parts. HBM provides the speed, HBF provides the capacity, and the combination delivers performance and cost-efficiency that neither could achieve alone. This is the essence of complementarity.

6. The Complementary Architecture in Detail

Let us examine the specific architectural design of a complementary HBM + HBF inference system, and how each component contributes to the whole.

6.1 The HBM Layer: Speed for the Critical Path

HBM remains the workhorse for the compute-critical path. It holds the active activations, the current layer's data, and the small, frequently-accessed buffers that the GPU needs at nanosecond latency. Its 24–36 GB capacity is sufficient for this role because the critical path data is small — it is the bulk data that is large.

6.2 The HBF Layer: Capacity for the Working Set

HBF holds the large, read-mostly working set: model weights, embeddings, and KV-cache. Its terabyte-scale capacity means the entire model can be resident in near-GPU memory, eliminating the need to fetch data from slow bulk storage. Its multi-TB/s aggregate bandwidth ensures that data can be moved to HBM quickly when needed.

6.3 The Interconnect: The Glue That Binds

The connection between HBM and HBF is critical. Whether through a wide on-package interface, CXL, or a proprietary link, the interconnect must provide sufficient bandwidth to move data from HBF to HBM quickly enough to hide the latency. This is where the "High-Bandwidth" in HBF earns its name — it is not just about capacity, but about the ability to deliver that capacity at speed.

6.4 The Software Orchestration Layer

The final component is software. A complementary HBM+HBF system requires intelligent memory management that decides which data goes where, when to prefetch, and how to handle misses. This orchestration layer is the key to making the complementary architecture work seamlessly — and it is also the area that requires the most development effort.

7. Real-World Benefits of the Complementary Approach

The complementary HBM+HBF architecture delivers concrete, measurable benefits for AI inference. These benefits span performance, cost, and scalability.

7.1 Lower Total Cost of Ownership

By reducing HBM requirements and leveraging cheaper NAND for bulk capacity, the complementary approach can reduce memory-related costs by 40–60% for inference workloads. This directly improves the economics of AI serving, making inference more affordable and more widely accessible.

7.2 Higher GPU Utilization

When the entire model working set is resident in near-GPU memory (HBF), the GPU no longer stalls waiting for data from slow storage. This dramatically improves GPU utilization, allowing the same hardware to serve more requests, more efficiently.

7.3 Support for Larger Models

The complementary architecture enables inference on models that are simply too large for HBM alone. Trillion-parameter models, long-context processing, and multi-model serving all become feasible when terabyte-scale HBF is available.

7.4 Easing the HBM Supply Crunch

Perhaps most importantly, the complementary approach reduces the industry's dependence on scarce HBM. By offloading capacity to HBF, the demand for HBM per accelerator decreases, easing the global HBM supply crunch and making AI infrastructure more scalable.

8. Challenges and Considerations

While the complementary approach is compelling, it is not without challenges. A balanced assessment requires acknowledging the hurdles that must be overcome.

8.1 Software Complexity

The complementary architecture requires sophisticated software to manage data placement, prefetching, and tiering. This software is not yet mature, and developing it is a significant engineering effort. Until robust frameworks and drivers exist, the benefits of complementarity will be limited to early adopters.

8.2 Prefetching Predictability

The effectiveness of the HBM+HBF architecture depends on the predictability of access patterns. While inference is generally predictable, some workloads (dynamic batching, irregular attention patterns) are less so. For these workloads, the latency gap between HBM and HBF may become more visible.

8.3 Endurance Management

While inference is read-dominated, there are still writes — KV-cache updates, logging, and occasional weight updates. HBF systems must manage these writes carefully to avoid premature wear. This adds complexity to the memory management layer.

8.4 Ecosystem Maturity

The broader ecosystem — from hardware vendors to framework developers to cloud providers — must embrace the complementary architecture for it to reach its full potential. This requires standardization, interoperability, and industry-wide collaboration that takes time to develop.

9. Conclusion

The framing of HBF as a "HBM replacement" is fundamentally misguided. HBF and HBM are not competitors vying for the same role; they are complementary technologies that together form a complete memory hierarchy capable of meeting the demands of modern AI inference.

HBM remains essential for the compute-critical path, delivering the nanosecond latency and extreme bandwidth that active computation demands. HBF provides the terabyte-scale capacity that AI inference requires, holding model weights, embeddings, and KV-caches in near-GPU memory at a fraction of HBM's cost. Together, they enable a system that is faster, cheaper, and more scalable than either technology could achieve alone.

The future of AI inference is not a choice between HBM and HBF — it is a synthesis of both. As the software ecosystem matures and the complementary architecture becomes standardized, we can expect to see HBM+HBF systems become the default for AI inference, delivering the performance and cost-efficiency that the AI revolution demands.

This analysis is provided for informational purposes and reflects industry research as of September 2026. Technology roadmaps and market conditions are subject to change.

❓ Frequently Asked Questions