HBM Capacity Limits? SanDisk HBF Delivers Terabyte-Scale Near-GPU Memory

1. Introduction

For years, the semiconductor industry has treated High-Bandwidth Memory (HBM) as the gold standard for GPU-adjacent memory. Its nanosecond-level latency and multi-terabyte-per-second bandwidth have made it indispensable for AI accelerators. Yet HBM carries a profound limitation that is increasingly constraining the AI revolution: capacity.

A state-of-the-art HBM stack today delivers roughly 24–36 GB per device. While impressive, this pales in comparison to the memory demands of modern AI workloads. Frontier large language models (LLMs) now exceed trillions of parameters, and their working sets — weights, activations, embeddings, and KV-cache — routinely demand hundreds of gigabytes to terabytes of fast memory. HBM's capacity ceiling has become a hard bottleneck.

This is where SanDisk's High-Bandwidth Flash (HBF) enters the picture. By applying HBM-style architectural principles — 3D stacking, wide parallel interfaces, and tight GPU integration — to NAND flash, HBF promises terabyte-scale memory pools positioned directly adjacent to the GPU. It does not replace HBM; it shatters the capacity barrier that HBM alone cannot overcome.

This article provides a comprehensive analysis of HBM's capacity limits, why those limits matter for AI, and how SanDisk HBF delivers a terabyte-scale near-GPU memory tier that could redefine the memory hierarchy.

2. The HBM Capacity Ceiling

To understand why HBF is necessary, we must first understand why HBM capacity is so constrained. The limitation is not a matter of engineering will — it is rooted in the fundamental economics and physics of DRAM stacking.

2.1 The Physical Limits of DRAM Stacking

HBM stacks DRAM dies vertically using through-silicon vias (TSVs). Each die adds capacity, but also adds cost, complexity, and thermal challenges. Current HBM3E stacks reach 8–12 dies, yielding 24–36 GB. Pushing to 16 dies is theoretically possible but encounters diminishing returns: yield drops, thermals worsen, and the package becomes prohibitively expensive.

2.2 The Cost-Per-Bit Problem

DRAM is fundamentally more expensive per bit than NAND — typically 10–20x more. Every additional gigabyte of HBM adds substantial cost to an accelerator that is already priced in the tens of thousands of dollars. System designers face a brutal trade-off: more HBM capacity means a dramatically higher bill of materials (BOM), yet insufficient capacity means the accelerator stalls waiting for data.

2.3 The Capacity-Bandwidth Mismatch

Here lies the core irony: HBM delivers far more bandwidth than most AI workloads actually need, but far less capacity than they require. Many inference and serving workloads are capacity-bound, not bandwidth-bound. They need to hold vast amounts of data close to the GPU, but they do not need every byte accessible at nanosecond latency. HBM over-delivers on speed and under-delivers on space — a mismatch that HBF is designed to correct.

3. Why Capacity Matters for AI Workloads

The AI revolution has transformed memory from a supporting component into a primary determinant of performance and cost. Understanding this shift explains why capacity — not just bandwidth — has become the critical battleground.

3.1 The Weight Storage Problem

A 1-trillion-parameter model, stored in 8-bit precision, requires roughly 1 TB of memory for weights alone. Even at 4-bit quantization, that is 500 GB. These weights must be accessible to the GPU at high speed during inference. HBM simply cannot hold them, forcing systems to shuttle data back and forth from slower storage — a process that cripples performance.

3.2 The KV-Cache Explosion

In transformer-based models, the key-value (KV) cache grows linearly with sequence length and batch size. For long-context inference — now exceeding 1 million tokens in frontier models — the KV-cache alone can consume hundreds of gigabytes. This is a capacity problem that bandwidth alone cannot solve.

3.3 The Cost of Capacity Starvation

When an accelerator runs out of fast memory, it must spill to slower tiers — PCIe-attached SSDs or network storage. This "capacity starvation" causes dramatic performance degradation, often reducing effective throughput by 10x or more. The economic cost is enormous: expensive GPUs sit idle while waiting for data that should have been nearby.

4. HBF: The Terabyte-Scale Answer

High-Bandwidth Flash (HBF) is SanDisk's answer to the capacity ceiling. It applies the architectural principles that made HBM successful — massive parallelism, 3D stacking, and tight integration with the accelerator — to NAND flash, the most abundant and cost-effective memory technology available.

4.1 The Capacity Breakthrough

The headline number is staggering: where HBM delivers 24–36 GB per stack, an HBF module of comparable physical footprint delivers 1–4 TB. This is a 30x to 100x capacity increase. For the first time, it becomes feasible to place an entire LLM's working set — weights, KV-cache, and embeddings — in a single near-GPU memory tier.

4.2 The Cost Revolution

NAND flash costs roughly one-tenth to one-twentieth of DRAM per bit. This means HBF delivers terabyte-scale capacity at a price point that would be unthinkable for HBM. A 4 TB HBF module can cost less than a 36 GB HBM stack, fundamentally changing the economics of AI infrastructure.

4.3 The Bandwidth Compromise

HBF does sacrifice some bandwidth and latency compared to HBM. NAND access latency is measured in microseconds versus nanoseconds for DRAM. However, HBF compensates through massive parallelism — thousands of independent channels operating simultaneously — to deliver aggregate bandwidth in the multiple-terabytes-per-second range. For capacity-bound workloads, this is more than sufficient.

5. HBF Architecture and Design

HBF's terabyte-scale capacity does not come from a single breakthrough, but from the intelligent combination of several mature technologies. Understanding the architecture reveals how SanDisk achieves such dramatic capacity gains.

5.1 3D NAND Stacking at Scale

Modern NAND flash already achieves 200+ layers of 3D stacking — far beyond what DRAM can economically achieve. HBF leverages this mature technology to pack enormous capacity into a compact package. The vertical density of NAND is the foundation of HBF's capacity advantage.

5.2 Wide Parallel Interface

HBF employs a wide, high-speed interface with thousands of I/O pins, mirroring HBM's approach. By parallelizing access across many independent NAND channels, HBF achieves aggregate bandwidth that far exceeds conventional SSDs. This parallelism is what makes HBF viable as a near-GPU memory tier rather than just another storage device.

5.3 Intelligent Data Placement

Because NAND has finite write endurance, HBF incorporates sophisticated data placement logic. Read-mostly data — such as model weights and embeddings — resides permanently on HBF, while write-intensive data is directed to HBM or managed through aggressive wear-leveling. This intelligent tiering maximizes both performance and longevity.

5.4 The Memory Hierarchy Redefined

HBF inserts a new tier into the traditional memory hierarchy:

This new tier means the GPU's "reachable" fast memory expands from tens of gigabytes to terabytes, fundamentally changing what is possible in AI system design.

6. The "Near-GPU" Advantage

The phrase "near-GPU memory" is not marketing hyperbole — it describes a specific architectural positioning that is central to HBF's value. The physical and logical proximity of memory to the compute engine determines everything about performance.

6.1 Physical Proximity and Latency

HBF is designed to be co-packaged with the GPU or connected via a short-reach, high-speed interface (such as CXL or a proprietary link). This physical proximity minimizes the latency penalty compared to PCIe-attached SSDs, which sit far from the compute die and traverse multiple protocol layers. HBF's near-GPU placement makes its microsecond latency far more tolerable than the millisecond latency of traditional storage.

6.2 Bandwidth Without the Bottleneck

A conventional SSD delivers 5–7 GB/s of bandwidth. HBF, by virtue of its wide parallel interface and near-GPU placement, delivers multiple terabytes per second. This is a 100x improvement over SSD bandwidth, bringing flash-class memory into the performance envelope required for AI workloads.

6.3 The "Working Set" Revolution

The most profound implication of near-GPU terabyte-scale memory is that an entire model's working set can now reside in a single fast tier. No more shuttling weights between SSD and HBM. No more capacity starvation. The GPU can access everything it needs from a memory pool that is both large enough and fast enough — a combination that has never before been achievable.

7. Key Use Cases for Terabyte-Scale HBF

HBF's terabyte-scale capacity opens up applications that are simply impossible with HBM alone. Here are the most compelling use cases.

7.1 Large Language Model Inference

Inference is the killer application for HBF. A trillion-parameter model's weights, when quantized to 4-bit, require ~500 GB — well beyond HBM capacity but comfortably within a single HBF module. By keeping the entire model resident in near-GPU memory, HBF eliminates the weight-loading bottleneck that plagues current inference systems, enabling dramatic improvements in tokens-per-second per dollar.

7.2 Long-Context Processing

Long-context inference (1M+ tokens) generates enormous KV-caches that HBM cannot hold. HBF's terabyte capacity makes it possible to maintain these caches in fast memory, enabling applications like document analysis, codebase understanding, and video processing that are currently constrained by memory limits.

7.3 Retrieval-Augmented Generation (RAG)

RAG systems must store and rapidly access massive embedding databases. HBF can hold billions of embeddings in near-GPU memory, dramatically accelerating retrieval operations and enabling real-time RAG at a scale previously impossible.

7.4 Multi-Tenant AI Serving

Cloud providers serving many models simultaneously face a capacity crunch. HBF allows a single GPU to host multiple models' weights simultaneously, improving GPU utilization and reducing the number of accelerators required for a given serving workload.

7.5 Edge and Cost-Optimized Inference

For edge deployments and cost-sensitive applications, HBF enables a new class of accelerator that prioritizes capacity over raw bandwidth. These systems can run substantial models at a fraction of the cost of HBM-based designs, expanding AI's reach into new markets.

8. Challenges and Trade-offs

HBF's promise is real, but it is not without significant challenges. A balanced assessment requires acknowledging the trade-offs.

8.1 The Latency Reality

NAND latency is fundamentally ~1,000x higher than DRAM. While HBF's parallelism and near-GPU placement mitigate this, latency-sensitive workloads (like training's frequent small random accesses) will still require HBM. HBF is not a universal replacement — it is a complement for capacity-bound workloads.

8.2 Write Endurance

NAND cells wear out after a finite number of write cycles. For read-dominated inference workloads, this is manageable. But for write-heavy workloads (training, frequent checkpointing), endurance becomes a genuine concern. HBF systems must implement sophisticated wear-leveling and over-provisioning to achieve acceptable lifespans.

8.3 Software Ecosystem Immaturity

HBM is deeply embedded in the software stack — from CUDA to PyTorch to inference frameworks. HBF requires new memory management abstractions, drivers, and framework integrations. Until this ecosystem matures, HBF adoption will be limited to sophisticated early adopters willing to invest in custom software.

8.4 Competitive Response

HBM incumbents (SK Hynix, Samsung, Micron) and NAND competitors (Kioxia, YMTC) will not stand still. They may develop their own HBF-like products or accelerate HBM capacity expansion. SanDisk's first-mover advantage must be defended through continuous innovation.

9. Strategic Outlook and Conclusion

HBM's capacity ceiling is not a temporary inconvenience — it is a structural limitation rooted in the economics of DRAM. As AI models continue to grow, the gap between HBM's capacity and AI's memory demands will only widen. This is the strategic context that makes HBF so significant.

9.1 The Inevitable Hybrid Future

The future of AI memory is not HBM or HBF — it is HBM and HBF. HBM will continue to serve latency-critical compute, while HBF provides the terabyte-scale capacity that AI workloads desperately need. Together, they form a hybrid memory architecture that finally matches the demands of modern AI.

9.2 The Timeline to Adoption

Expect early HBF adoption in 2026–2027 in inference-optimized systems built by hyperscalers. Broader adoption will follow in 2028–2029 as software support matures. By 2030, HBF is likely to be a standard component of AI infrastructure, with terabyte-scale near-GPU memory becoming the norm rather than the exception.

9.3 Conclusion

SanDisk's High-Bandwidth Flash represents a fundamental rethinking of the memory hierarchy. By applying HBM's architectural principles to NAND flash, HBF delivers terabyte-scale near-GPU memory at a fraction of the cost of HBM — shattering the capacity ceiling that has constrained AI development.

HBF is not without challenges: latency, endurance, and software maturity all demand attention. But for the capacity-bound workloads that dominate modern AI — inference, long-context processing, RAG, and multi-tenant serving — HBF offers a solution that HBM alone cannot provide.

The question is no longer whether AI needs terabyte-scale memory, but which technology will deliver it. SanDisk HBF has positioned itself as a leading answer. This analysis is provided for informational purposes and reflects industry research as of September 2026.

❓ Frequently Asked Questions