HBM Is Too Expensive: Is SanDisk’s HBF the AI Memory Alternative?

Executive Summary

High Bandwidth Memory, or HBM, has become one of the most strategically important components in artificial intelligence infrastructure. Its exceptionally wide interface feeds GPUs and AI accelerators at enormous speed, but limited capacity, difficult manufacturing, advanced packaging requirements, and premium pricing make it an expensive resource. These constraints become especially visible during inference, where large model weights, long contexts, key-value caches, and retrieval data must remain close to processors.

SanDisk’s proposed High Bandwidth Flash, or HBF, applies high-bandwidth packaging and parallelism concepts to NAND flash. The objective is not to make NAND behave exactly like DRAM at the cell level. Instead, HBF seeks to create a very large, cost-efficient memory tier with enough aggregate bandwidth to keep inference processors productive. SanDisk has discussed HBF configurations offering dramatically more capacity than HBM at a comparable package footprint and potentially similar aggregate bandwidth for workloads engineered around sequential, parallel access.

The central conclusion is nuanced: HBF is unlikely to eliminate HBM. NAND latency, write behavior, endurance, software requirements, and product maturity prevent it from being a universal substitute. Its stronger opportunity is as a complementary memory tier—or as an HBM alternative in selected read-heavy inference systems—where capacity per dollar matters more than nanosecond-class latency. If the hardware and software ecosystem matures, HBF could move a substantial part of AI model storage from distant SSDs into a near-accelerator package, creating a new category between HBM and conventional enterprise flash.

01 · Market Context

AI Has Become a Memory Problem

The popular image of artificial intelligence is a warehouse filled with GPUs performing endless matrix multiplications. Compute is certainly essential, but modern AI systems are increasingly constrained by their ability to move data. An accelerator can contain thousands of arithmetic units, yet those units deliver useful performance only when weights, activations, cache data, and intermediate results arrive at the correct time. A processor waiting for memory is an expensive asset consuming power without generating tokens.

Large language models intensify this problem. A model with hundreds of billions of parameters may require hundreds of gigabytes even after using reduced numerical precision. Mixture-of-experts architectures can reach much larger total parameter counts, while only activating part of the model for each token. Long context windows add another demand: the key-value cache grows with the number of users, layers, attention heads, and stored tokens. Retrieval-augmented generation introduces vector indexes and external knowledge. Multimodal systems may simultaneously process text, images, audio, and video.

These workloads create several distinct requirements. AI training needs rapid, frequent reads and writes for parameters, gradients, optimizer states, activations, and collective communication. Inference usually performs more repetitive reads of relatively stable model weights, although cache management and batching still generate writes. Fine-tuning occupies a position between the two. No single memory technology optimally satisfies every requirement.

Traditional server architecture divides the hierarchy into small on-chip SRAM caches, larger off-chip DRAM, and high-capacity SSD storage. AI accelerators inserted HBM into this hierarchy because ordinary server DRAM could not provide enough bandwidth. HBM places multiple DRAM dies in a vertically stacked package, uses through-silicon vias and a very wide interface, and positions memory close to the processor through advanced packaging. This delivers extraordinary throughput, but it also creates a premium and capacity-constrained resource.

The result is the AI memory wall: organizations can acquire more arithmetic capability, but memory capacity, bandwidth, packaging availability, and power prevent the system from using that capability efficiently. The emergence of HBF should therefore be understood not as a minor flash-product update but as an attempt to redesign the hierarchy around the data-access characteristics of AI inference.

02 · Cost Structure

Why HBM Is So Expensive

Saying that “HBM is too expensive” is an effective headline, but it requires context. HBM is expensive because it solves an extremely difficult engineering problem. It should not be compared with commodity flash merely by the price of one stored bit. Its value comes from delivering a combination of low latency, high bandwidth, energy-efficient data movement, and tight integration with an accelerator.

Capacity compounds the issue. Even when HBM delivers excellent bandwidth, an accelerator package can hold only a limited number of stacks. Larger models must be divided across more devices or repeatedly transferred from host memory and storage. Both approaches carry a cost. Adding accelerators solely to obtain more memory means buying compute resources that may be underutilized. Moving weights through PCI Express, Ethernet, or another fabric adds latency, consumes bandwidth, and increases system complexity.

HBM shortages also affect planning. AI infrastructure builders must negotiate allocations long before deployment, while accelerator vendors design products around the HBM generation and packaging technologies available during a narrow production window. If the balance among GPU supply, HBM supply, and packaging capacity is imperfect, one expensive component can strand another.

Nevertheless, price alone does not prove that HBM is economically inefficient. For training or latency-sensitive inference, premium memory may increase accelerator utilization enough to lower total cost per result. The relevant question is not “Which memory has the lowest cost per gigabyte?” It is “Which architecture produces the required tokens, images, or predictions at the lowest total cost while meeting service-level targets?” HBF becomes compelling only if its capacity advantage survives this system-level calculation.

03 · The New Category

What Is SanDisk’s High Bandwidth Flash?

High Bandwidth Flash is a proposed memory technology that combines dense NAND flash with a highly parallel interface and advanced package architecture. SanDisk introduced the concept as a way to bring the capacity economics of flash much closer to AI processors while approaching the aggregate bandwidth expected from high-performance accelerator memory.

The concept borrows the broad physical philosophy of HBM: rather than treating storage as a distant device connected through a narrow protocol stack, memory dies can be organized in stacks with many channels operating in parallel. NAND is inherently denser and less expensive per bit than DRAM because it stores information differently and can be fabricated in very tall three-dimensional structures. This makes terabyte-class capacity inside an accelerator-adjacent package more plausible.

SanDisk has described an ambition to deliver multiple times the capacity of HBM at comparable cost, with public discussions pointing to capacity advantages in the range of approximately eight to sixteen times under certain assumptions. Industry reporting has also discussed the possibility of multi-terabyte accelerator memory configurations, including systems approaching four terabytes. Such numbers are design targets and architectural illustrations rather than a guarantee that every commercial HBF product will have that capacity.

It is important to distinguish HBF from simply attaching an ordinary NVMe SSD to a GPU. An SSD contains NAND packages behind controllers, firmware, mapping tables, error correction, queues, and a host interface. This design provides compatibility and reliability but adds overhead and limits fine-grained parallel access. HBF aims to expose substantially more internal NAND parallelism through an interface optimized for AI data movement. The package, controller, media organization, and software stack would be co-designed rather than inherited unchanged from storage conventions.

HBF is therefore best described as memory-like flash, not as faster conventional storage and not as a direct physical equivalent of DRAM. That distinction explains both its disruptive potential and its limitations.

04 · Architecture

How HBF Could Deliver High Bandwidth

A single NAND operation is much slower than a DRAM access. HBF cannot repeal that physical fact. Its strategy is to overcome individual-device limitations through massive parallelism, deep queues, predictable access, and workload-aware data placement. If many NAND dies, planes, channels, and banks operate concurrently, the total stream of data can be extremely large even when each operation has relatively high latency.

1. 1 Dense 3D NAND provides capacity. Hundreds of flash layers allow a relatively small silicon footprint to store far more data than a comparable DRAM footprint.
2. 2 Multiple dies create parallelism. A stack can distribute requests across dies and internal planes so numerous reads remain in flight simultaneously.
3. 3 A wide interface aggregates throughput. More physical connections and channels can move data directly toward the accelerator without relying on an ordinary SSD interface.
4. 4 Controllers schedule around latency. Intelligent request ordering, prefetching, caching, compression, and error correction can ensure data is prepared before compute units need it.
5. 5 Software arranges model data predictably. Compilers and runtimes can place tensors according to execution order and issue asynchronous reads well ahead of consumption.

This architecture is especially suitable for transformer inference. Model execution follows a known sequence of layers. Weights are largely immutable during inference, and operations consume large tensors rather than continually modifying small random locations. The runtime may know what data will be needed microseconds or milliseconds in advance. With sufficient buffering, it can hide some NAND read latency behind ongoing computation.

HBF would still need faster memory. SRAM caches, local buffers, conventional DRAM, or a smaller pool of HBM could absorb hot data, irregular metadata, activations, and writes. The most credible system is therefore hierarchical: HBF holds a large model close to the accelerator, while HBM or another high-speed tier handles data requiring the lowest latency. This design can provide HBF’s capacity without forcing every memory operation onto flash.

05 · Competitive Positioning

HBF vs. HBM vs. Conventional SSD Storage

No comparison is meaningful without separating latency, aggregate bandwidth, capacity, endurance, cost, persistence, and software behavior. HBF may approach HBM-class aggregate bandwidth in a sufficiently parallel workload while remaining far behind HBM in access latency. Both statements can be true.

The comparison reveals that HBF occupies a new position rather than neatly replacing an existing component. It is closer to compute than an SSD, denser than HBM, more persistent than DRAM, and more specialized than general-purpose storage. Its commercial value will depend on whether this new tier simplifies the system or merely adds another layer requiring expensive engineering.

06 · Business Case

The Economic Case for HBF

The simplest economic argument is that NAND provides far more bits per unit of silicon and per dollar than DRAM. If HBF can package those bits beside an accelerator and deliver sufficient read throughput, a server may host a much larger model without adding GPUs merely for their HBM capacity. This changes both capital expenditure and operating expenditure.

Consider a system in which an inference model does not fit on one accelerator. The operator may divide the model across four or eight GPUs. Every generated token then requires communication among devices, and system performance depends on expensive scale-up interconnects. Some compute units may remain idle because the deployment was sized for memory rather than arithmetic. A high-capacity HBF configuration could theoretically allow fewer accelerators to hold the same model, improving effective compute utilization and reducing network traffic.

Persistence offers another possible advantage. HBM loses its contents when power is removed and must be reloaded from storage. HBF could retain model weights. Faster restart, migration, failure recovery, and model switching could improve fleet utilization, although the practical benefit depends on controller design, security procedures, integrity checks, and whether the surrounding system preserves package power states.

Power must be evaluated carefully. NAND retains data without refresh and may consume less power per stored bit than DRAM. Reduced model shuffling can also save energy in host memory, SSDs, and network links. However, reading from many NAND channels, performing error correction, maintaining large buffers, and operating complex controllers still consumes power. A high-bandwidth HBF package is not equivalent to an idle SSD. Vendors must publish measured energy per delivered byte and energy per token before broad efficiency claims can be validated.

The most useful commercial metric is total cost per token at a defined latency and quality level. That calculation includes accelerator price, memory, packaging, server design, networking, power, cooling, software development, expected utilization, failure rates, and equipment lifetime. HBF can tolerate lower raw performance if lower cost and higher capacity allow operators to serve more total users from each rack. Conversely, an inexpensive memory device has little value if latency causes accelerators to sit idle.

07 · Workload Fit

Why Inference Is HBF’s Natural Target

Training changes model parameters continuously and generates substantial write traffic. It also depends on rapid exchange among memory, accelerator cores, and other devices. NAND’s program latency, erase behavior, endurance limits, and lack of native byte addressability make a complete substitution for HBM unattractive in this environment.

Inference is different. Once a model is deployed, most weights remain unchanged for long periods. Each request reads those weights to calculate outputs. Large sequential or tiled tensor reads can be scheduled predictably, making them well suited to parallel flash. Updates can be handled as bulk model deployments rather than constant fine-grained writes. This reduces endurance concerns and allows data layout to be optimized in advance.

Autoregressive generation is often memory-bandwidth-bound because every token requires the model to revisit a large set of weights while doing a limited amount of computation per byte. If those weights can be streamed at sufficient speed, aggregate memory bandwidth matters more than the latency of any single read. Batching multiple users increases parallelism and gives a controller more opportunities to keep channels busy.

That does not make every inference workload an ideal candidate. Small interactive models may fit comfortably inside HBM, where the lowest possible latency remains valuable. Sparse models with unpredictable expert selection may generate irregular access. Speculative decoding, dynamic routing, retrieval, and long-context attention change the access pattern. Very small batches may expose flash latency more clearly than large batches. HBF’s strongest economics may therefore appear first in high-throughput services, offline inference, recommender systems, model hosting, and multi-tenant environments.

Hybrid systems could allocate data by temperature. Frequently used layers, active experts, cache data, and current activations remain in HBM. Less frequently selected experts or additional models remain in HBF. The runtime promotes and evicts tensors according to demand. This resembles caching, but the unit of management could be a tensor, expert, layer, or model segment rather than an operating-system page.

08 · Benchmark Interpretation

What “Near-HBM Performance” Really Means

SanDisk has publicized research indicating that simulated HBF-backed inference can approach the performance of a system with very large or effectively unlimited HBM capacity, with one company discussion citing a result within approximately 2.2 percent for a studied configuration. This is encouraging, but it must be interpreted as evidence for a particular modeled architecture and workload—not a universal rule.

A simulation may assume a certain number of channels, queue depth, model layout, prefetch accuracy, batch size, flash latency, controller capability, and compute-to-memory ratio. Changing any of these variables can change the result. Real products must also account for thermal throttling, bad-block management, error correction, manufacturing variation, firmware overhead, package power, and contention between reads and writes.

Bandwidth claims require similar care. Peak aggregate bandwidth describes how quickly a fully utilized device can transfer large amounts of data. Random access latency measures how long an individual request waits. HBM performs well in both dimensions. HBF seeks to be competitive primarily in the first. AI software must issue enough independent operations to fill the pipeline; otherwise the bandwidth remains theoretical.

Independent testing should compare HBF systems against several baselines: all-HBM accelerators, GPUs using host DRAM, SSD-based model offloading, distributed multi-GPU deployments, and emerging pooled-memory architectures. Benchmarks should cover dense and mixture-of-experts models, multiple parameter counts, different precision formats, short and long contexts, varying batch sizes, and both throughput-oriented and latency-oriented service profiles.

Bottom line: “Near-HBM performance” should mean near-HBM application throughput for specified inference workloads—not equivalent media latency or interchangeable behavior. Maintaining this distinction is essential for credible purchasing decisions.

09 · Ecosystem

The Software Stack May Decide HBF’s Future

A memory technology can possess excellent silicon characteristics and still fail if programmers cannot use it efficiently. HBM succeeded partly because accelerator vendors made it appear as ordinary high-performance device memory. HBF is more difficult: software must respect NAND’s access granularity, latency, persistence, and write constraints while hiding those differences from most application developers.

Compilers and Graph Optimization

AI compilers understand model graphs, tensor shapes, execution order, and data dependencies. They can lay out weights across HBF channels, schedule reads ahead of computation, fuse operations, and maximize sequential access. Static knowledge is a major advantage over general-purpose storage workloads, which are less predictable.

Runtime Memory Management

The runtime must decide which tensors remain in HBM, which reside in HBF, and when movement occurs. It must avoid repeatedly transferring hot data between tiers. Policies may consider access frequency, tensor size, expert popularity, expected request mix, and service-level objectives. Because model-serving conditions change over time, adaptive algorithms may outperform fixed placement.

Programming and Coherency Models

Developers need clarity about whether HBF is exposed as a memory address space, a managed device tier, a block-like resource, or an object store for tensors. Hardware-coherent access may simplify programming but increase controller complexity. Explicit transfer APIs can offer control but create software friction. Industry standards will be important for preventing every accelerator vendor from implementing an incompatible solution.

Observability and Operations

Data-center operators need telemetry for bandwidth, latency, temperature, wear, error correction, bad blocks, cache hit rates, and data movement. Orchestration software must know which servers can host particular models and how quickly a package can be reprogrammed. Security tools must erase or cryptographically invalidate persistent model weights when hardware changes tenants.

The likely winning implementation will make HBF visible to system software but mostly invisible to AI application developers. Frameworks such as model-serving engines should automatically exploit it, while advanced users retain control for specialized deployments.

10 · Commercial Applications

Where HBF Could Create the Most Value

The common factor is a large, mostly read-oriented dataset whose value increases when placed close to the processor. Workloads dominated by small random writes, strict microsecond tail latency, or continuous parameter updates are less suitable.

11 · Risk Analysis

Limitations HBF Cannot Ignore

These risks do not invalidate HBF. They show why the technology should be evaluated as part of a complete system rather than as an isolated memory chip. HBF wins only when the benefit of capacity and cost exceeds the latency, software, and packaging penalties.

12 · Industry Impact

How HBF Could Reshape the AI Supply Chain

The AI boom has shifted semiconductor value toward accelerators, HBM suppliers, foundries, and advanced packaging. HBF could broaden participation by turning sophisticated 3D NAND into an AI infrastructure component rather than a commodity storage product. This is strategically attractive to SanDisk because it creates a differentiated, higher-value market for flash technology.

For NAND manufacturers, HBF could improve product mix and reduce dependence on volatile consumer-device and SSD cycles. It also demands new capabilities: high-bandwidth interfaces, package stacking, accelerator collaboration, controller design, and model-serving software. Not every NAND supplier will be equally positioned to deliver the full stack.

For HBM vendors, HBF is both a potential competitor and a complementary opportunity. If HBF expands the total AI memory market, suppliers with expertise in HBM interfaces, stacking, base dies, and advanced packaging may participate in standardization or manufacturing partnerships. Industry collaboration around HBF specifications—such as publicly reported cooperation involving SanDisk and SK hynix—suggests that vendors see value in establishing a shared category rather than leaving the technology as a proprietary experiment.

Accelerator vendors will have considerable influence. An HBF stack requires compatible memory controllers, package designs, thermal budgets, firmware, and compiler support. If leading GPU and custom-ASIC companies adopt it, an ecosystem can form quickly. If they prefer larger HBM pools or proprietary memory hierarchies, HBF may remain limited to specialized inference accelerators.

Cloud providers may support HBF if it lowers model-serving costs, but they will also demand multi-source supply and operational portability. Hyperscalers generally avoid architectures that lock a critical fleet component to one supplier unless the performance advantage is overwhelming. Standardized electrical, physical, management, and software interfaces could therefore be as important as the flash media itself.

Advanced packaging remains a shared constraint. HBF may reduce reliance on scarce DRAM stacks, but accelerator-adjacent integration still consumes substrate area, assembly capacity, and engineering resources. The supply chain benefit will depend on whether HBF can use more scalable or less expensive packaging than leading HBM implementations.

13 · Go-to-Market Roadmap

What Must Happen Before Broad Adoption?

Market observers should separate announcements at each phase. A standards milestone does not mean production silicon is available; a successful prototype does not mean competitive manufacturing yield; and a design win does not mean high-volume adoption. The transition from promising architecture to dependable data-center component often takes multiple product cycles.

14 · Strategic Conclusion

HBF Is More Likely to Complement HBM Than Kill It

HBF addresses a genuine and expanding problem. AI inference needs more memory capacity, and simply adding premium HBM is not always the most economical answer. NAND’s density gives SanDisk a credible foundation for a much larger near-compute memory tier. Highly parallel access, predictable model execution, and sophisticated software could make that tier fast enough for many inference workloads.

Yet “HBM replacement” oversimplifies the opportunity. HBM remains the superior medium for low-latency random access, frequent writes, training, activations, cache data, and the hottest parts of inference. HBF is strongest when storing stable weights and other large read-mostly structures. The natural architecture places both technologies in the same system and uses each for the task it performs best.

In the near term, HBF’s most important contribution may be forcing the industry to think beyond a binary choice between expensive accelerator memory and distant SSD storage. AI systems need a continuum: SRAM for immediate reuse, HBM for active computation, HBF for massive near-compute model capacity, pooled DRAM for flexible expansion, and SSDs for durable large-scale repositories.

The commercial prize is significant. If one accelerator equipped with a carefully designed HBM-HBF hierarchy can serve a model that previously required several accelerators, infrastructure economics could change substantially. Cloud providers would reduce stranded compute, model developers could deploy larger architectures, and NAND suppliers would gain access to a strategic new market.

But success is not guaranteed. HBF must prove its claims with shipping hardware, independent benchmarks, competitive power consumption, acceptable endurance, reliable manufacturing, multi-vendor standards, and software that hides complexity. Until then, it should be regarded as one of the most interesting emerging AI memory architectures—not as a confirmed replacement for an established technology.

Final verdict: HBM is not universally “too expensive”; it is too expensive to use for every byte of a rapidly expanding AI working set. HBF’s opportunity is to ensure that organizations no longer have to.

15 · Decision Framework

Questions AI Infrastructure Buyers Should Ask

• Workload: Is the deployment read-heavy inference, or does it involve frequent parameter and cache writes?
• Access pattern: Can reads be predicted, batched, prefetched, and distributed across many independent channels?
• Capacity: Are additional GPUs being purchased primarily to gain memory rather than compute?
• Latency: What happens to time to first token and tail latency under small batches and cache misses?
• Efficiency: What is measured system power per token, including controllers and data movement?
• Durability: What endurance, retention, error-rate, and failure-recovery guarantees are provided?
• Software: Which compilers, frameworks, model servers, and orchestration tools support the memory tier?
• Supply: Is there a second source, an open specification, and a credible multi-generation roadmap?
• Total cost: Does HBF improve cost per token after packaging, software, networking, utilization, and support are included?

Frequently Asked Questions

Five Common Questions About HBF