Sandisk HBF: The Missing Memory Tier to Unlock HBM-Powered AI Inference Scaling

Sandisk HBF · High-Bandwidth Flash · HBM DRAM · AI Inference Scaling · Memory Tiering

Sandisk HBF: The Missing Memory Tier to Unlock HBM-Powered AI Inference Scaling

An in-depth B2B analysis of why AI inference is hitting a memory capacity wall, how Sandisk High-Bandwidth Flash (HBF) is positioned as the missing tier between HBM DRAM and storage, and what datacenter operators, server OEMs, system integrators, and procurement teams should understand before the next hardware refresh cycle.

Editorial note: This article is written for global B2B readers, including AI datacenter operators, cloud and neocloud service providers, GPU server buyers, semiconductor and memory supply-chain professionals, storage solution vendors, enterprise IT architects, AI system integrators, and technology investors. Sandisk High-Bandwidth Flash (HBF) is discussed as an emerging NAND-based memory tier designed to sit alongside HBM DRAM in AI accelerator systems. Where public vendor statements are referenced, they should be treated as roadmap positioning rather than guaranteed shipping specifications.

The central thesis is simple: HBM DRAM is the engine of AI compute, but HBM alone cannot economically scale inference to the capacity levels that modern models, long contexts, multi-tenant serving, and retrieval-heavy applications demand. A “missing tier” of high-bandwidth, high-capacity, non-volatile memory is required, and HBF is one of the most credible candidates to fill it.

1. Introduction: Inference Has Become a Memory Problem

For most of the generative AI boom, the industry conversation about infrastructure has been dominated by compute. How many accelerators can a cloud provider secure? How many FLOPS does the newest GPU deliver? How large a training cluster can be assembled, and how fast can the interconnect move gradients between nodes? These questions still matter, but they no longer define the frontier of AI infrastructure economics. As the industry shifts from building models to serving them at scale, the defining constraint is increasingly memory: how much of it sits close to the accelerator, how fast it can be read, how much it costs per gigabyte, and whether it survives a power cycle.

Inference is fundamentally different from training. Training is a batch process that runs for weeks on a fixed dataset and a single model. Inference is a continuous service that runs forever, responds to unpredictable user demand, serves many models and many tenants, and must meet latency targets measured in milliseconds. Every architectural decision in an inference datacenter is ultimately expressed in a single business metric: cost per token, or more broadly, cost per useful response. Memory architecture is the largest lever on that metric that most operators have not yet fully pulled.

High Bandwidth Memory, or HBM DRAM, is the reason modern accelerators can execute large models at all. It delivers terabytes per second of bandwidth directly into the compute die. But HBM is expensive, supply-constrained, volatile, and limited in capacity per package. Inference workloads, meanwhile, are expanding in every dimension that consumes memory: larger models, mixture-of-experts architectures with hundreds of billions of parameters, context windows measured in hundreds of thousands or millions of tokens, multi-tenant serving with thousands of fine-tuned adapters, retrieval pipelines that touch enormous vector indexes, and agentic systems that maintain persistent state across long sessions.

The result is a structural mismatch. HBM provides the bandwidth that inference needs but not the capacity. Conventional storage provides the capacity but not the bandwidth or the proximity. Host DRAM sits in the middle but is bottlenecked by the interconnect between CPU and accelerator, and it is itself becoming expensive and constrained as AI demand consumes memory supply. Something is missing from the hierarchy.

Sandisk High-Bandwidth Flash, known as HBF, is a direct response to that gap. It applies the packaging concepts that made HBM successful—vertical die stacking, through-silicon vias, wide parallel interfaces, and placement adjacent to the compute die—to NAND flash instead of DRAM. The resulting promise is a memory tier with bandwidth in the same conversation as HBM, capacity many times larger, non-volatility, and a cost per bit that reflects NAND economics rather than DRAM economics.

This article examines HBF as the “missing memory tier” for HBM-powered inference scaling. It explains why inference has become a memory problem, how HBM works and where it hits its limits, what HBF actually is and how it would integrate with accelerators, which workloads benefit, what the trade-offs are, how the economics might work, what software must change, and what buyers should ask before committing to platforms that include this technology. The goal is not to promote a product but to give infrastructure decision-makers a rigorous framework for thinking about the next phase of AI memory architecture.

2. From Training-Centric to Inference-Centric Datacenters

The first wave of AI datacenter investment was built for training. Training clusters prioritize raw compute density, high-bandwidth interconnect between accelerators, synchronized all-reduce operations, and massive amounts of HBM to hold model parameters, gradients, optimizer states, and activations. A training job is a single enormous workload with predictable access patterns, and the economics are driven by time-to-train.

The second wave, which is now underway across hyperscalers, neoclouds, sovereign AI programs, and enterprise deployments, is built for inference. Industry analysts have consistently projected that inference will account for the majority of AI compute cycles and the majority of AI infrastructure spending over the coming years. This shift changes what a datacenter must be good at. Inference clusters must handle bursty traffic, heterogeneous models, strict latency service-level agreements, high concurrency, and continuous availability. They are judged on tokens per second per dollar, tokens per second per watt, and time-to-first-token.

Inference also changes the memory profile. In training, the dominant memory consumer is the model itself plus training state, and the whole thing lives in HBM across many accelerators. In inference, the memory footprint is dominated by three things that scale with usage rather than with the model: the key-value cache that grows with context length and concurrent users, the model pool that grows with the number of variants and adapters served, and the retrieval corpus that grows with the amount of enterprise knowledge the system must access. None of these scale gracefully in an HBM-only architecture.

Consider a commercial LLM API. A single deployment might serve a flagship model, several smaller distilled models for cheaper tiers, specialized coding and reasoning variants, multimodal models for vision and audio, safety and moderation models, embedding models for retrieval, and hundreds or thousands of customer fine-tunes. Each active request generates a KV cache that may occupy gigabytes for long contexts. Each retrieval call may touch a vector index of hundreds of gigabytes or terabytes. Serving all of this from HBM would require an absurd number of accelerators, most of them sitting underutilized simply to provide memory capacity.

This is why inference-centric datacenters need a fundamentally different memory hierarchy. The question is no longer “how much HBM can we fit on the package?” but “how do we build a hierarchy where each byte lives in the cheapest tier that still meets its performance requirement?” HBF is a candidate for one of the most important rungs in that hierarchy.

The shift also has geographic and commercial dimensions. Training remains concentrated in a small number of frontier labs and hyperscale campuses. Inference is distributed everywhere: regional clouds, enterprise on-premises deployments, sovereign national AI facilities, telecom edge sites, and industrial environments. For hardware vendors, system integrators, and export-oriented server manufacturers, inference is the larger and more diverse addressable market. Memory architecture that lowers the cost and complexity of inference deployment has direct commercial consequences across that entire supply chain.

3. HBM DRAM: The Engine of AI Acceleration

To understand why a missing tier exists, one must first understand what HBM does and why it is irreplaceable in its role. HBM is a form of DRAM in which multiple memory dies are stacked vertically and connected through thousands of through-silicon vias. The stack is mounted on a silicon interposer or advanced substrate alongside the GPU or accelerator die, with an interface that is roughly a thousand bits wide per stack. Because the physical distance between memory and compute is measured in millimeters rather than centimeters, HBM achieves very high bandwidth at relatively low energy per bit compared with conventional DDR memory on a motherboard.

Each generation of HBM has increased per-stack bandwidth, stack height, and per-stack capacity. Modern accelerators integrate multiple HBM stacks around the compute die, delivering aggregate memory bandwidth measured in multiple terabytes per second and capacities in the range of one to two hundred gigabytes per accelerator, with roadmaps pushing higher. This bandwidth is what allows an accelerator to stream billions of model weights through its matrix units for every generated token.

The importance of this cannot be overstated for inference. Autoregressive text generation is notoriously memory-bandwidth-bound during the decode phase. For every output token, the accelerator must read the relevant model weights and the accumulated KV cache from memory. Compute utilization during decode is often low precisely because the bottleneck is how fast data can be pulled from HBM, not how fast the arithmetic units can multiply. Increasing HBM bandwidth translates almost directly into higher tokens per second for a given batch.

HBM is also central to the prefill phase, where the input prompt is processed in parallel. Prefill is more compute-bound than decode, but it still relies on HBM to hold the weights and to write the KV cache that decode will later read. And in training, HBM holds everything: parameters, gradients, optimizer moments, and activations, with bandwidth requirements even more demanding than inference.

Nothing on the horizon replaces HBM in this role. NAND flash cannot match DRAM latency for random fine-grained access. Host DRAM cannot match HBM bandwidth because it is separated from the accelerator by PCIe or similar links with bandwidth an order of magnitude lower. On-die SRAM is faster than HBM but far too small and expensive to hold model weights at scale. HBM is the right technology for the hottest data in the system, and every AI accelerator roadmap continues to rely on it.

The argument for HBF, therefore, is not that HBM is inadequate at what it does. It is that HBM is being asked to do too much. When HBM is the only high-bandwidth memory available to the accelerator, every byte that inference needs quickly must compete for HBM residency, including bytes that are read infrequently, read sequentially, or read in large blocks that could tolerate higher latency if bandwidth were sufficient. Relieving HBM of that burden is the core value proposition of a complementary tier.

4. The HBM Capacity Gap in Inference Scaling

The HBM capacity gap can be described quantitatively. A frontier-class model with several hundred billion to over a trillion parameters, even at 8-bit or 4-bit quantization, occupies hundreds of gigabytes to over a terabyte of weights. Mixture-of-experts models are particularly challenging: only a fraction of experts are active for any given token, but all experts must be resident somewhere fast enough to be swapped in within the latency budget. An accelerator with roughly 150 to 300 gigabytes of HBM cannot hold such a model alone, which is why inference for large models is spread across multiple accelerators using tensor and expert parallelism, consuming interconnect bandwidth and multiplying hardware cost.

On top of weights sits the KV cache. For a transformer model, the KV cache size scales with the number of layers, the hidden dimension, the number of attention heads, the precision, the context length, and the number of concurrent sequences. For long-context workloads, a single request’s KV cache can reach tens of gigabytes. Serving many concurrent long-context users pushes total KV cache into the terabyte range for a single deployment. When HBM fills up, serving engines must either reject requests, reduce batch size, or evict and recompute cache, all of which degrade throughput and increase cost.

Then there is the model pool. Multi-tenant platforms routinely serve many models and thousands of parameter-efficient fine-tuned adapters. Each adapter is small relative to a base model, but the aggregate library can reach hundreds of gigabytes or more. Keeping all adapters in HBM is impossible; loading them from network storage on demand introduces cold-start latency that violates service-level objectives.

Finally, retrieval systems add vector indexes and embedding tables that can reach terabytes for enterprise knowledge bases. These are not held in HBM today, but they are increasingly on the critical path for time-to-first-token, and moving them closer to the accelerator would improve end-to-end latency.

The industry’s current responses to this capacity gap all have costs. Adding more accelerators purely for memory capacity wastes compute and power. Aggressive quantization reduces quality. Offloading to host DRAM over PCIe introduces a bandwidth cliff. Offloading to NVMe SSDs introduces a larger latency and bandwidth cliff plus host-side software overhead. CXL memory expansion helps with DRAM capacity but does not change DRAM economics and still sits behind an interconnect. None of these approaches provide what inference actually wants: a large, persistent, high-bandwidth capacity pool physically adjacent to the accelerator.

This is the gap. HBM sits at the top with extreme bandwidth and limited capacity. NVMe storage sits at the bottom with vast capacity and comparatively modest bandwidth and long latency. The space between them—high-bandwidth, high-capacity, low-cost-per-bit, non-volatile, and close to compute—is empty in most current accelerator designs. Filling it is the difference between inference that scales economically and inference that scales only by throwing more accelerators at memory problems.

5. Defining the Missing Memory Tier

A useful way to think about memory hierarchies is that each tier should be roughly an order of magnitude larger and cheaper than the tier above it, while being close enough in performance that the tier above can effectively cache it. This principle has governed CPU design for decades: registers, L1, L2, L3, DRAM, and storage each play a role, and software and hardware cooperate to keep hot data at the top. The AI accelerator memory hierarchy has been missing a rung.

In today’s accelerator systems, the jump from HBM to the next tier is enormous. HBM delivers multiple terabytes per second with sub-microsecond access. Host DRAM over PCIe or a proprietary link delivers tens to low hundreds of gigabytes per second. NVMe SSDs deliver single-digit to low double-digit gigabytes per second per device, with latencies of tens of microseconds. That is a bandwidth drop of one to two orders of magnitude and a latency increase of two orders of magnitude at the first step down from HBM. Caching across such a cliff is inefficient because a miss is catastrophically expensive relative to a hit.

The missing tier should have several properties. It should offer bandwidth within striking distance of HBM—not necessarily equal, but close enough that streaming large blocks of weights or cache does not stall the accelerator. It should offer capacity many times larger than HBM at a fraction of the cost per bit. It should be physically attached to the accelerator package or module rather than reached through the host CPU. It should ideally be non-volatile, so that model assets persist across restarts and can be provisioned once rather than reloaded constantly. And it should be exposed to software in a way that enables tier-aware placement and prefetching.

DRAM cannot fill this tier economically because DRAM cost per bit is fundamentally higher than NAND and DRAM cell scaling has slowed. Conventional SSDs cannot fill it because they are architected for the block storage stack, sit behind PCIe, and were never designed for accelerator-attached bandwidth. Emerging memory technologies such as phase-change or resistive memories have not achieved the density and cost required at scale. This leaves NAND flash, re-architected for bandwidth and proximity, as the most practical option available in the near term.

The insight behind HBF is that NAND’s weaknesses—higher latency and finite endurance—are tolerable for a large class of inference data if NAND’s strengths—density, cost, and non-volatility—can be combined with HBM-class packaging to deliver bandwidth. Weights are read far more often than written. Adapter libraries are essentially read-only after provisioning. Vector indexes update slowly relative to how often they are queried. Even KV cache, which is written once per token and read many times, has a write pattern that can be managed with appropriate wear-leveling and buffering if the hottest cache stays in HBM.

When the missing tier is filled, the accelerator memory hierarchy becomes coherent again: on-die SRAM for immediate operands, HBM for hot working sets, HBF for warm high-capacity data, host DRAM for orchestration, NVMe for bulk local storage, and networked object storage for the data lake. Each tier can now cache the tier below it at a reasonable cost ratio, and inference systems can be designed to exploit that structure.

6. Sandisk HBF Explained: Architecture and Positioning

Sandisk introduced High-Bandwidth Flash publicly in early 2025, following its separation from Western Digital as an independent flash company. The concept builds directly on the company’s long history in 3D NAND and applies HBM-style packaging to it. In broad terms, HBF stacks multiple high-density NAND dies vertically, connects them with through-silicon vias to a logic base die, and presents a wide, highly parallel interface intended to be compatible with the electrical and mechanical footprint of an HBM stack. The stack is then mounted alongside the accelerator die on the interposer, in the positions where HBM would normally sit.

The headline claims Sandisk has made in public presentations are that HBF can deliver bandwidth comparable to HBM at the stack level while offering roughly eight to sixteen times the capacity of an HBM stack of the same footprint, at a significantly lower cost per bit. The company has framed the target application as AI inference, explicitly acknowledging that HBF is not intended to replace HBM for latency-critical operations but to sit alongside it. Sandisk has also formed a technical advisory board including prominent computer architects from academia and industry to guide the specification, and it has announced collaboration with a leading HBM supplier on standardizing the HBF interface, signaling an intent to make HBF an ecosystem technology rather than a proprietary product.

From a technology standpoint, the key enablers are several. First, high-layer-count 3D NAND provides extreme density per die, which is what allows a stack to reach hundreds of gigabytes. Second, NAND’s inherent parallelism—many planes and many dies operating concurrently—is what makes high aggregate bandwidth possible when the interface is wide enough. Traditional SSDs are throttled by a narrow PCIe link and a controller optimized for block I/O; HBF removes that throttle by connecting NAND directly through thousands of TSVs. Third, the logic base die can implement error correction, wear leveling, address translation, and prefetch buffering close to the flash, hiding much of the management complexity from the accelerator.

HBF also benefits from Sandisk’s existing investments in wafer bonding and advanced NAND process technology, where CMOS logic and memory arrays are manufactured on separate wafers and bonded together. This approach maps naturally onto a stacked architecture where a logic base die manages multiple memory dies. The company has indicated that sample availability would occur in the second half of 2026 with first inference devices expected to follow in early 2027, which places HBF on the horizon of the current planning cycle for AI system designers.

Positioning is critical to understanding HBF correctly. Sandisk is not claiming that NAND achieves DRAM latency. Random access to a NAND page still takes microseconds rather than nanoseconds. What HBF offers is the ability to move large quantities of data into or through the accelerator at HBM-class throughput, from a pool far larger than HBM could ever provide, at NAND prices, and with persistence. For workloads where the access pattern is predictable enough to prefetch or coarse-grained enough to stream, that combination is transformative. For workloads that require nanosecond random access to arbitrary bytes, HBM remains the answer, and HBF acts as its backing store.

A realistic accelerator configuration might therefore combine a small number of HBM stacks with a number of HBF stacks, or offer variants with different ratios depending on the target workload. A training-optimized part would be HBM-heavy. An inference-optimized part could trade some HBM for HBF, achieving terabytes of near-package memory for a model pool and KV cache while retaining enough HBM for hot working sets. This flexibility is what makes HBF interesting to accelerator designers as well as to memory buyers.

7. How HBF Complements HBM Inside the Accelerator Package

The most consequential aspect of HBF is where it lives. Previous attempts to expand accelerator memory—host DRAM offload, NVMe offload, CXL pooling—all place the additional capacity on the far side of an interconnect controlled by the host. HBF is designed to sit on the accelerator package itself, sharing the interposer with HBM and the compute die. This changes the data path fundamentally. Data moving from HBF to HBM or directly into compute never traverses PCIe, never involves the host CPU, and never competes with network or storage traffic for interconnect bandwidth.

In such a design, HBM and HBF form a two-level near-package hierarchy. HBM holds the active working set: the weights of the currently executing layers or experts, the KV cache for active sequences, activations, and scratch buffers. HBF holds the extended working set: the full model or model pool, the KV cache for paused or older sequences, adapter libraries, embedding tables, and any other data the accelerator may need within the latency budget of a prefetch. A memory controller or runtime moves data between the two based on access patterns, much as a CPU cache hierarchy moves lines between L2 and L3.

Consider mixture-of-experts inference as a concrete example. An MoE model may contain dozens or hundreds of experts, but each token activates only a handful. If all experts must reside in HBM, the model requires many accelerators regardless of how much compute is actually used. With HBF, the full expert set can reside in near-package flash, and the runtime can stream the required experts into HBM on demand. Because expert routing decisions are made a layer ahead, and because expert weights are read as large contiguous blocks, this access pattern is well suited to HBF’s bandwidth-rich, latency-tolerant profile. The result is that a single accelerator, or a much smaller group of accelerators, can serve a model that previously required a large cluster.

Weight streaming for dense models follows similar logic. During decode, each layer’s weights are read once per token. If HBF bandwidth is sufficient to stream a layer’s weights in the time it takes to compute the previous layer, weights need not permanently reside in HBM at all; HBM becomes a staging buffer, and the model size the accelerator can serve is bounded by HBF capacity rather than HBM capacity. Whether this is fully achievable depends on the actual HBF bandwidth and the model’s arithmetic intensity, but even partial weight streaming significantly relaxes the HBM constraint.

For KV cache, the two-tier structure enables a natural policy. Active sequences currently generating tokens keep their cache in HBM. Sequences waiting for the next user turn, which in conversational applications may be idle for seconds to minutes, can have their cache migrated to HBF rather than evicted and recomputed. When the user sends the next message, the cache is restored from HBF far faster than it could be recomputed through prefill, and without consuming network bandwidth to a remote cache store. Because HBF is non-volatile, cache for long-running sessions can even survive an accelerator restart.

Non-volatility also changes provisioning. Today, an inference node loads its model from network storage into HBM at startup, a process that for large models can take minutes and saturates storage networks when many nodes start simultaneously. With HBF, model weights can be provisioned once into near-package flash and remain there across reboots, software updates, and power events. Node startup becomes a matter of warming HBM from local HBF rather than pulling terabytes over the network. For fleets of thousands of inference nodes, this is a substantial operational improvement.

The integration also has implications for accelerator designers. Because HBF is designed to be footprint-compatible with HBM, a package designer can allocate stack positions between the two technologies without redesigning the interposer. This lowers the barrier to producing inference-optimized variants of existing accelerators and allows the HBM-to-HBF ratio to be tuned per product. Standardization efforts between Sandisk and HBM manufacturers are intended to make this interoperability practical across vendors.

8. Anatomy of an Inference Workload: Weights, KV Cache, Context

To evaluate HBF rationally, it helps to decompose an inference workload into its memory components and ask, for each one, what access pattern it exhibits and which tier suits it best. Inference memory can be divided into five categories: model weights, KV cache, activations and scratch, auxiliary model assets, and external data such as retrieval indexes.

This decomposition reveals that only a minority of inference memory by volume truly requires HBM. Activations, scratch space, and active KV cache are latency-critical and must stay in HBM. Everything else is either read-mostly, coarse-grained, or predictable enough to prefetch. In a typical large-scale deployment, the HBM-mandatory portion might be tens of gigabytes per accelerator while the total working set including model pool, idle cache, adapters, and warm retrieval data may be terabytes. Today, that terabyte-scale working set is either spread across far more accelerators than compute requires or pushed out to slow tiers with painful latency consequences.

The decomposition also clarifies where HBF does not help. If a workload is dominated by a single small model with short contexts and low concurrency, the entire working set already fits in HBM and HBF adds cost without benefit. HBF is a technology for scaling: large models, many models, long contexts, high concurrency, or heavy retrieval. Buyers should map their own workload against this taxonomy before assuming HBF is relevant to them.

One subtle point concerns the prefill phase. Prefill processes the entire prompt in parallel and is compute-heavy, but for very long prompts it produces enormous KV cache that must be written somewhere. Writing multi-gigabyte cache into HBM and then immediately spilling it to HBF as the sequence transitions to decode is an access pattern that suits HBF well, provided write bandwidth and endurance are managed. The base die in an HBF stack can buffer writes and coalesce them into flash-friendly sizes, and the write volume per token during decode is small relative to reads.

9. Multi-Tenant Serving, Model Pools, and Adapter Libraries

Commercial AI inference is overwhelmingly multi-tenant. A cloud provider or AI platform serves many customers, each of whom may have their own fine-tuned models or adapters, their own retrieval corpora, their own prompt templates, and their own latency and isolation requirements. The economics of such platforms depend on packing many tenants onto shared hardware while preserving the illusion, and the guarantees, of dedicated service.

Parameter-efficient fine-tuning, and LoRA in particular, has become the dominant mechanism for customization. Rather than duplicating an entire base model per customer, a platform stores a small low-rank adapter per customer and applies it at inference time. A single base model can thus serve thousands of tenants. The challenge is that the adapter library grows without bound as tenants are added, and each request must have its adapter available within the latency budget. Serving engines have developed techniques to batch requests across different adapters efficiently, but those techniques assume the adapters are already in accelerator memory.

In an HBM-only design, the platform faces an unpleasant trade-off. Either it limits the number of concurrently loaded adapters to what fits alongside the base model in HBM, and accepts cold-start latency when a less popular tenant’s request arrives, or it provisions more accelerators than compute requires purely to hold adapters. Fetching adapters from host DRAM over PCIe is workable but consumes link bandwidth that is also needed for input and output data. Fetching from network storage introduces latency of hundreds of milliseconds or more, which is unacceptable for interactive workloads.

HBF resolves this cleanly. The entire adapter library for a node’s tenant population can reside in near-package flash. When a request arrives for a tenant whose adapter is not in HBM, the adapter is loaded from HBF in the time it takes to schedule the request, with no host involvement and no network traffic. Because HBF is non-volatile, the library persists across restarts, and provisioning a new tenant means writing a few hundred megabytes to HBF once rather than distributing it to every node’s startup script.

The same logic extends to full model pools. A platform serving multiple base models—different sizes for different price tiers, specialized models for code or vision, safety models, embedding models—can hold all of them in HBF and rotate them through HBM based on demand. This enables a form of model-level oversubscription analogous to how virtual machines oversubscribe physical memory. Demand for different models fluctuates throughout the day and across regions; the ability to shift accelerator capacity between models in seconds rather than minutes improves utilization and reduces the number of accelerators that must be dedicated to low-traffic models.

Isolation and security considerations also favor a near-package tier. Tenant adapters and data held in HBF remain within the accelerator’s trust boundary rather than transiting host memory or the network on every request. For platforms serving regulated industries or sovereign customers, minimizing the movement of customer-specific model assets across shared infrastructure is a meaningful compliance benefit, though it also raises questions about secure erasure and encryption at rest within HBF that vendors will need to address.

For B2B buyers evaluating inference platforms, the practical question is how many tenants and models a given hardware configuration can serve at target latency. HBF-equipped systems should demonstrate materially higher tenant density and lower cold-start latency than HBM-only systems at equivalent accelerator count. Vendors should be asked to publish those numbers under realistic mixed-tenant traffic, not only under single-model benchmarks.

10. RAG, Vector Search, and Enterprise Knowledge at Scale

Retrieval-Augmented Generation has become the default architecture for enterprise AI because it solves the two problems that make raw language models unsuitable for business use: they do not know a company’s private information, and they cannot be trusted to answer factually without grounding. RAG addresses both by retrieving relevant documents from an enterprise corpus and injecting them into the prompt. The language model remains general; the knowledge lives in the retrieval layer.

The retrieval layer is a memory-intensive system in its own right. Documents are chunked, embedded into high-dimensional vectors, and stored in an index optimized for approximate nearest-neighbor search. For a mid-sized enterprise with millions of documents, the vector index may reach hundreds of gigabytes. For a large enterprise, a legal or financial data provider, or a platform serving many customers’ corpora, indexes reach terabytes. Alongside the vectors sit the original text chunks, metadata for filtering, and often multiple indexes for different embedding models or languages.

Today, vector search infrastructure typically runs on CPU servers with large DRAM pools, or on GPUs with the hot portion of the index in HBM and the remainder in host memory or on SSDs. Both approaches have costs. DRAM-resident indexes are expensive and increasingly compete with AI demand for DRAM supply. SSD-resident indexes work well for graph-based search algorithms that access a limited number of nodes per query, but latency rises with the number of disk accesses, and throughput under concurrent load is bounded by device IOPS and the host storage stack.

HBF creates a new option: co-locating warm retrieval data with the accelerator that performs the generation. In an inference node equipped with HBF, the vector index partitions relevant to that node’s tenants can reside in near-package flash. The accelerator can execute embedding computation, vector search, reranking, and generation without the retrieved data ever leaving the package boundary. This eliminates a network round-trip to a separate vector database cluster, reduces time-to-first-token, and simplifies the deployment topology.

Vector search access patterns fit HBF reasonably well. Graph-based approximate search touches a bounded number of nodes per query, and each node visit reads a contiguous block of vector data plus adjacency information. These are moderately sized reads that benefit from bandwidth and tolerate microsecond-scale latency. Batching many concurrent queries amortizes latency further and exploits NAND parallelism. Index updates are comparatively rare and can be staged in DRAM before being written to flash in bulk, keeping write amplification under control.

The enterprise value goes beyond performance. Many organizations deploying RAG are doing so on-premises or in private clouds for data governance reasons. An inference appliance in which the model, the retrieval index, and the generation compute all reside in a single accelerator package is architecturally simpler, easier to secure, and easier to operate than a distributed system of GPU servers, vector database nodes, and storage arrays. For system integrators and OEMs selling to enterprises, HBF-equipped nodes could enable a new category of self-contained “knowledge inference” appliances.

Semantic caching, a close cousin of RAG, benefits similarly. A semantic cache stores previous prompts and responses keyed by embedding, so that a new query semantically close to a previous one can be served from cache rather than regenerated. At scale, a semantic cache is itself a large vector index plus stored responses. Holding it in HBF near the accelerator turns cache hits into near-instant responses at negligible compute cost, directly reducing cost per query for repetitive enterprise workloads such as customer support and internal help desks.

11. Long Context, Agentic AI, and Persistent Memory

Context window length has grown from a few thousand tokens to hundreds of thousands and, in some models, into the millions. Users now expect to load an entire codebase, a full legal contract set, a year of financial filings, or a long multi-session conversation into a single request. Long context is one of the most powerful capabilities of modern models and one of the most expensive to serve, because KV cache scales linearly with context length and must be read in full for every generated token.

For a large model at long context, a single sequence’s KV cache can exceed the HBM capacity of an entire accelerator when combined with weights. Serving many such sequences concurrently is impossible without either massive parallelism across accelerators or aggressive cache compression techniques that degrade quality. Serving engines have responded with paged attention, quantized cache, and cache eviction heuristics, but all of these work within a fixed HBM budget. The budget itself is the constraint.

HBF changes the budget. With terabytes of near-package flash, KV cache for long-context sequences can be tiered. The portion of the cache corresponding to the most recent tokens, which attention weights tend to favor most heavily, remains in HBM. Older portions migrate to HBF and are streamed back in when attention computation needs them. Research into attention sparsity and retrieval-based attention suggests that for very long contexts, only a subset of cache blocks are materially relevant to any given token, which means a well-designed system can keep HBM traffic bounded while the full context lives in HBF. Even without sparsity, HBF bandwidth allows a larger context to be served at acceptable throughput than HBM alone would permit.

Agentic AI intensifies the requirement. An AI agent executing a multi-step task—researching, calling tools, writing code, verifying results, iterating—accumulates context continuously over sessions that may last hours. Agents also need persistent memory: user preferences, project state, prior decisions, tool outputs, and learned facts that should survive across sessions. This memory is written incrementally, read selectively, and must be durable. It maps naturally onto a non-volatile near-package tier. An agent’s working memory can live in HBF, be paged into HBM as needed for reasoning, and persist through node restarts without a round trip to external storage.

Multi-turn conversational applications present a related pattern. A user sends a message, the model responds, and the user thinks for thirty seconds before sending the next message. During that pause, the session’s KV cache is idle. In an HBM-only system, the operator must choose between holding the cache in HBM, wasting scarce capacity, or evicting it and recomputing prefill on the next turn, wasting compute and increasing time-to-first-token. With HBF, idle session cache is simply parked in flash and restored in milliseconds when the next turn arrives. For platforms serving millions of concurrent conversations, the effect on both HBM efficiency and user-perceived latency is significant.

The endurance question is most relevant here, since KV cache is written continuously. Mitigation strategies include keeping the hottest cache in HBM so that HBF sees only spill traffic, buffering writes in the HBF base die to coalesce them into flash-optimal sizes, applying compression before writes, and using high-endurance NAND configurations for cache regions. Vendors will need to publish realistic drive-writes-per-day equivalents for HBF under representative inference workloads, and buyers should insist on those figures.

12. Semantic Caching, Prefix Reuse, and KV Offload

Three optimization techniques that have become central to inference efficiency all depend on having a large, fast, ideally persistent memory pool near the accelerator: prefix caching, KV offload, and semantic caching. Each of them is constrained today by HBM capacity, and each becomes far more effective with HBF.

Prefix caching exploits the fact that many requests share a common prefix. System prompts, few-shot examples, tool schemas, retrieved documents, and long conversation histories are often identical across many requests. If the KV cache for that shared prefix is computed once and stored, subsequent requests can skip prefill for the shared portion and begin generation almost immediately. The savings are largest for long system prompts and RAG contexts, where the shared prefix may be tens of thousands of tokens. The limitation is storage: a prefix cache is only useful if it is large enough to hold the prefixes that will recur, and HBM has no room for a large prefix library alongside weights and active cache.

HBF provides that room. A node can maintain a prefix cache of hundreds of gigabytes or more in near-package flash, indexed by prefix hash, and restore matching prefixes into HBM on request. Because HBF is persistent, the prefix cache survives restarts and accumulates value over time. For enterprise deployments where a handful of system prompts and document sets dominate traffic, hit rates can be very high and the reduction in prefill compute substantial.

KV offload generalizes the idea. Rather than only caching shared prefixes, the serving engine treats HBM as a cache for a much larger KV store in HBF, moving cache blocks in and out based on scheduling. This is analogous to virtual memory, and it allows the effective concurrency of a node—the number of sequences it can hold in flight—to be bounded by HBF capacity rather than HBM capacity. Higher concurrency means larger batches, and larger batches mean better utilization of the accelerator’s compute during decode, which is the phase where HBM bandwidth otherwise leaves compute idle.

Disaggregated inference architectures, in which prefill and decode run on different accelerators, add another dimension. Prefill nodes produce KV cache that must be transferred to decode nodes. Today this transfer competes for interconnect bandwidth. With HBF on both sides, prefill nodes can write cache to persistent flash and decode nodes can read it at high bandwidth, and if the two roles are co-located on the same package, the transfer disappears entirely. Even where they are separate, HBF provides a landing zone that decouples the timing of prefill and decode.

Semantic caching, discussed earlier in the RAG context, is the coarsest form of reuse: whole responses cached by meaning rather than exact match. It requires a vector index of past queries and a store of responses. Both live comfortably in HBF. For high-volume, repetitive workloads, semantic cache hit rates of a meaningful fraction of traffic translate directly into fewer tokens generated and lower cost per query.

The common thread is that inference efficiency increasingly comes from remembering rather than recomputing, and remembering requires memory. HBF is the tier that makes large-scale remembering economically feasible on the accelerator itself.

13. Technical Comparison: HBM, DRAM, HBF, and NVMe SSD

The table below places HBF in context relative to the tiers it sits between. Figures are indicative of the class of technology and intended to convey relative positioning, not vendor-specific guarantees. Actual HBF performance will depend on final specifications and system integration.

The comparison makes the structural argument visible. HBF is the only tier that combines accelerator-package proximity, HBM-class bandwidth ambition, multi-terabyte capacity, NAND cost, and persistence. Its weaknesses—latency and endurance—are exactly the properties that matter least for the warm data classes identified earlier. No other technology in the table occupies the same position.

14. Bandwidth vs Latency: Matching HBF to Access Patterns

The single most important concept for evaluating HBF correctly is the distinction between bandwidth and latency. Bandwidth is how much data can be moved per unit time. Latency is how long a single access takes from request to first byte. HBM excels at both. HBF aims to match HBM on bandwidth while accepting substantially higher latency. Whether this trade-off is acceptable depends entirely on the access pattern of the data in question.

Access patterns can be characterized along two axes: granularity and predictability. Fine-grained, unpredictable accesses—reading a few bytes from an arbitrary address determined only at the moment of access—are latency-bound. Nothing can hide the latency because the system does not know what to fetch until it needs it. Coarse-grained or predictable accesses—reading large contiguous blocks, or reading data whose need can be anticipated ahead of time—are bandwidth-bound. Latency can be hidden by issuing the request early or by streaming continuously.

Model weight reads during decode are coarse-grained and perfectly predictable: the runtime knows exactly which layer executes next and which weights it needs. Expert reads in MoE are coarse-grained and predictable one layer ahead, because routing for layer N is computed at the end of layer N-1. Adapter loads are coarse-grained and predictable at request scheduling time. KV cache restoration for a resumed session is coarse-grained and predictable when the request arrives. Vector index traversals are moderately grained and partially predictable through batching and graph locality. All of these tolerate microsecond latency if bandwidth is high enough to complete the transfer within the available window.

Active KV cache reads during attention, by contrast, are fine-grained and driven by the current token’s attention pattern. Activation and scratch buffer accesses are fine-grained and immediate. These must stay in HBM. Any design that attempts to serve attention directly from HBF for the active sequence will suffer, regardless of HBF bandwidth.

The practical consequence is that HBF’s value is realized through prefetching and staging, not through transparent random access. The runtime must know, or predict, what data will be needed and move it into HBM in advance. Fortunately, inference runtimes already have this knowledge for most data classes because model execution is highly structured. The engineering work is in building the tiering logic, not in discovering the access patterns.

Buyers should demand workload-level benchmarks that reflect this. A raw sequential read bandwidth figure for HBF is necessary but not sufficient. The meaningful questions are: what is the sustained tokens-per-second for a model that does not fit in HBM but does fit in HBM plus HBF? What is the time-to-first-token for a resumed long-context session restored from HBF? What is the cold-start latency for an adapter loaded from HBF versus from host DRAM versus from network? What is the achieved concurrency at target latency with KV offload to HBF enabled? These are the numbers that determine business value.

There is also a systems-level bandwidth question. If HBF is streaming weights into HBM continuously, that traffic consumes HBM write bandwidth that would otherwise serve compute. The net benefit depends on the ratio of HBF bandwidth to HBM bandwidth, the model’s arithmetic intensity, and how well the runtime overlaps staging with computation. Early implementations will likely favor use cases where staging traffic is modest relative to compute time—expert swapping, adapter loading, cache restoration—before attempting full weight streaming for dense models.

15. Power, Cooling, and Rack Density Implications

Power is the binding constraint on AI datacenter growth in most regions. Operators face limits on grid connection capacity, per-rack power density, and cooling infrastructure. Any technology that improves inference throughput per watt has value beyond the accelerator bill of materials, because it increases the useful work extractable from a fixed power envelope.

HBF affects power in several ways. First, NAND flash consumes less standby power than DRAM because it does not require refresh. A terabyte of idle HBF costs far less energy than a terabyte of idle DRAM. For a memory tier that holds a large warm working set most of which is not being accessed at any instant, this matters. Second, moving data within the accelerator package consumes much less energy per bit than moving it across PCIe from host memory or across the network from storage. By keeping model pools, cache, and indexes on-package, HBF reduces the energy cost of every data movement that would otherwise leave the accelerator. Third, and most significantly, if HBF allows a given model to be served on fewer accelerators—because memory capacity rather than compute was the reason for spreading it across many—the power savings are proportional to the accelerator count reduction.

There are offsetting considerations. Adding HBF stacks to a package adds active power when those stacks are being read at high bandwidth. NAND read energy per bit is higher than DRAM read energy per bit, so a workload that streams heavily from HBF will pay for it. The net effect depends on the workload mix, but for the warm-data use cases where HBF is intended, average access intensity is low and the standby advantage dominates.

Thermal integration is a genuine engineering challenge. HBM stacks already present cooling difficulties because heat from the compute die and from the stacked memory must be extracted through a shared package. NAND has different thermal characteristics and a different temperature sensitivity profile than DRAM; elevated temperatures accelerate NAND wear and affect data retention. Accelerator designers integrating HBF will need to model thermal behavior carefully, and it is likely that HBF-equipped packages will be deployed predominantly in liquid-cooled systems, which are becoming standard for high-end AI infrastructure regardless.

From a rack density perspective, HBF is attractive. Today, expanding inference memory capacity often means adding host DRAM, adding NVMe drives, or adding entire accelerator nodes, all of which consume rack space and power distribution. Putting terabytes of memory on the accelerator package itself increases memory per rack unit without increasing node count. For operators constrained by floor space or power distribution rather than raw energy, this density improvement is directly valuable.

Operators should ask vendors for power figures under representative inference workloads rather than peak or idle numbers, and should model the fleet-level effect of accelerator count reduction rather than only per-package power increase. In most realistic scenarios where HBF is relevant, the fleet-level effect is favorable.

16. TCO, Cost per Token, and Capacity Economics

The business case for HBF rests on total cost of ownership rather than component price. An HBF-equipped accelerator will cost more than the same accelerator without HBF. The question is whether the additional memory allows the system to do enough more useful work that the cost per unit of work falls. For inference, the unit of work is the token, and the relevant metric is fully loaded cost per token including hardware depreciation, power, cooling, facility, networking, and operations.

The mechanisms by which HBF reduces cost per token are concrete. Higher concurrency from KV offload increases batch size during decode, which raises accelerator utilization and amortizes fixed costs over more tokens. Fewer accelerators per model reduce hardware and power cost when memory capacity rather than compute was the constraint. Faster cold starts and adapter loads reduce wasted accelerator time and allow more aggressive model oversubscription. Prefix and semantic caching reduce the compute per request. Reduced network traffic to storage and cache systems lowers infrastructure cost outside the accelerator node.

The magnitude of each effect depends on workload. For a hyperscaler serving a small number of very large models to millions of users with long contexts, the concurrency and accelerator-count effects dominate and could be large. For an enterprise serving a mid-sized model with RAG to thousands of employees, the retrieval co-location and semantic caching effects may matter more. For a multi-tenant platform, adapter density and cold-start improvement are central. A buyer should model their own traffic against these mechanisms rather than relying on a generic vendor claim.

A second economic dimension is supply. HBM and DRAM in general have experienced tight supply and rising prices as AI demand outpaces manufacturing capacity, and DRAM manufacturers have shifted wafer allocation toward HBM at the expense of conventional memory. NAND supply, while cyclical, comes from a broader base and has different capacity dynamics. A memory hierarchy that shifts a portion of AI memory demand from DRAM to NAND diversifies supply risk for buyers and may moderate the price pressure on HBM by removing the need to use it for capacity rather than bandwidth.

A third dimension is asset longevity. HBF-equipped systems with terabytes of near-package memory may remain useful for inference longer than HBM-only systems, because their memory capacity headroom accommodates model growth. In a market where accelerator generations turn over every twelve to eighteen months, extending the useful life of an inference fleet by even one cycle has significant depreciation implications.

None of this is guaranteed. The TCO case depends on HBF achieving its bandwidth and capacity targets at a cost premium that is modest relative to the accelerator, and on software delivering the utilization improvements in practice. But the structure of the argument is sound, and it is the reason major accelerator vendors and memory suppliers are engaging with the technology.

17. The Software Stack: Tiering, Prefetching, and Observability

Hardware creates possibility; software realizes value. HBF will deliver nothing if inference runtimes treat it as an opaque block device. The full benefit requires changes across several layers of the software stack, and the pace of those changes will determine how quickly HBF moves from interesting to essential.

The lowest layer is the accelerator driver and memory manager. HBF must be exposed to the runtime as a distinct memory space with known characteristics, and the runtime must be able to issue asynchronous transfers between HBF and HBM without host involvement. Ideally, the hardware provides DMA engines that move data between the tiers while compute continues, and the driver exposes completion signals that the runtime can use to sequence dependent operations. Address translation, wear leveling, and error correction should be handled in the HBF base die or the memory controller so that the runtime sees a flat, reliable address space.

The next layer is the inference serving engine. Engines must become tier-aware in their allocation decisions. The KV cache manager should support migrating cache blocks between HBM and HBF based on sequence state. The model loader should support placing weights in HBF and streaming layers or experts into HBM according to an execution schedule. The adapter manager should treat HBF as the adapter library and HBM as the active set. Scheduling must account for staging latency, issuing prefetches early enough that data arrives before it is needed. Several open-source serving engines already implement KV offload to host memory and disk, which provides a foundation; extending them to a near-package tier is an evolution rather than a rewrite.

Above the serving engine sit orchestration and placement systems. In a fleet, decisions about which models, adapters, and index partitions are provisioned into which nodes’ HBF become a placement problem analogous to content distribution. Nodes with a tenant’s adapter and index in HBF should receive that tenant’s traffic. Provisioning new assets into HBF should be scheduled to avoid disrupting serving. Because HBF is persistent, provisioning is a one-time operation per node rather than a per-restart cost, which simplifies orchestration but requires tracking what each node holds.

Retrieval systems need integration too. Vector search libraries that run on accelerators must be extended to read index partitions from HBF and to manage a DRAM or HBM cache of hot graph nodes. Database vendors offering accelerator-native vector search will need to add HBF as a storage backend. Semantic cache layers need the same treatment.

Observability is essential and often neglected. Operators need to see HBF bandwidth utilization, read and write volumes, staging latency distributions, HBM hit rates for tiered data, prefetch accuracy, wear indicators, thermal state, and the downstream effect on time-to-first-token and tokens per second. Without this telemetry, tuning tiering policies is guesswork, and problems such as prefetch misses or write amplification go undiagnosed. Vendors should ship HBF with a complete metrics surface exposed through standard monitoring interfaces.

Standardization would accelerate all of this. If HBF is exposed through a common interface across accelerator vendors, software investments transfer. If each accelerator vendor implements a proprietary HBF integration, the software ecosystem fragments and adoption slows. Sandisk’s engagement with HBM suppliers on interface standards and its advisory board of architects suggest an awareness of this risk. Buyers should favor platforms whose HBF integration aligns with emerging standards over those that lock software to a single vendor’s approach.

18. Deployment Scenarios: Hyperscale, Enterprise, Sovereign, and Edge

HBF’s value manifests differently across deployment contexts. Understanding these differences helps vendors position products and helps buyers assess relevance.

19. Supply Chain, Standardization, and the Vendor Ecosystem

HBF sits at the intersection of two supply chains that have historically been separate: the DRAM and advanced packaging ecosystem that produces HBM, and the NAND ecosystem that produces SSDs. Its success depends on cooperation across that boundary.

On the NAND side, Sandisk is the originator, but the concept is not inherently exclusive. Other NAND manufacturers have the process technology to produce high-layer-count 3D NAND and the wafer bonding capability to integrate logic dies. A collaboration between Sandisk and a major HBM supplier to standardize the HBF interface has been publicly announced, which suggests the technology is being positioned as a category with multiple sources rather than a single-vendor product. For buyers, multi-sourcing is important; a memory tier that is central to inference economics should not depend on one supplier.

On the packaging side, integrating HBF stacks alongside HBM on an interposer requires the same advanced packaging capacity that is already a bottleneck for HBM-equipped accelerators. If HBF is footprint-compatible with HBM, it does not require new packaging processes, but it does compete for the same interposer and assembly capacity. Accelerator vendors will need to decide how to allocate stack positions between HBM and HBF, and packaging partners will need to qualify NAND stacks in a thermal environment designed for DRAM.

Accelerator vendors are the gatekeepers. HBF only reaches the market when a GPU or custom accelerator designer integrates it into a product. Public statements suggest interest across the industry, and the composition of Sandisk’s advisory board indicates engagement with the architects who influence these decisions. Custom accelerator programs at hyperscalers, which are designed specifically for inference and have more freedom to depart from general-purpose GPU designs, may be early adopters. Merchant GPU vendors may follow with inference-specific variants.

Standards bodies matter. HBM is standardized through JEDEC, which is why multiple DRAM vendors can supply interoperable stacks to multiple accelerator vendors. A comparable standardization path for HBF would enable the same interoperability. Industry observers should watch for HBF-related activity in relevant standards organizations as a signal of ecosystem maturity.

For server OEMs, ODMs, and export-oriented system builders, HBF represents an opportunity to differentiate inference products. The current accelerator market is highly commoditized around a small number of reference designs. Inference-optimized systems with tuned HBM-to-HBF ratios, integrated retrieval software, and validated tiering stacks could command premium positioning in enterprise and sovereign markets. Storage solution vendors have an adjacent opportunity in the software and services layer that makes HBF useful.

Timing is a consideration for procurement planning. With sampling expected in the second half of 2026 and first inference devices in early 2027, HBF is relevant to hardware decisions being made now for deployments in 2027 and beyond. Buyers with long procurement cycles should engage vendors on HBF roadmaps today rather than waiting for general availability.

20. Risks, Limitations, and Open Questions

A balanced analysis must be clear about what could prevent HBF from delivering on its promise. Several risks are material.

Bandwidth delivery. The claim that HBF can approach HBM bandwidth at the stack level is ambitious. NAND read throughput per die is far lower than DRAM, so achieving high aggregate bandwidth requires massive parallelism across many dies and planes plus a very wide interface. If shipping products deliver a fraction of the targeted bandwidth, the range of workloads that benefit narrows to the most latency-tolerant cases such as adapter and model loading, and the weight-streaming and expert-swapping cases become marginal.

Latency and tail behavior. NAND latency is not only higher than DRAM but also more variable, due to background operations such as garbage collection and wear leveling. Inference service-level objectives are sensitive to tail latency. The HBF base die and controller must manage background operations so that they do not produce latency spikes during serving. This is a solved problem in enterprise SSDs but must be re-solved in a different architectural context.

Endurance. If HBF is used for write-heavy data such as active KV cache, wear could shorten device life. Proper tiering keeps the hottest writes in HBM, but operators and software must enforce that discipline. Vendors need to specify endurance under realistic inference write patterns, and buyers need to model their write volumes. Data retention at elevated package temperatures is a related concern that requires thermal design attention.

Thermal integration. Placing NAND next to a compute die dissipating several hundred to over a thousand watts is a challenging thermal environment. NAND performance and reliability degrade at high temperatures. Liquid cooling mitigates this, but package-level thermal design will be a gating factor for early adopters.

Software readiness. The value of HBF is realized only through tier-aware software. If serving engines, retrieval systems, and orchestration platforms are slow to add support, early HBF hardware may be underutilized. Buyers should assess the software roadmap alongside the hardware roadmap.

Ecosystem adoption. HBF requires accelerator vendors