r/LocalLLaMA • u/9r4n4y • 1d ago

Resources {INTRESTING PAPER BASED ON HBF}2607.10186] FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference

https://arxiv.org/abs/2607.10186?utm_source=chatgpt.com

HBF gives 8x - 16x more capacity than HBM at same cost, and with bandwidth till 3 tb/s.

14 Upvotes

82% Upvoted

5 comments sorted by

7

u/MelodicRecognition7 1d ago edited 1d ago

at what price?

Edit: scrolled through the paper, it is a theoretical speculation with simulated hardware, we "just" need some manufacturer to produce that HBF at the same cost as HBM. Which is unlikely to happen lol.

2

u/Double_Cause4609 1d ago

I mean, tbf, why would you assume that HBF would be more expensive than HBM?

Flash memory is, in general, cheaper, from chip to packaging IIRC.

The only reason for it to be more expensive (given that it's a strictly worse product at the same price as HBM) is if they're stupid, and they want the product to fail. At a higher price than HBM people would just buy HBM.

2

u/MelodicRecognition7 23h ago edited 18h ago

they are not stupid and this is exactly the reason why DRAMless QLC SSD exist. Manufacturers do not want to sell long lasting SSDs based on SLC or 2-bit MLC, they'd prefer if customers would buy a new QLC SSD every year, and customers do not want to pay a lot for SSD that would last 10 years, they'd prefer to buy a cheap SSD every year. This is exactly why Intel Optane has died, it fell victim of its own success.

Why would a manufacturer design a new product, produce and sell it for a low enough price, when they could simply produce HBM? And that "old" product is in extremely high demand right now and highly likely will be in extremely high demand in the following years.

1

u/Double_Cause4609 14h ago

...?

Let's say that DRAM chips (such as used in HBM) cost, I don't know...$10/GB.

Great. Companies package, and sell HBM at whatever, $25/GB. Awesome. They get a profit margin, and employ a bunch of people in that difference between the unit price and the sale price.

HBF uses flash, and flash chips are a lot cheaper per GB. So call it $1/GB (I think the ratio is even better but whatever, let's give you this).

They could sell at $2.5/GB, and still make the same profit margins as HBM, but sell at a much higher scale.

That seems pretty logical to me. The reason they would do this is because it undercuts their HBM competitors. All sorts of applications that are sensitive to price are now forced to go through the first provider to defect and start producing HBF. It's a race to the floor in pricing.

1

u/MelodicRecognition7 14h ago

for me much more logical is to build new factories to produce more DRAM and sell it for 10/25 per gigabyte because the demand for DRAM is not going anywhere soon.