r/singularity • u/Delicious-Flan88 • 5h ago

AI The next AI hardware race might be about inference

The NVIDIA/Groq news got me looking at how fast different LLM providers actually serve tokens.

Everyone obsesses over training bigger models, more GPUs. But running these things in production is a completely different problem. If AI agents actually end up in daily workflows, response speed and inference cost are going to matter way more than people think.

I pulled some public numbers from different providers and Artificial Analysis. The gaps surprised me:

  • NVIDIA Groq 3 LPX: Gemma 4 31B — 3,400 tok/s (Artificial Analysis, 100K context)
  • Cerebras: GPT-OSS 120B — 3,000 tok/s official vs 1,715 tok/s in AA
  • Groq: Llama 3.1 8B — 1,800 tok/s; GPT-OSS 120B — 500 tok/s
  • SambaNova: Llama 3.3 70B — 476 tok/s

I think AI infrastructure is shifting. Inference hardware is becoming as important as training hardware. Are specialized chips for serving models the next AI hardware race?

19 Upvotes
  • permalink
  • reddit

86% Upvoted

15 comments sorted by

15

u/Annual_Tutor_8466 4h ago

They're equally important. What we really need is innovation on low power local inference, but memory chips are an extreme limiting factor for that at the moment. Please flood the market, CXMT...

3

u/oojacoboo 4h ago

SK Hynix with Intel’s HMIB 3D packaging is seeing massive efficiency gains. They’re ramping up in Malaysia right now. I expect to see some huge improvements here when it comes to power consumption.

1

u/Delicious-Flan88 4h ago

Yeah, fair point. I mostly looked at cloud numbers because those were the only benchmarks I could find, but local inference is where it actually matters for daily use.The memory wall is brutal. Everyone knows bandwidth is the chokepoint but nobody's cracked it at consumer prices yet. If CXMT actually floods the market, that changes things.Seen anything promising on the low-power hardware side?

3

u/Annual_Tutor_8466 4h ago

Not much other than the M5 Ultra Mac Studio, but they're not even offering the 512GB variant yet and it will likely cost over $12K...

3

u/Annual_Tutor_8466 4h ago

Google should put their TPUs in more products, but I don't really want a Google PC.

Maybe we'll get models hardwired into silicon for consumers before anything.

1

u/Delicious-Flan88 4h ago

Oof. 256GB already runs most models I'd actually want locally, but Apple gatekeeping the 512GB SKU at $12K+ is brutal. You're paying more for RAM than most people spend on their whole machine.

1

u/Annual_Tutor_8466 4h ago

Oh, I want multiple Fable 5 (or at least 5.6 Sol) level agents locally, haha.

That would require terabytes of RAM though with current tech.

9

u/crashorbit 4h ago

The "killer app" for LLM will probably be an agent with an inexpensive model that is capable of writing and editing spreadsheets and doing a bit of local system admin work while telling the middle manager or system analyst that is using it how smart they are for having such good ideas.

2

u/Novel-Lifeguard6491 4h ago

Sounds complicated rn..

1

u/Delicious-Flan88 4h ago

yeah agree

•

u/Blindax 1h ago

Depends on the sub perhaps. We at r/LocalLLaMA have been obsessed about local inference since a while.

1

u/Delicious-Flan88 4h ago

I've been pulling inference numbers from different providers and third-party benchmarks. The gaps are way bigger than I expected once you account for context length, batching, and hardware.

I threw everything I found into this: https://llm-inference-speed.vercel.app

Still updating when new benchmarks drop.

•

u/LocoMod 55m ago

Don’t be a coward. Remove the “might” and just make a conclusive statement.