kaifaai.app AI 快讯 · 2026-10-23

MIT揭示AI推理隐性存储成本被严重低估

MIT Technology Review指出，AI推理产生的缓存、日志与中间状态存储开销，常被忽略却占总TCO 30%以上。

研究显示，在典型企业LLM服务链路中，KV缓存、prompt历史归档、结构化输出日志及RAG向量索引更新等持久化操作，其存储I/O成本与冷热分层管理开销，平均占推理总拥有成本（TCO）的32%-47%，远超GPU算力支出占比。更关键的是，这些成本随请求并发度非线性增长，且缺乏统一监控工具——导致多数团队在压测阶段才发现存储瓶颈。这对早期AI团队意味着：模型选型必须同步评估其缓存友好性（如是否支持PagedAttention）、服务网关需内置存储成本计量模块，并优先采用可压缩/可裁剪的中间表示格式。

MIT reveals AI inference’s hidden storage cost is 30%+ of TCO

Empirical analysis shows KV cache persistence, prompt history archiving, structured output logging, and RAG vector index updates contribute 32–47% of LLM service TCO — dwarfing GPU compute costs in many high-concurrency scenarios. Critically, these costs scale non-linearly with request volume and lack standardized observability tooling, causing teams to discover bottlenecks only during late-stage load testing. For early AI teams, this implies three immediate actions: prioritize models with cache-efficient attention (e.g., PagedAttention support); embed storage cost telemetry into API gateways from day one; and adopt compressible, trimmable intermediate representations — because infrastructure cost predictability starts with visibility into what your model *leaves behind*.

来源：MIT Technology Review — AI’s hidden storage bill: What inference leaves behind