The $4.1M Reality Check: What It Actually Costs to Run a 2.8-Trillion-Parameter MoE Model On-Prem

Recently, after Kimi K3 went online, everyone exclaimed that it's another DeepSeek moment. Haha, maybe it should be called the Kimi moment instead.

The release of K3 is no less significant than the sensation caused by the initial release of DeepSeek R1. Our boss was also among the first to top up and try it out (pretty trendy, I haven't even used it yet, it's too expensive).

He came back and found me: "Xiao X, we also need to learn from the experience of leading companies in the industry! From now on, let everyone fully use Kimi K3 for R&D."

I said: "Boss, the official API pricing for Kimi right now isn't cheap. The cheapest 49-yuan plan on the official site doesn't support K3.

You need at least the 99-yuan plan. But the 99-yuan plan has a 256K context window. To properly use the 1-million-token version, you need the 699-yuan plan.

Our R&D department currently has dozens of people. Even if we buy 30 subscriptions, that's still over 20,000 yuan per month."

Boss: "Is it that expensive? I think the 99-yuan one works pretty well."

I said: "Actually, for simple trials, the 99-yuan one is fine, but for our R&D use, the 699-yuan one is really better."

Boss: "Oh, okay. That cost is indeed quite high. How about this: we wait for K3 to be open-sourced and then deploy it locally. That way, we only spend money once and don't have to top up every month.

Go back and write a proposal, calculate the costs, and hand it in to me later."

So I walked out the door, utterly baffled, silently cursing the name of a certain plant in my heart.

---

Since you want it, I'll calculate it for you:

Kimi K3, as an ultra-large-scale Mixture of Experts (MoE) model with 2.8 trillion parameters, requires a full enterprise-level engineering effort for local deployment.

It can't be done with one of those so-called DeepSeek all-in-one machines; it absolutely must be a cluster.

Let's not talk about anything else first; let's talk about GPUs, which are the biggest expense.

The official recommendation requires at least a supernode composed of 64 or more accelerator cards for a complete deployment.

This type of supernode is typically composed of 64-1024 domestic accelerator cards (such as Huawei Ascend 910B, etc.) or overseas flagship cards (such as H200, H100) interconnected via high-speed links, and requires supporting 800G/1.6T high-speed optical modules and liquid cooling solutions.

Let's take the H200 as an example here. Currently, this GPU is absolutely leading in the computing field. Roughly 64 H200s with 141GB of VRAM are needed to barely run it and reach a basically usable state.

However, because MoE models require high-frequency data synchronization between experts during inference, network bandwidth is the key factor determining performance.

So, you also need NVLink switches / interconnect bridges to achieve high-bandwidth, low-latency communication between GPUs.

Additionally, you need at least 800G-level optical modules (1.6T if conditions permit) for high-speed data transmission within or across racks.

Finally, you need high-speed Ethernet switches supporting the RDMA/RoCE protocol to ensure that intra-cluster data exchange does not become a bottleneck.

These components need to be installed on enterprise-grade motherboards to function. Custom motherboards supporting multiple PCIe 5.0 lanes and multiple GPU slots (such as 8-card or 4-card server motherboards) are required. Taking an 8-card motherboard as an example, 64 H200s require at least 8 top-tier server chassis.

The CPU can't be too poor either. It needs dual-socket enterprise-grade CPUs, at least Intel Xeon Sapphire Rapids or AMD EPYC Genoa series, to provide sufficient PCIe lanes and prevent the CPU from becoming a bottleneck in data scheduling.

Memory can't be too small either, given the context of millions of tokens. Besides the GPU's HBM, a massive amount of system memory is needed to store the KV Cache and intermediate activation values.

Therefore, each server must be equipped with at least 1TB - 2TB of DDR5 memory.

With memory, you also need storage space.

Enterprise-grade SSDs must be used to store the 1.4TB model weight files, datasets, and cache disks for inference.

This requires a RAID array, with no less than 8TB per server, and you can't use poor-quality drives because the read/write I/O bandwidth can't be too low.

The main body of the machine is basically complete, but this isn't everything.

With the main body ready, cooling is needed.

The H200 has extremely high single-card power consumption under high-load MoE scenarios. With 64 cards packed into one cabinet, traditional air cooling simply won't work.

So, you absolutely must use cold-plate liquid cooling or immersion liquid cooling.

But the machine still needs to power on, and powering on requires electricity. A single cabinet might need 160kW, so it has to be split.

The cabinet must be equipped with enterprise-grade titanium power supplies. This isn't even mentioning the cost of renovating the server room! (We'll assume here that the server room doesn't need renovation.)

Alright, everything is basically assembled here. We won't count the cost of miscellaneous small accessories.

What is the total cost?

A conservative estimate is 30 million!

---

It's important to note that this 30 million is a one-time investment and does not include the cost of consumables or subsequent usage.

Let's assume the company has a high-spec server room, doesn't need to rent a professional IDC data center, and has its own high-bandwidth dedicated network line.

We'll only calculate the electricity cost here, nothing else.

The power consumption of a single H200 GPU under full load is about 700W (0.7kW).

So, the total power consumption of the 64 H200 GPUs themselves is: 64 × 0.7kW = 44.8kW.

In an actual server room, besides the power consumption of the GPUs themselves, you also need to consider the energy consumption of supporting facilities like server motherboards, CPUs, memory, network switches, and air conditioning for cooling.

Data centers typically use PUE (Power Usage Effectiveness) to measure this. The average PUE for domestic data centers is about 1.55.

So, the actual power draw of the entire cluster needs to be multiplied by a factor of 1.55: 44.8kW × 1.55 ≈ 69.44kW.

Since industrial electricity is not residential electricity, let's use 1.5 yuan per kWh as a baseline. For 1 hour of operation, the electricity cost is: 69.44 kWh × 1.5 yuan/kWh ≈ 104.16 yuan/hour.

Let's take 365 days a year, 24 hours of full-load operation as an example: 104.16 x 24 x 365 ≈ 912,000 yuan.

That means the annual electricity cost alone is 910,000 yuan, not counting anything else. This is purely the electricity cost.

---

So, I excitedly took the bill to the boss: "Boss, look, the cost for local deployment isn't that much. The equipment cost alone is about 30 million yuan. This doesn't include costs for the site, server room renovation, etc.

Then, the subsequent annual electricity cost for putting it into use is a little over 910,000 yuan. This doesn't include consumables and other equipment costs, only the electricity cost."

I watched the boss's face turn from white to red, then a bit purple, and finally a bit black. He didn't say a word.

After a long silence, the boss spoke: "Xiao X, the company needs to hold a meeting to discuss this matter. Leave the materials and go back for now."

The moment I walked out of the office, I saw the boss pick up that bill, his hand trembling for some reason.