大模型為什麼可以又大又快

Why big models can still be fast

只叫醒兩位專家

Wake only two experts

一顆巨大的 AI 腦袋，在任何一個瞬間，大部分都在睡覺—— 而且是故意設計成這樣的。

A giant AI brain where almost all of it is asleep at any given moment — on purpose.

Sparse MoE ＝稀疏混合專家・一屋子的小專家＋一個決定叫誰起床的 router

Sparse MoE = sparse mixture of experts · a room of small experts + a router that decides who wakes up

問題

The problem

每一個字，都驚動整顆腦袋。

Every word wakes the whole brain.

傳統的密集模型（dense model）沒有選擇：你打進去的每一個 token，都得穿過全部的參數走一遍。模型愈大，每個字就愈貴。稀疏混合專家換掉的就是這條規則——同樣的知識量留在箱子裡，但一次只點亮其中一小片。

A dense model has no choice: every token you type must push through all of the parameters. The bigger the model, the more every word costs. Sparse MoE changes that rule — the same knowledge stays in the box, but only a sliver of it lights up at a time.

兩邊裝在機器裡的知識一樣多。差別只在於：稀疏的那一邊不把全部都打開。

The same knowledge sits in the box either way. The sparse one just doesn't switch all of it on.

零件

The parts

一個門房，和一屋子的專家。

A doorman, and a room full of specialists.

一個字進來。一個很小的 router（路由器，也叫 gate）幫每一位專家打分數，然後只把這個字交給分數最高的兩位。其他人繼續睡。點點看下面的字。

A word arrives. A tiny router (also called the gate) scores every expert, then hands the word to just the best two. The rest keep sleeping. Tap a word below.

router 幫 8 位專家打分 router scores all 8

最高的 2 位醒來 top 2 wake up

真正的專家沒有職稱。它們學到的是「數字後面通常接什麼」這類沒有名字的模式，不是「化學」「歷史」這種人看得懂的分類。上面這些標籤是個善意的謊—— 它讓機制看得見，但別把它當成模型內部真正的樣子。

Real experts have no job titles. They specialise in patterns nobody named — "what usually follows a number" — not human categories like "chemistry" or "history". The tidy labels above are a kind lie: they make the mechanism visible, but don't mistake them for what is really inside.

意外的地方

The surprise

而且每一層，都重新挑一次。

And it happens again on every floor.

專家不是一個「你去拜訪一次」的房間。模型的每一層都有自己的一組專家，也都有自己的 router。同一個字往上走，每一層被送去的地方都可能不一樣。

The experts aren't a room you visit once. Every layer of the model has its own set and its own router. The same word travels upward, and each floor can send it somewhere different.

8 選 2 ＝每層 28 種組合。疊 32 層，同一個字可以走的路線超過 10⁴⁶ 條。這些路線沒有人設計，是訓練長出來的。

Pick 2 of 8 → 28 combinations per floor. Over 32 floors that's more than 10⁴⁶ different routes one word could take. Nobody designed those routes; training grew them.

代價

The catch

整間廚房要租，只付有下廚的廚師。

You rent the whole kitchen. You pay only the chefs who cook.

睡著的專家還是佔位子。它們全部都得待在記憶體裡，隨時可能被叫到。所以 MoE 換來的不是「變小」，而是記憶體照樣大、但每個字的計算量變小。

Sleeping experts still take up space. All of them must sit in memory, ready to be called. So MoE doesn't make the model smaller — it keeps memory large while the compute per word gets small.

100%

記憶體 — 不管醒著還是睡著，每一位專家都得放在 RAM／VRAM 裡

Memory — every expert sits in RAM/VRAM, awake or not

12.5%

計算 — 只有被選中的專家真的做乘法

Compute — only the chosen experts actually multiply anything

這就是整個交易：跑起來便宜，放起來昂貴。 也因此 MoE 在資料中心的大卡上非常划算，在你的筆電上卻很尷尬—— 算力用不完，記憶體先爆掉。

That's the whole trade: cheap to run, expensive to hold. Which is why MoE is a bargain on datacentre cards and awkward on your laptop — you run out of memory long before you run out of compute.

12.9 / 46.7B

活躍／總參數

active / total

Mixtral 8×7B　8 位專家選 2，把 MoE 帶進開源主流的那一個。

Mixtral 8×7B — 8 experts, pick 2. The one that brought MoE into the open-source mainstream.

37 / 671B

活躍／總參數

active / total

DeepSeek-V3　每個字只動用 5.5%。極端到讓人重新算成本。

DeepSeek-V3 — just 5.5% per word. Extreme enough to make people redo their cost maths.

22 / 235B

活躍／總參數

active / total

Qwen3-235B-A22B　名字裡的 A22B 就是「活躍 22B」的意思。

Qwen3-235B-A22B — the A22B in the name literally means 22B active.

會出錯的地方

What goes wrong

router 會偏心。

The router plays favourites.

沒有人管的話，router 會一直把東西丟給它已經信任的那一位。那位專家被操到爆，其他人則因為從來沒拿到題目，永遠學不會—— 於是你花錢養的參數，有一大半是死的。

Left alone, the router keeps sending everything to the expert it already trusts. That one drowns, and the others never learn because they never get any work — so half the parameters you paid for sit dead.

訓練時多加一條小小的懲罰項：只要分配得不平均就扣分。這叫 auxiliary load-balancing loss（輔助負載平衡損失）。它會犧牲一點準確度，換回來的是每一位專家都真的被用到。

Training adds one small extra penalty whenever the load is lopsided — the auxiliary load-balancing loss. It costs a little accuracy and buys back every expert.

一句話

In one breath

蓋一顆很大的腦袋，然後每個字只打開它需要的那一小片。

Build a huge brain, then switch on only the sliver each word needs.

會聽到的幾個詞

Words you'll hear

token: 被路由的最小單位，大約是一個字或一個詞片段。; One word-ish chunk of text — the thing being routed.
expert: 不是一個獨立的模型，只是某一層裡面的一塊前饋網路（FFN）。一層可以有幾十到幾百個。; Not a separate model — just one slab of feed-forward network inside a layer. Dozens to hundreds per layer.
router / gate: 很小的一個評分器，讀進 token，幫所有專家排名次。; A tiny scorer that reads the token and ranks every expert.
top-k: router 留下幾位贏家。常見是 2 或 8。; How many winners the router keeps. Usually 2 or 8.
sparse: 整件事的重點：對大多數 token 來說，大多數專家什麼都沒做。; The whole point: for most tokens, most experts do nothing.
active params: 實際跑過的參數量。決定你的速度和帳單的，是這個數字，不是總參數。; How much of the model actually ran. This number predicts your speed and your bill — not the total.

哪裡看得到

Where you'll find it

Mixtral 8×7B — 12.9B / 46.7B DeepSeek-V3 — 37B / 671B Qwen3-235B-A22B Llama 4 Scout — 17B / 109B gpt-oss-120b — 5.1B / 117B Switch Transformer（2021，起點） Switch Transformer (2021, the origin)

三個值得知道的簡化。第一，專家不是獨立的模型，你沒辦法把「化學專家」單獨拆出來用—— 它只是某一層裡的一塊權重。第二，不是每一層都是 MoE：多數模型是 dense 層和 MoE 層交錯。第三，有些設計會放共享專家（shared expert），對每個 token 都保持開啟，專門吃掉共通的部分，讓其他專家可以更專心地分工；DeepSeek 系列就是這樣做的。另外 router 的選擇是離散的（挑了就是挑了），沒辦法直接微分—— 怎麼把梯度傳回 router，本身就是這個領域一大半論文在處理的事。

Three simplifications worth knowing. First, experts are not standalone models — you cannot pull "the chemistry expert" out and run it; it is just a block of weights inside one layer. Second, not every layer is MoE: most models interleave dense layers with MoE layers. Third, some designs add a shared expert that stays on for every token, absorbing the common work so the others can specialise harder — the DeepSeek family does this. And the router's choice is discrete, so it can't be differentiated directly; how to get gradients back to the router is what a good half of the papers in this area are about.

← knowhow