AI 寫字的時候在做什麼

What an AI is doing while it writes

一次只吐一個字

One token at a time

你看到的句子是長出來的，不是一次寫好的。每多一個字，整顆模型就得從頭再跑一次——這一條限制，決定了它有多慢、多貴。

The sentence grows; it isn't written all at once. Every extra word costs another full pass through the model — and that one constraint sets the speed and the bill.

Autoregressive ＝自迴歸・「自己剛剛的輸出，變成自己下一步的輸入」

Autoregressive = its own last output becomes its own next input

機制

The loop

它把自己剛寫的字，再讀一次。

It reads back what it just wrote.

模型一次只做一件事：看完目前所有的字，然後猜下一個。猜完，把猜到的字 接到句子尾巴，再從頭看一次——現在長了一個字。然後再猜。這就是自迴歸（autoregressive）：自己的輸出，變成自己的下一個輸入。

The model does exactly one thing: read everything so far, then guess the next token. Then it appends that guess and reads the whole thing again — now one token longer — and guesses again. That is autoregression: its own output becomes its own next input.

每一列都是一次完整的模型運算。注意左邊的句子每列長一格—— 而且每一列都得把整句重看一遍。

Each row is one complete run of the model. Notice the sentence on the left grows by one each time — and each row re-reads the whole thing.

自己跑一次

Run it yourself

它其實沒有在「選字」。

It isn't really picking a word.

每一步，模型吐出來的不是一個字，而是整本字典的機率分布—— 十幾萬個 token，每個都有一個分數。真正「選字」的是外面那層程式。按下面的按鈕，一次走一步。

At each step the model doesn't emit a word — it emits a probability distribution over the whole vocabulary: a hundred thousand-odd tokens, each with a score. The picking happens in code outside the model. Step through it below.

這裡示範的是最單純的貪婪解碼（greedy decoding）：每次都挑機率最高的。實際的產品通常改成照機率抽樣（sampling）， temperature 就是控制抽得多隨機的那個旋鈕—— 調高，它會挑到下面幾名；調到 0，就退回貪婪。同一個問題問兩次答案不一樣，原因通常在這裡，不在模型「改變主意」。

This demo uses the simplest rule, greedy decoding: always take the highest. Real products usually sample from the distribution instead, and temperature is the knob for how random that draw is — turn it up and the runners-up start winning; turn it to 0 and you're back to greedy. That, not the model "changing its mind", is usually why the same question gives two different answers.

為什麼快不起來

Why it can't go faster

一次請求，其實分成完全不同的兩段。

One request is really two different jobs.

這裡有一個很多人搞混的地方。「只能看左邊」不等於「只能一個一個算」。 causal attention 規定的是哪個位置可以看到哪些資訊，不是token 必須照什麼順序計算。

Here is the part almost everyone gets wrong. "Can only look left" is not the same as "must be computed one at a time." Causal attention governs which positions may see which information — not the order the tokens must be executed in.

你打進去的 prompt，每一個字都已經存在了。所以模型可以把它們全部一起丟進去算，一層一層往上推——這段叫 prefill（預填）。但生成就不行了：第 5 個字要等第 4 個字被挑出來，才知道它是什麼。這段叫 decode（解碼），而它是一條沒辦法插隊的鏈。

Every token of the prompt you typed already exists. So the model can push all of them through together, layer by layer — that stage is prefill. Generation cannot work that way: the 5th token doesn't exist until the 4th has been chosen. That stage is decode, and it is a chain you cannot skip ahead in.

想輸出 500 個字，就是 500 次完整的模型運算，而且不能同時做。這兩段的瓶頸也不一樣：prefill 卡在算力，decode 卡在記憶體頻寬—— 每跑一次都得把整份權重從記憶體搬過來，只為了產生一個字。

Want 500 tokens out? That's 500 full passes, and they cannot overlap. The two stages even bottleneck on different things: prefill is limited by compute, decode by memory bandwidth — every pass drags the whole weight set out of memory just to produce one token.

TTFT

第一個字的等待

time to first token

由 prefill 決定，所以跟你的 prompt 長度有關。長文件貼進去，第一個字就是會比較久。

Set by prefill, so it scales with how long your prompt is. Paste a long document and the first word takes longer.

tok/s

之後的吐字速度

tokens per second

由 decode 決定，幾乎跟 prompt 多長無關。這就是為什麼字會一個一個穩定地冒出來。

Set by decode, and nearly independent of prompt length. That's why the words then appear at a steady drip.

救命的那一招

The saving trick

那為什麼不用每次都從頭讀？

So why isn't it re-reading everything?

因為舊的字沒有變。既然「只能看左邊」，那第 1 個字的中間計算結果，在第 2、第 3、第 100 步都是一模一樣的——存起來就好。這個存起來的東西叫 KV cache（鍵值快取）。有了它，每一步只要算新的那一個字。

Because the old tokens haven't changed. Since each position only looks left, the intermediate work for token 1 is identical at step 2, step 3 and step 100 — so store it. That store is the KV cache. With it, each step only has to compute the one new token.

虛線格子＝從快取讀出來的，不用重算。沒有快取的話，生成 N 個字的成本是 N 的平方等級；有了快取，就變成線性。代價是：這份快取會一路長大，而且住在顯示記憶體裡。你的對話愈長，它吃掉的 VRAM 愈多—— 這就是「context 開很大很貴」的真正原因。

Dashed cells are read from the cache, not recomputed. Without it, generating N tokens costs on the order of N²; with it, linear. The price: the cache keeps growing and it lives in GPU memory. The longer your conversation, the more VRAM it eats — which is the real reason a big context window is expensive.

一句話

In one breath

每一個字，都是一次完整的模型運算；而且它得等前一個字先出生。

Every token is one full run of the model — and it has to wait for the token before it to be born.

會聽到的幾個詞

Words you'll hear

token: 模型處理的最小單位。不一定等於一個字：英文常是一個詞或詞根，中文常是一到兩個字，數字則常常一位數一個。; The smallest unit the model handles. Not quite a word: often a word or word-piece in English, one or two characters in Chinese, frequently one digit at a time for numbers.
autoregressive: 自迴歸。自己上一步的輸出，當成自己下一步的輸入。; Its own previous output becomes its own next input.
causal mask: 因果遮罩。規定每個位置只能看到自己左邊的內容。它管的是「能看到什麼」，不是「要照什麼順序算」。; The rule that each position may only see what is to its left. It governs what can be seen, not what order things are computed in.
prefill: 預填。把已知的 prompt 一次算完的那一段。可以平行，卡在算力。; The stage that processes the known prompt in one go. Parallel, and compute-bound.
decode: 解碼。一個字一個字生成的那一段。不能平行，卡在記憶體頻寬。; The stage that emits tokens one at a time. Not parallel, and memory-bandwidth-bound.
KV cache: 鍵值快取。把已經算過的 token 的中間結果存起來，讓每一步只算新的那一個。會隨對話變長而吃掉更多記憶體。; Stored intermediate results for tokens already processed, so each step computes only the new one. It grows with the conversation and eats memory.
temperature: 抽樣時的隨機程度旋鈕。0 ＝每次都挑最高分的；調高＝比較可能挑到後面幾名。; The randomness knob when sampling. 0 means always take the top scorer; higher means the runners-up get a real chance.
TTFT: time to first token，第一個字出現前的等待。由 prefill 決定。; Time to first token — the wait before anything appears. Set by prefill.

哪裡看得到

Where you'll find it

聊天機器人一個字一個字冒出來 chatbots typing one word at a time API 帳單按 output token 計價 APIs billing per output token 長 prompt → 第一個字比較慢 long prompt → slow first word streaming / SSE context 開大很貴 long context costing more

三件事這頁刻意簡化了。第一，「每個字一次完整運算」對單一請求成立，但伺服器會同時服務很多人（batching），把每一次運算攤給幾十個請求分，所以整體吞吐遠比你直覺的高。第二，有人在想辦法繞過這條鏈： speculative decoding（投機解碼）讓一個小模型先猜好幾個字，再讓大模型一次驗證，猜對就賺到；擴散式語言模型（diffusion LM）則乾脆整段一起改。第三，不是所有模型都是自迴歸的——分類器、encoder，以及像 Jev 那種直接讀出機率的決策模型，都在 prefill 結束後就停了，根本沒有 decode 這一段。順帶一提，模型怎麼知道要停？它會吐出一個特殊的結束 token（EOS）。至於它為什麼一次只能想一個字卻還這麼聰明，答案在另一頁。

Three deliberate simplifications. First, "one full pass per token" holds for a single request, but a server batches many requests into each pass, so total throughput is far higher than your intuition suggests. Second, people are working around the chain: speculative decoding has a small model guess several tokens ahead and the big one verify them in one pass, keeping whatever it got right; diffusion language models revise a whole span at once instead. Third, not every model is autoregressive — classifiers, encoders, and decision models that read probabilities out directly stop after prefill and never decode at all. And how does it know to stop? It emits a special end-of-sequence token. As for how it can be this capable while thinking one token at a time — that's the next page.

← knowhow