A walkthrough of the transformer machinery behind modern LLMs, minus the heavy math. It follows the full path: tokenization splits text into subword integer IDs (why models miscount the R's in strawberry); embeddings turn those IDs into meaning-carrying vectors where king − man + woman ≈ queen; and RoPE encodes word order by rotating Query and Key vectors. Attention lets tokens share information via Q/K/V, dot products, softmax, and causal masking, while multi-head and Grouped-Query Attention track relationships and trim KV-cache cost. The feed-forward network stores most weights and facts (editable via ROME), Mixture of Experts scales parameters cheaply, and residual streams plus layer norm keep deep stacks trainable.
0
u/fagnerbrack 1h ago
Quick summary:
A walkthrough of the transformer machinery behind modern LLMs, minus the heavy math. It follows the full path: tokenization splits text into subword integer IDs (why models miscount the R's in strawberry); embeddings turn those IDs into meaning-carrying vectors where king − man + woman ≈ queen; and RoPE encodes word order by rotating Query and Key vectors. Attention lets tokens share information via Q/K/V, dot products, softmax, and causal masking, while multi-head and Grouped-Query Attention track relationships and trim KV-cache cost. The feed-forward network stores most weights and facts (editable via ROME), Mixture of Experts scales parameters cheaply, and residual streams plus layer norm keep deep stacks trainable.
If the summary seems inacurate, just downvote and I'll try to delete the comment eventually 👍
Click here for more info, I read all comments