Language Models
Tokenization
Turning text into model input
About
How language is split into tokens before a model sees it. Vocabulary size, subword units, and the tradeoffs that show up later in prompting and RAG.
Language Models
Turning text into model input
How language is split into tokens before a model sees it. Vocabulary size, subword units, and the tradeoffs that show up later in prompting and RAG.