Token
Also known as AI token · tokenization
A unit of input or output processed by a model, often part of a word.
In the real world
The word 'unbelievable' may be split into several tokens.
Also known as AI token · tokenization
A unit of input or output processed by a model, often part of a word.
The word 'unbelievable' may be split into several tokens.