Qwen-7B uses BPE tokenization on UTF-8 bytes using the `tiktoken` package.
Sign in to like and favorite skills
Qwen-7B uses BPE tokenization on UTF-8 bytes using the
tiktokenbytesstrfrom transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True)
The regular tokens are BPE tokens learned from byte sequences of texts encoded using the UTF-8 encoding. While this allows tokenization of all texts and no unknown token exists, it may fall back to using single bytes when tokenizing uncommon texts. You may encounter UTF-8 decoding errors and as the errors are default to
replaceerrors="ignore"decodefrom_pretrainederrors>>> tokenizer.decode([51461]) ' �' >>> tokenizer.convert_ids_to_tokens([51461]) [b' \xe6\xa0'] >>> b' \xe6\xa0'.decode("utf-8", errors='replace') ' �' >>> tokenizer.decode([51461, 117]) ' 根' >>> tokenizer.convert_ids_to_tokens([51461, 117]) [b' \xe6\xa0', b'\xb9'] >>> b' \xe6\xa0\xb9'.decode("utf-8", errors='replace') ' 根'
The mapping from regular tokens (in
bytestokenizer.get_vocab()The special tokens signify special functions to the model, e.g., reaching the end of a document. In theory, they do not exist in the input texts and only appear after the input texts are processed. Their surface forms, e.g.,
<|endoftext|><|endoftext|><|endoftext|><|im_start|><|im_end|><|extra_0|><|extra_204|>strtokenizer.special_tokensThe concepts of
boseosunkpadmaskseppadfrom transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, pad_token='<|endoftext|>')
WARNING: For our pretrained models, setting
,bos,eos, and such makes no sense. Unknown behavior may be introduced if you set them without fine-tuning that designates their meanings to the model. Especially, you should not useunkas<|endoftext|>, unless you are sure that the end of a sentence and the end of a document, which may contain many sentences, are the same in your scenario.eos
As special tokens are different from regular tokens, what will happen if the surface forms of a control token appear in the input texts? For example, note that a piece of text like this
print("<|endoftext|>")
should be tokenized as
ids:[1350, 9639, 91, 8691, 723, 427, 91, 82598] tokens: [b'print', b'("<', b'|', b'endo', b'ft', b'ext', b'|', b'>")']
not
ids: [1350, 445, 151643, 899] tokens: [b'print', b'("', '<|endoftext|>', b'")']
Our default used to be the correct one, that is, treating the surface forms of special tokens just like regular texts, and special tokens should be taken cared of by developers after tokenization of the texts. However, this conflicts with (albeit unsafe) practice in the community, and adds another step for developers to reuse their wheels.
The default behavior has been changed to parse the surface forms of all the known special tokens as special tokens. To enable injection prevention, pass
allowed_special=set()>>> tokenizer('print("<|endoftext|>")', allowed_special=set()) {'input_ids': [1350, 9639, 91, 8691, 723, 427, 91, 82598], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
You can control the behavior in a fine-grained manner by passing a set of
strallowed_special>>> tokenizer('print("<|extra_0|>")<|endoftext|>', allowed_special={'<|endoftext|>'}) {'input_ids': [1350, 9639, 91, 15460, 62, 15, 91, 82598, 151643], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}
You can also make the tokenizer raise errors if the surface forms of certain special tokens are encountered in the input texts by passing a collection of
strdisallowed_special>>> tokenizer('print("<|extra_0|>")<|endoftext|>', allowed_special={'<|endoftext|>'}, disallowed_special=('<|extra_0|>', )) ... ValueError: Encountered text corresponding to disallowed special token '<|extra_0|>'. If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|extra_0|>', ...}`. If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|extra_0|>'})`. To disable this check for all special tokens, pass `disallowed_special=()`.
For more information on
allowed_specialdisallowed_specialtiktokenThe new default is the same as
>>> tokenizer('print("<|endoftext|>")', allowed_special="all", disallowed_special=()) {'input_ids': [1350, 445, 151643, 899], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}
WARNING: Read carefully, be aware of what you are doing, and use at your own risk. There are certain caveats regarding how your vocabulary is produced.
The tokenizer of Qwen models are based on BPE and you cannot directly expand the vocabulary by adding words to the vocabulary. The intermediate merges are needed for tokenization. Please follow the steps to obtain such information.
Prepare a plain text file
qwen_extra_vocab.txt\tAn example is given below:
我是一只猫 20 你是一只猫 10 他是一只猫 5 一只 200 一只猫 100 夸张的 比喻手法 20
The frequencies are needed to compute the BPE.
Prepare the base vocabulary file, e.g.,
qwen.tiktokenThere are 151,643 regular tokens and 208 control tokens in the vocabulary for Qwen models. For simplicity, the start index can be set as 151,851, which is the default value. You can, of course, override the many inactive control tokens, but you will need to modify the tokenizer code.
Run the following command:
python add_merges.py qwen.tiktoken qwen_extra.tiktoken qwen_extra_vocab.txt
add_merges.pyqwen_extra_vocab.txtqwen_extra.tiktokenIt is a pure Python implementation, so please expect it to be slow if you are adding a lot of words.
Please note that not all words can be added due to pre-tokenization. You will get warnings if you try to add such word:
WARNING - 夸张的 比喻手法 would be pre-tokenized to ['夸张的', ' 比喻手法'], and thus cannot be added to vocabulary WARNING - word 一只 is already a token b'\xe4\xb8\x80\xe5\x8f\xaa', skipping INFO - number of existing merges: 151643 INFO - number of words for expanding: 4 DEBUG - (b'\xe4\xb8\x80\xe5\x8f\xaa', b'\xe7\x8c\xab') (一只猫) is selected as the next merge with freq 100 DEBUG - (b'\xe5\x8f\xaa', b'\xe7\x8c\xab') (只猫) is selected as the next merge with freq 35 DEBUG - (b'\xe6\x98\xaf\xe4\xb8\x80', b'\xe5\x8f\xaa\xe7\x8c\xab') (是一只猫) is selected as the next merge with freq 35 DEBUG - (b'\xe6\x88\x91', b'\xe6\x98\xaf\xe4\xb8\x80\xe5\x8f\xaa\xe7\x8c\xab') (我是一只猫) is selected as the next merge with freq 20 DEBUG - (b'\xe4\xbd\xa0', b'\xe6\x98\xaf\xe4\xb8\x80\xe5\x8f\xaa\xe7\x8c\xab') (你是一只猫) is selected as the next merge with freq 10 DEBUG - (b'\xe4\xbb\x96', b'\xe6\x98\xaf\xe4\xb8\x80\xe5\x8f\xaa\xe7\x8c\xab') (他是一只猫) is selected as the next merge with freq 5 INFO - number of newly learned merges: 6
The
qwen_extra.tiktoken5LiA5Y+q54yr 151851 5Y+q54yr 151852 5piv5LiA5Y+q54yr 151853 5oiR5piv5LiA5Y+q54yr 151854 5L2g5piv5LiA5Y+q54yr 151855 5LuW5piv5LiA5Y+q54yr 151856
You may use the file as follows in your code:
from transformers import AutoTokenizer >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True, extra_vocab_file="qwen_extra.tiktoken") >>> len(tokenizer) 151857 >>> tokenizer("我是一只猫") {'input_ids': [151854], 'token_type_ids': [0], 'attention_mask': [1]}
Note: You need the latest tokenizer code, i.e., after 2023-10-08, to use the
extra_vocab_fileqwen.tiktokenqwen_extra.tiktokenCertainly, you will need to finetune the model for the new tokens to work.
The tokenizer of Qwen operates directly on UTF-8 byte sequences, unlike others, e.g., SentencePiece that operates on Unicode codepoints/characters and falls back to UTF-8 byte sequences for the unknown (IIRC). The thing is if the frequencies are computed on limited data, the Unicode codepoint boundary may not be correctly recognized. In theory, it could be a problem for fine-tuned models using the expanded vocabulary with limited data.
For example, it could happen that
b'\x80\xe5'b'\xe4\xb8\x80\xe5\x8f\xaa'一只一b'\xe4\xb8\x80'只b'\xe5\x8f\xaa'Our advice is that to be safe, you should gather the Unicode codepoints from all the words you need to add, and also add them to the file with frequencies higher than the sum of the frequencies of the corresponding words. But since Qwen has most of the Chinese words, it could be okay to just add the Chinese words alone.
For curious minds, you will also notice that in the given example,
一只只猫是一一只是|一|只|猫是一|只|猫 -> 是一|只猫 -> 是一只猫This is the characteristic for plain BPE: it is based solely on distribution, meaning it does not have knowledge of which bytes can form a valid Unicode codepoint, character, or meaningful word.
The byproduct is that text may be sub-tokenized differently in different contexts, even for words containing only ASCII characters.
>>> tokenizer.tokenize("Panda") [b'P', b'anda'] >>> tokenizer.tokenize(" Panda") [b' Panda'] >>> tokenizer.tokenize("Pandas") [b'P', b'andas'] >>> tokenizer.tokenize(" Pandas") [b' Pand', b'as']
This simply suggests that those combinations occur more frequently in the data. If you have vast amount of training data, it should not be a problem.