Flash attention is an option for accelerating training and inference. Only NVIDIA GPUs of Turing, Ampere, Ada, and Hopper architecture, e.g., H100, A100, RTX 3090, T4, RTX 2080, can support flash attention. **You can use our models without installing it.**
Sign in to like and favorite skills
Flash attention is an option for accelerating training and inference. Only NVIDIA GPUs of Turing, Ampere, Ada, and Hopper architecture, e.g., H100, A100, RTX 3090, T4, RTX 2080, can support flash attention. You can use our models without installing it.
4.32.0 is preferred.
Please check if you have updated the code to the latest, and correctly downloaded all the sharded checkpoint files.
qwen.tiktokenThis is the merge file of the tokenizer. You have to download it. Note that if you just git clone the repo without git-lfs, you cannot download this file.
Run the command
pip install -r requirements.txtYes, see
web_demo.pycli_demo.pyYes, run
python cli_demo.py --cpu-onlyYes. See the function
chat_streammodeling_qwen.pyThis is because tokens represent bytes and a single token may be a meaningless string. We have updated the default setting of our tokenizer to avoid such decoding results. Please update the code to the latest version.
Please check if you are loading Qwen-Chat instead of Qwen. Qwen is the base model without alignment, which behaves differently from the SFT/Chat model.
Yes, the quantization is supported by AutoGPTQ.
Updating the code to the latest version can help.
Please ensure that NTK is applied.
use_dynamc_ntkuse_logn_attnconfig.jsontruetrueYes, we now support SFT, including full-parameter finetuning, LoRA, and Q-LoRA. Also you can check other projects like FastChat, Firefly, LLaMA Efficient Tuning, etc.
However, temporarily we do not support RLHF. We will provide the code in the near future.
In our training, we only use
<|endoftext|>When downloading our official docker image, you may have a slow download speed due to some network issues. You can refer to Alibaba Cloud Container Image Service to accelerate the download of official images.