| [**Demo**](https://lmarena.ai/) | [**Discord**](https://discord.gg/6GXcFg3TH8) | [**X**](https://x.com/lmsysorg) |
Sign in to like and favorite skills
FastChat is an open platform for training, serving, and evaluating large language model based chatbots.
FastChat's core features include:
pip3 install "fschat[model_worker,webui]"
git clone https://github.com/lm-sys/FastChat.git cd FastChat
If you are running on Mac:
brew install rust cmake
pip3 install --upgrade pip # enable PEP 660 support pip3 install -e ".[model_worker,webui]"
Vicuna is based on Llama 2 and should be used under Llama's model license.
You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos. Downloaded weights are stored in a
.cache~/.cache/huggingface/hub/<model_name>See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below.
NOTE:
is required for 16K versions.transformers>=4.31
| Size | Chat Command | Hugging Face Repo |
|---|---|---|
| 7B | | lmsys/vicuna-7b-v1.5 |
| 7B-16k | | lmsys/vicuna-7b-v1.5-16k |
| 13B | | lmsys/vicuna-13b-v1.5 |
| 13B-16k | | lmsys/vicuna-13b-v1.5-16k |
| 33B | | lmsys/vicuna-33b-v1.3 |
Old weights: see docs/vicuna_weights_version.md for all versions of weights and their differences.
Besides Vicuna, we also released two additional models: LongChat and FastChat-T5. You can use the commands below to chat with them. They will automatically download the weights from Hugging Face repos.
| Model | Chat Command | Hugging Face Repo |
|---|---|---|
| LongChat-7B | | lmsys/longchat-7b-32k |
| FastChat-T5-3B | | lmsys/fastchat-t5-3b-v1.0 |
(Experimental Feature: You can specify
--style richFastChat supports a wide range of models, including LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, xDAN-AI and more.
See a complete list of supported models and instructions to add a new model here.
The command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B. See the "Not Enough Memory" section below if you do not have enough memory.
--model-pathpython3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5
You can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2
Tips: Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs. You can use
--max-gpu-memorypython3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiB
This runs on the CPU only and does not require GPU. It requires around 30GB of CPU memory for Vicuna-7B and around 60GB of CPU memory for Vicuna-13B.
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu
Use Intel AI Accelerator AVX512_BF16/AMX to accelerate CPU inference.
CPU_ISA=amx python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu
Use
--device mps--load-8bitpython3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit
Vicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.
Install the Intel Extension for PyTorch. Set the OneAPI environment variables:
source /opt/intel/oneapi/setvars.sh
Use
--device xpupython3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpu
Vicuna-7B can run on an Intel Arc A770 16GB.
Install the Ascend PyTorch Adapter. Set the CANN environment variables:
source /usr/local/Ascend/ascend-toolkit/set_env.sh
Use
--device npupython3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu
Vicuna-7B/13B can run on an Ascend NPU.
If you do not have enough memory, you can enable 8-bit compression by adding
--load-8bitVicuna-13B with 8-bit compression can run on a single GPU with 16 GB of VRAM, like an Nvidia RTX 3090, RTX 4080, T4, V100 (16GB), or an AMD RX 6800 XT.
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit
In addition to that, you can add
--cpu-offloadingFor Chinese users, you can use models from www.modelscope.cn via specify the following environment variables.
export FASTCHAT_USE_MODELSCOPE=True
To serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. You can learn more about the architecture here.
Here are the commands to follow in your terminal:
python3 -m fastchat.serve.controller
This controller manages the distributed workers.
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller .
To ensure that your model worker is connected to your controller properly, send a test message using the following command:
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5
You will see a short output.
python3 -m fastchat.serve.gradio_web_server
This is the user interface that users will interact with.
By following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now. If the models do not show up, try to reboot the gradio web server.
Currently, Chatbot Arena is powered by FastChat. Here is how you can launch an instance of Chatbot Arena locally.
FastChat supports popular API-based models such as OpenAI, Anthropic, Gemini, Mistral and more. To add a custom API, please refer to the model support doc. Below we take OpenAI models as an example.
Create a JSON configuration file
api_endpoint.json{ "gpt-4o-2024-05-13": { "model_name": "gpt-4o-2024-05-13", "api_base": "https://api.openai.com/v1", "api_type": "openai", "api_key": [Insert API Key], "anony_only": false } }
For Anthropic models, specify
"api_type": "anthropic_message""api_type": "gemini"To serve your own model using local gpus, follow the instructions in Serving with Web GUI.
Now you're ready to launch the server:
python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file api_endpoint.json
# worker 0 CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5 --controller http://localhost:21001 --port 31000 --worker http://localhost:31000 # worker 1 CUDA_VISIBLE_DEVICES=1 python3 -m fastchat.serve.model_worker --model-path lmsys/fastchat-t5-3b-v1.0 --controller http://localhost:21001 --port 31001 --worker http://localhost:31001
python3 -m fastchat.serve.gradio_web_server_multi
FastChat provides OpenAI-compatible APIs for its supported models, so you can use FastChat as a local drop-in replacement for OpenAI APIs. The FastChat server is compatible with both openai-python library and cURL commands. The REST API is capable of being executed from Google Colab free tier, as demonstrated in the FastChat_API_GoogleColab.ipynb notebook, available in our repository. See docs/openai_api.md.
See fastchat/serve/huggingface_api.py.
See docs/langchain_integration.
We use MT-bench, a set of challenging multi-turn open-ended questions to evaluate models. To automate the evaluation process, we prompt strong LLMs like GPT-4 to act as judges and assess the quality of the models' responses. See instructions for running MT-bench at fastchat/llm_judge.
MT-bench is the new recommended way to benchmark your models. If you are still looking for the old 80 questions used in the vicuna blog post, please go to vicuna-blog-eval.
Vicuna is created by fine-tuning a Llama base model using approximately 125K user-shared conversations gathered from ShareGPT.com with public APIs. To ensure data quality, we convert the HTML back to markdown and filter out some inappropriate or low-quality samples. Additionally, we divide lengthy conversations into smaller segments that fit the model's maximum context length. For detailed instructions to clean the ShareGPT data, check out here.
We will not release the ShareGPT dataset. If you would like to try the fine-tuning code, you can run it with some dummy conversations in dummy_conversation.json. You can follow the same format and plug in your own data.
Our code is based on Stanford Alpaca with additional support for multi-turn conversations. We use similar hyperparameters as the Stanford Alpaca.
| Hyperparameter | Global Batch Size | Learning rate | Epochs | Max length | Weight decay |
|---|---|---|---|---|---|
| Vicuna-13B | 128 | 2e-5 | 3 | 2048 | 0 |
pip3 install -e ".[train]"
--model_name_or_path--data_pathtorchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \ --model_name_or_path meta-llama/Llama-2-7b-hf \ --data_path data/dummy_conversation.json \ --bf16 True \ --output_dir output_vicuna \ --num_train_epochs 3 \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 2 \ --gradient_accumulation_steps 16 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 1200 \ --save_total_limit 10 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --fsdp "full_shard auto_wrap" \ --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \ --tf32 True \ --model_max_length 2048 \ --gradient_checkpointing True \ --lazy_preprocess True
Tips:
fastchat/train/train_mem.pyreport_to--report_to wandbMore instructions to train other models (e.g., FastChat-T5) and use LoRA are in docs/training.md.
SkyPilot is a framework built by UC Berkeley for easily and cost effectively running ML workloads on any cloud (AWS, GCP, Azure, Lambda, etc.). Find SkyPilot documentation here on using managed spot instances to train Vicuna and save on your cloud costs.
The code (training, serving, and evaluation) in this repository is mostly developed for or derived from the paper below. Please cite it if you find the repository helpful.
@misc{zheng2023judging, title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena}, author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica}, year={2023}, eprint={2306.05685}, archivePrefix={arXiv}, primaryClass={cs.CL} }
We are also planning to add more of our research to this repository.