LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython. Current focus is on pretraining, in particular reproducing the [GPT-2](https://github.com/openai/gpt-2) and [GPT-3](https://arxiv.org/abs/2005.14165) miniseries, along with a parallel PyTorch reference implementation
Sign in to like and favorite skills
LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython. Current focus is on pretraining, in particular reproducing the GPT-2 and GPT-3 miniseries, along with a parallel PyTorch reference implementation in train_gpt2.py. You'll recognize this file as a slightly tweaked nanoGPT, an earlier project of mine. Currently, llm.c is a bit faster than PyTorch Nightly (by about 7%). In addition to the bleeding edge mainline code in train_gpt2.cu, we have a simple reference CPU fp32 implementation in ~1,000 lines of clean code in one file train_gpt2.c. I'd like this repo to only maintain C and CUDA code. Ports to other languages or repos are very welcome, but should be done in separate repos, and I am happy to link to them below in the "notable forks" section. Developer coordination happens in the Discussions and on Discord, either the
#llmc#llmdotcThe best introduction to the llm.c repo today is reproducing the GPT-2 (124M) model. Discussion #481 steps through this in detail. We can reproduce other models from the GPT-2 and GPT-3 series in both llm.c and in the parallel implementation of PyTorch. Have a look at the scripts README.
debugging tip: when you run the
make-O3-gIf you won't be training on multiple nodes, aren't interested in mixed precision, and are interested in learning CUDA, the fp32 (legacy) files might be of interest to you. These are files that were "checkpointed" early in the history of llm.c and frozen in time. They are simpler, more portable, and possibly easier to understand. Run the 1 GPU, fp32 code like this:
chmod u+x ./dev/download_starter_pack.sh ./dev/download_starter_pack.sh make train_gpt2fp32cu ./train_gpt2fp32cu
The download_starter_pack.sh script is a quick & easy way to get started and it downloads a bunch of .bin files that help get you off the ground. These contain: 1) the GPT-2 124M model saved in fp32, in bfloat16, 2) a "debug state" used in unit testing (a small batch of data, and target activations and gradients), 3) the GPT-2 tokenizer, and 3) the tokenized tinyshakespeare dataset. Alternatively, instead of running the .sh script, you can re-create these artifacts manually as follows:
pip install -r requirements.txt python dev/data/tinyshakespeare.py python train_gpt2.py
The "I am so GPU poor that I don't even have one GPU" section. You can still enjoy seeing llm.c train! But you won't go too far. Just like the fp32 version above, the CPU version is an even earlier checkpoint in the history of llm.c, back when it was just a simple reference implementation in C. For example, instead of training from scratch, you can finetune a GPT-2 small (124M) to output Shakespeare-like text, as an example:
chmod u+x ./dev/download_starter_pack.sh ./dev/download_starter_pack.sh make train_gpt2 OMP_NUM_THREADS=8 ./train_gpt2
If you'd prefer to avoid running the starter pack script, then as mentioned in the previous section you can reproduce the exact same .bin files and artifacts by running
python dev/data/tinyshakespeare.pypython train_gpt2.pyThe above lines (1) download an already tokenized tinyshakespeare dataset and download the GPT-2 (124M) weights, (3) init from them in C and train for 40 steps on tineshakespeare with AdamW (using batch size 4, context length only 64), evaluate validation loss, and sample some text. Honestly, unless you have a beefy CPU (and can crank up the number of OMP threads in the launch command), you're not going to get that far on CPU training LLMs, but it might be a good demo/reference. The output looks like this on my MacBook Pro (Apple Silicon M3 Max):
[GPT-2] max_seq_len: 1024 vocab_size: 50257 num_layers: 12 num_heads: 12 channels: 768 num_parameters: 124439808 train dataset num_batches: 1192 val dataset num_batches: 128 num_activations: 73323776 val loss 5.252026 step 0: train loss 5.356189 (took 1452.121000 ms) step 1: train loss 4.301069 (took 1288.673000 ms) step 2: train loss 4.623322 (took 1369.394000 ms) step 3: train loss 4.600470 (took 1290.761000 ms) ... (trunctated) ... step 39: train loss 3.970751 (took 1323.779000 ms) val loss 4.107781 generating: --- Come Running Away, Greater conquer With the Imperial blood the heaviest host of the gods into this wondrous world beyond. I will not back thee, for how sweet after birth Netflix against repounder, will not flourish against the earlocks of Allay ---
The data files inside
/dev/data/(dataset).pypython dev/data/tinyshakespeare.py
We download and tokenize the tinyshakespeare dataset. The output of this looks like this:
writing 32,768 tokens to ./dev/data/tinyshakespeare/tiny_shakespeare_val.bin writing 305,260 tokens to ./dev/data/tinyshakespeare/tiny_shakespeare_train.bin
The .bin files contain a short header (1024 bytes) and then a stream of tokens in uint16, indicating the token ids with the GPT-2 tokenizer. More datasets are available in
/dev/dataI am also attaching a simple unit test for making sure our C code agrees with the PyTorch code. On the CPU as an example, compile and run with:
make test_gpt2 ./test_gpt2
This now loads the
gpt2_124M_debug_state.bin# fp32 test (cudnn not supported) make test_gpt2cu PRECISION=FP32 && ./test_gpt2cu # mixed precision cudnn test make test_gpt2cu USE_CUDNN=1 && ./test_gpt2cu
This tests both the fp32 path and the mixed precision path. The test should pass and print
overall okay: 1I attached a very small tutorial here, in doc/layernorm/layernorm.md. It's a simple, step-by-step guide to implementing a single layer of the GPT-2 model, the layernorm layer. This is a good starting point to understand how the layers are implemented in C.
flash attention. As of May 1, 2024 we use the Flash Attention from cuDNN. Because cuDNN bloats the compile time from a few seconds to ~minute and this code path is right now very new, this is disabled by default. You can enable it by compiling like this:
make train_gpt2cu USE_CUDNN=1
This will try to compile with cudnn and run it. You have to have cuDNN installed on your system. The cuDNN installation instructions with apt-get will grab the default set of cuDNN packages. For a minimal setup, the cuDNN dev package is sufficient, e.g. on Ubuntu 22.04 for CUDA 12.x:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install libcudnn9-dev-cuda-12
On top of this you need the cuDNN frontend, but this is just header files. Simply clone the repo to your disk. The Makefile currently looks for it in either your home directory or the current directory. If you have put it elsewhere, add
CUDNN_FRONTEND_PATH=/path/to/your/cudnn-frontend/includemakeMake sure you install MPI and NCCL, e.g. on Linux:
sudo apt install openmpi-bin openmpi-doc libopenmpi-dev
For NCCL follow the instructions from the official website (e.g. network installer)
and then:
make train_gpt2cu mpirun -np <number of GPUs> ./train_gpt2cu
or simply run one of our scripts under
./scripts/Make sure you've installed
NCCLThere are 3 ways we currently support that allow you to run multi-node training:
./scripts/multi_node/run_gpt2_124M_mpi.sh./scripts/multi_node/run_gpt2_124M_fs.sbatch./scripts/multi_node/run_gpt2_124M_tcp.sbatchNote:
slurm-wlmsrun --mpi=listpmixmpirunNone of these 3 methods is superior, we just offer you options so that you can run in your specific environment.
Just as an example process to sweep learning rates on a machine with 4 GPUs on TinyStories. Run a shell script
sweep.shchmod u+x sweep.sh#!/bin/bash learning_rates=(3e-5 1e-4 3e-4 1e-3) for i in {0..3}; do export CUDA_VISIBLE_DEVICES=$i screen -dmS "tr$i" bash -c "./train_gpt2cu -i data/TinyStories -v 250 -s 250 -g 144 -l ${learning_rates[$i]} -o stories$i.log" done # you can bring these down with # screen -ls | grep -E "tr[0-3]" | cut -d. -f1 | xargs -I {} screen -X -S {} quit
This example opens up 4 screen sessions and runs the four commands with different LRs. This writes the log files
stories$i.logA few more words on what I want this repo to be:
First, I want
llm.cdev/cudaThat said, I also want
llm.cHowever, as a constraint, I want to keep the mainline
llm.cdev/cudaLastly, I will be a lot more sensitive to complexity in the root folder of the project, which contains the main / default files of the project. In comparison, the
dev/AMD support
C#
CUDA C++
C++/CUDA
WebGPU C++
C++
Go
Java
Metal
Mojo
OpenCL
Rust
Swift
Zig
Habana Gaudi2
Nim
Ways of organizing development:
#llmcMIT