Smol-Llama

A library for training small Llama style (and potentially other architecture) models.

Status

This library is a work in progress, and currently it supports training models similar to SmolLM models from Huggingface.

For now, read this medium article to understand this library better.

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
data		data
scaling_laws		scaling_laws
vibe-checks		vibe-checks
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
benchmark.py		benchmark.py
hf_utils.py		hf_utils.py
model.py		model.py
multiple_runs.sh		multiple_runs.sh
requirements.txt		requirements.txt
run_ddp_train-math.py		run_ddp_train-math.py
run_ddp_train-think-fineweb-edu.py		run_ddp_train-think-fineweb-edu.py
run_ddp_train-wiki.py		run_ddp_train-wiki.py
run_ddp_train.py		run_ddp_train.py
test_benchmark.ipynb		test_benchmark.ipynb
think_benchmark.ipynb		think_benchmark.ipynb
think_model.py		think_model.py
tokenize_fineweb-math.py		tokenize_fineweb-math.py
tokenize_fineweb.py		tokenize_fineweb.py
tokenize_wikipedia.py		tokenize_wikipedia.py
train.py		train.py
train_fineweb_test.ipynb		train_fineweb_test.ipynb
train_hindi_wiki.ipynb		train_hindi_wiki.ipynb
train_shakespeare.ipynb		train_shakespeare.ipynb
train_think_shakespeare.ipynb		train_think_shakespeare.ipynb