This is a collection of code from exercises that could potentially be useful for a DTU course. "CMU-MiniLlama-exercise-original" contains the original code for a CMU exercise to code a small version on LLama 2 and use it for text generation. "MiniLlama-exercise-alex-solution" contains my solution for that exercise. This could be run at an exercise session
"argilla_generate_a_preference_dataset" is a tutorial about using LLM's to synthetically generate a preference dataset. It will probably not be possible to run at an exercise session, but is interesting as a proof of concept - although you would need another inference engine than the free Huggingface account.
"unsloth_fine_tuning_llama_3" is a tutorial about fine-tuning a Llama 3 model with Unsloth, which is the fastest and most memory efficient training library for a single GPU. This is not really feasible without a GPU - and it might be a bit "high-level". Maybe its better to start with a pytorch exercise to gain a more in depth understanding. But it is certainly useful for a project.
"DPO_implementation_EPFL" contains some scripts with my implementation for DPO and MCQA evaluation as a part of a project I made at EPFL. Here the sections about DPO reward implementation is most relevant. I uploaded it as a possible inspiration for future exercises (and to show that it is feasible)
I have also uploaded my requirements file for the environments I used to run the code in this course. Note that I created conda environments with the yml files (that in turn called the text files for pip install). I created a separate environment for fine-tuning with Unsloth since there were some incompatibilities between the triton installaton and torchvision and torchaudio. Alternatively you can just avoid installing torchvision and torchaudio (since they are not necessary). Then everything can run with one environment. But I had already created the other environment with the default pytorch installation commands.
IMPORTANT: Note that there have probably been some changes to Unsloth - so the current requirements file will result in a broken installation (deep pip error). I tried this on another PC, and followed their new procedure (and installed all required packages) - but this still failed. Therefore I recommend installing it on WSL (since this is what we will probably use for Docker anyways). Here the installation process is much more smooth (only pip install of Unslioth is necessary). Note that Linux has always been the primary development environment for Unsloth and its subpackages (such as Triton) - and I have previously experienced issues with the Windows installation.