Skip to content

l#173

Merged
jamesthesnake merged 61 commits intojamesthesnake:mostfrom
hpcaitech:main
Sep 28, 2023
Merged

l#173
jamesthesnake merged 61 commits intojamesthesnake:mostfrom
hpcaitech:main

Conversation

@jamesthesnake
Copy link
Copy Markdown
Owner

@jamesthesnake jamesthesnake commented Sep 28, 2023

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

flybird11111 and others added 30 commits September 12, 2023 15:14
@flybird11111
[shardformer] update shardformer readme (#4689)
8844691 
* [shardformer] update shardformer readme

* [shardformer] update shardformer readme

* [shardformer] update shardformer readme

* [shardformer] update shardformer readme

* [shardformer] update shardformer readme
[hotfix] fix typo in hybrid parallel io (#4697)
d8ceeac
@digger-yu @flybird11111
fix some typo with colossalai/device colossalai/tensor/ etc. (#4171)
9c2feb2 
Co-authored-by: flybird11111 <1829166702@qq.com>
[doc] add potential solution for OOM in llama2 example (#4699)
068372a
@flybird11111
[shardformer] fix GPT2DoubleHeadsModel (#4703)
c7d6975
@yuanheng-zhao
[hotfix] Fix import error: colossal.kernel without triton installed (#…
e2c0e7f 
…4722)

* [hotfix] remove triton kernels from kernel init

* revise bloom/llama kernel imports for infer
@flybird11111
[shardformer] to fix whisper test failed due to significant accuracy …
20190b4 
…differences. (#4710)

* [shardformer] fix whisper test failed

* [shardformer] fix whisper test failed

* [shardformer] fix whisper test failed

* [shardformer] fix whisper test failed
@binmakeswell
[doc] fix llama2 code link (#4726)
ce97790 
* [doc] fix llama2 code link

* [doc] fix llama2 code link

* [doc] fix llama2 code link
[doc] Add user document for Shardformer (#4702)
f911d5b 
* create shardformer doc files

* add docstring for seq-parallel

* update ShardConfig docstring

* add links to llama example

* add outdated massage

* finish introduction & supporting information

* finish 'how shardformer works'

* finish shardformer.md English doc

* fix doctest fail

* add Chinese document
@github-actions
[format] applied code formatting on changed files in pull request 4726 (
8c2dda7 
#4727)

Co-authored-by: github-actions <github-actions@github.com>
[doc] add shardformer support matrix/update tensor parallel documents (…
50e5602 
…#4728)

* add compatibility matrix for shardformer doc

* update tp doc
@digger-yu @flybird11111
Optimized some syntax errors in the documentation and code under appl…
e4fc57c 
…ications/ (#4127)

Co-authored-by: flybird11111 <1829166702@qq.com>
@flybird11111
[shardformer] update pipeline parallel document (#4725)
4616263 
* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document

* [shardformer] update pipeline parallel document
@ppt0011
[legacy] remove deterministic data loader test
cd4e61d
@FoolPlayer
[shardformer] update seq parallel document (#4730)
6a03c93 
* update doc of seq parallel

* fix typo
@FoolPlayer
[example] add gpt2 HybridParallelPlugin example (#4653)
608cffa 
* add gpt2 HybridParallelPlugin example

* update readme and testci

* update test ci

* fix test_ci bug

* update requirements

* add requirements

* update requirements

* add requirement

* rename file
@ppt0011
Merge pull request #4738 from ppt0011/main
73eb3e8 
[legacy] remove deterministic data loader test
[doc] polish shardformer doc (#4735)
451c346 
* arrange position of chapters

* fix typos in seq parallel doc
@oahzxl
[shardformer] add custom policy in hybrid parallel plugin (#4718)
ac27979 
* add custom policy

* update assert
@flybird11111
[example] llama2 add fine-tune example (#4673)
4c4482f 
* [shardformer] update shardformer readme

[shardformer] update shardformer readme

[shardformer] update shardformer readme

* [shardformer] update llama2/opt finetune example and shardformer update to llama2

* [shardformer] update llama2/opt finetune example and shardformer update to llama2

* [shardformer] update llama2/opt finetune example and shardformer update to llama2

* [shardformer] change dataset

* [shardformer] change dataset

* [shardformer] fix CI

* [shardformer] fix

* [shardformer] fix

* [shardformer] fix

* [shardformer] fix

* [shardformer] fix

[example] update opt example

[example] resolve comments

fix

fix

* [example] llama2 add finetune example

* [example] llama2 add finetune example

* [example] llama2 add finetune example

* [example] llama2 add finetune example

* fix

* update llama2 example

* update llama2 example

* fix

* update llama2 example

* update llama2 example

* update llama2 example

* update llama2 example

* update llama2 example

* update llama2 example

* Update requirements.txt

* update llama2 example

* update llama2 example

* update llama2 example
[doc] explaination of loading large pretrained models (#4741)
d151dca
@oahzxl
[kernel] update triton init #4740 (#4740)
32e7f99
@ver217
[legacy] clean up legacy code (#4743)
b5f9e37 
* [legacy] remove outdated codes of pipeline (#4692)

* [legacy] remove cli of benchmark and update optim (#4690)

* [legacy] remove cli of benchmark and update optim

* [doc] fix cli doc test

* [legacy] fix engine clip grad norm

* [legacy] remove outdated colo tensor (#4694)

* [legacy] remove outdated colo tensor

* [test] fix test import

* [legacy] move outdated zero to legacy (#4696)

* [legacy] clean up utils (#4700)

* [legacy] clean up utils

* [example] update examples

* [legacy] clean up amp

* [legacy] fix amp module

* [legacy] clean up gpc (#4742)

* [legacy] clean up context

* [legacy] clean core, constants and global vars

* [legacy] refactor initialize

* [example] fix examples ci

* [example] fix examples ci

* [legacy] fix tests

* [example] fix gpt example

* [example] fix examples ci

* [devops] fix ci installation

* [example] fix examples ci
@github-actions
[format] applied code formatting on changed files in pull request 4743 (
3c6b831 
#4750)

Co-authored-by: github-actions <github-actions@github.com>
@ver217
[misc] update pre-commit and run all files (#4752)
079bf3c 
* [misc] update pre-commit

* [misc] run pre-commit

* [misc] remove useless configuration files

* [misc] ignore cuda for clang-format
@ppt0011
[doc] explain suitable use case for each plugin
10513f2
@ppt0011
[doc] put individual plugin explanation in front
a04337b
@ppt0011
[doc] add model examples for each plugin
e10d9f0
@ppt0011
[doc] put native colossalai plugins first in description section
4d7537b
@ppt0011
Merge pull request #4757 from ppt0011/main
07c2e3d 
[doc] explain suitable use case for each plugin
Baizhou Zhang and others added 29 commits September 21, 2023 10:42
[bug] fix get_default_parser in examples (#4764)
df66741
@ver217
[doc] clean up outdated docs (#4765)
66f3926 
* [doc] clean up outdated docs

* [doc] fix linking

* [doc] fix linking
[doc] add shardformer doc to sidebar (#4768)
493a5ef
@cwher
[chat]: add lora merge weights config (#4766)
901ab1e 
* feat: modify lora merge weights fn

* feat: add lora merge weights config
@ver217
[lazy] support torch 2.0 (#4763)
3e05c07 
* [lazy] support _like methods and clamp

* [lazy] pass transformers models

* [lazy] fix device move and requires grad

* [lazy] fix requires grad and refactor api

* [lazy] fix requires grad
@littsk
[bug] Fix the version check bug in colossalai run when generating the…
1e0e080 
… cmd. (#4713)

* Fix the version check bug in colossalai run when generating the cmd.

* polish code
@Xu-Kai
[feature] add gptq for inference (#4754)
946ab56 
* [gptq] add gptq kernel (#4416)

* add gptq

* refactor code

* fix tests

* replace auto-gptq

* rname inferance/quant

* refactor test

* add auto-gptq as an option

* reset requirements

* change assert and check auto-gptq

* add import warnings

* change test flash attn version

* remove example

* change requirements of flash_attn

* modify tests

* [skip ci] change requirements-test

* [gptq] faster gptq cuda kernel (#4494)

* [skip ci] add cuda kernels

* add license

* [skip ci] fix max_input_len

* format files & change test size

* [skip ci]

* [gptq] add gptq tensor parallel (#4538)

* add gptq tensor parallel

* add gptq tp

* delete print

* add test gptq check

* add test auto gptq check

* [gptq] combine gptq and kv cache manager (#4706)

* combine gptq and kv cache manager

* add init bits

* delete useless code

* add model path

* delete usless print and update test

* delete usless import

* move option gptq to shard config

* change replace linear to shardformer

* update bloom policy

* delete useless code

* fix import bug and delete uselss code

* change colossalai/gptq to colossalai/quant/gptq

* update import linear for tests

* delete useless code and mv gptq_kernel to kernel directory

* fix triton kernel

* add triton import
@CjhHa1
[inference] chatglm2 infer demo (#4724)
ce7ade3 
* add chatglm2

* add

* gather needed kernels

* fix some bugs

* finish context forward

* finish context stage

* fix

* add

* pause

* add

* fix bugs

* finish chatglm

* fix bug

* change some logic

* fix bugs

* change some logics

* add

* add

* add

* fix

* fix tests

* fix
@ver217
[release] update version (#4775)
4146f1c 
* [release] update version

* [doc] revert versions
@TongLi3701
initial commit: add colossal llama 2 (#4784)
74aa7d9
@chengeharrison @TongLi3701
[feature] ColossalEval: Evaluation Pipeline for LLMs (#4786)
ce77785 
* Add ColossalEval

* Delete evaluate in Chat

---------

Co-authored-by: Xu Yuanchen <yuanchen.xu00@gmail.com>
Co-authored-by: Tong Li <tong.li352711588@gmail.com>
@binmakeswell
[doc] add llama2 domain-specific solution news (#4789)
d512a4d 
* [doc] add llama2 domain-specific solution news
@flybird11111
[fix] fix weekly runing example (#4787)
26cd6d8 
* [fix] fix weekly runing example

* [fix] fix weekly runing example
[doc] polish shardformer doc (#4779)
a2db755 
* fix example format in docstring

* polish shardformer doc
[checkpointio] support unsharded checkpointIO for hybrid parallel (#4774
64a08b2 
)

* support unsharded saving/loading for model

* support optimizer unsharded saving

* update doc

* support unsharded loading for optimizer

* small fix
@TongLi3701
update readme
bd01467
@ver217
[lazy] support from_pretrained (#4801)
4965c0d 
* [lazy] patch from pretrained

* [lazy] fix from pretrained and add tests

* [devops] update ci
@TongLi3701
update
8cbce61
@Desperado-Jia
Merge pull request #4805 from TongLi3701/docs/fix
62b6af1 
[doc] Update TODO in README of Colossal-LLaMA-2
@Chandler-Bing
[hotfix] change llama2 Colossal-LLaMA-2 script filename (#4800)
b6cf0ac 
change filename:
pretraining.py -> trainin.py
there is no file named pretraing.py. wrong writing
@hova88
[misc] add last_epoch in CosineAnnealingWarmupLR (#4778)
a227063
@ver217
[doc] add lazy init docs (#4808)
da15fdb
@littsk
[hotfix] fix norm type error in zero optimizer (#4795)
54b3ad8
@littsk
[hotfix] Correct several erroneous code comments (#4794)
11f1e42
@github-actions
[format] applied code formatting on changed files in pull request 4595 (
fb46d05 
#4602)

Co-authored-by: github-actions <github-actions@github.com>
@TongLi3701
fix format (#4815)
bbbcac2
@flybird11111
[chat] fix gemini strategy (#4698)
be400a0 
* [chat] fix gemini strategy

* [chat] fix gemini strategy

* [chat] fix gemini strategy

* [chat] fix gemini strategy

* g# This is a combination of 2 commits.

[chat] fix gemini strategy

fox

* [chat] fix gemini strategy

update llama2 example

[chat] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* [fix] fix gemini strategy

* fix

* fix

* fix

* fix

* fix

* Update train_prompts.py
@chengeharrison
Update Qwen-7B results (#4821)
1fa8c5e 
Co-authored-by: Xu Yuanchen <yuanchen.xu00@gmail.com>
@binmakeswell
[doc] update slack link (#4823)
822051d
@jamesthesnake jamesthesnake merged commit be3cdef into jamesthesnake:most Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.