[gemini] gemini support tensor parallelism.#4942

Merged

flybird11111 merged 47 commits intohpcaitech:mainfrom

flybird11111:gemini-tp

Nov 10, 2023

Commits on Nov 9, 2023

[colossalai]fix typo
flybird11111
committed
[inference] Add smmoothquant for llama (#4904)

Xu-Kai
authored and
flybird11111
committed
Update flash_attention_patch.py

Orion-Zheng
authored and
flybird11111
committed
[kernel] support pure fp16 for cpu adam and update gemini optim tests (#4921)

ver217
authored and
flybird11111
committed
[format] applied code formatting on changed files in pull request 4908 (#4918)

authored and
flybird11111
committed
[gemini] support gradient accumulation (#4869)

Baizhou Zhang
authored and
flybird11111
committed
[hotfix] fix torch 2.0 compatibility (#4936)

ver217
authored and
flybird11111
committed
[test] add no master test for low level zero plugin (#4934)

KKZ20
authored and
flybird11111
committed
[format] applied code formatting on changed files in pull request 4820 (#4886)

authored and
flybird11111
committed
[nfc] fix some typo with colossalai/ docs/ etc. (#4920)

digger-yu
authored and
flybird11111
committed
[Refactor] Integrated some lightllm kernels into token-attention (#4946)

authored and
flybird11111
committed
[test] merge old components to test to model zoo (#4945)

ver217
authored and
flybird11111
committed
[inference] add reference and fix some bugs (#4937)

authored and
flybird11111
committed
[Inference]ADD Bench Chatglm2 script (#4963)

CjhHa1
authored and
flybird11111
committed
[Pipeline inference] Combine kvcache with pipeline inference (#4938)

FoolPlayer
authored and
flybird11111
committed
updated c++17 compiler flags (#4983)

kurisusnowdeng
authored and
flybird11111
committed
[Inference] Dynamic Batching Inference, online and offline (#4953)

authored and
flybird11111
committed
[Kernels]Updated Triton kernels into 2.1.0 and adding flash-decoding for llama token attention (#4965)

authored and
flybird11111
committed
fix ColossalEval (#4992)

authored and
flybird11111
committed
[doc]Update doc for colossal-inference (#4989)

authored and
flybird11111
committed
[hotfix] Fix the bug where process groups were not being properly released. (#4940)

littsk
authored and
flybird11111
committed
[hotfix] fix the bug of repeatedly storing param group (#4951)

Baizhou Zhang
authored and
flybird11111
committed
[doc] add supported feature diagram for hybrid parallel plugin (#4996)

ppt0011
authored and
flybird11111
committed
[Pipeline Inference] Merge pp with tp (#4993)

FoolPlayer
authored and
flybird11111
committed
[release] update version (#4995)

ver217
authored and
flybird11111
committed
[gemini] gemini support tp
flybird11111
committed
fix
flybird11111
committed
update checkpointIO
flybird11111
committed
support fused layernorm
flybird11111
committed
update fusedlayernorm
flybird11111
committed
add sequence parallel to gemini
flybird11111
committed
fix
flybird11111
committed
fix comments
flybird11111
committed
fix
flybird11111
committed
fix t5
flybird11111
committed
clear cache
flybird11111
committed
fix
flybird11111
committed
activate ci
flybird11111
committed
activate ci
flybird11111
committed
fix
flybird11111
committed
fix
flybird11111
committed
fix
flybird11111
committed
fix
flybird11111
committed
revert
flybird11111
committed
modify tp gather method
flybird11111
committed
fix test
flybird11111
committed
Merge branch 'main' into gemini-tp
flybird11111
authored