Modular SpeechLLM implementation for Sept. 2023 submission (SALM)#7634

Merged

zhehuaichen merged 12 commits intoNVIDIA-NeMo:modular_speechllmfrom

zhehuaichen:speechllm_rebase

Oct 9, 2023

Commits on Oct 4, 2023

add initial impl of ModularizedSpeechGPTModel and integration test

Zhehuai Chen
authored and
zhehuaichen
committed
fix typo in the test name (#1)
zhehuaichen
committed
clean a initial version of example config; make sure it works by test (#2)
zhehuaichen
committed
add the test for training_step and fix the code correspondingly (test passed now) (#3)
zhehuaichen
committed
add test for validation_step (#4)
zhehuaichen
committed
mv audio and text emb concat to prepare_llm_input so as to write test to guard the llm input
zhehuaichen
committed
Merge heh and zhehuai's initial version of frozen am+llm (#5)

zhehuaichen
and
stevehuang52
committed
fix a nit init bug broke test (#6)
zhehuaichen
committed
Merge branch 'main_1.20.0' into speechllm
zhehuaichen
committed
Clean up implementation for SALM paper and sync to NEMO v1.20.0 (#18)

zhehuaichen
and
stevehuang52
committed
rename 'ModularizedAudioGPTModel' to 'ModularAudioGPTLoRAModel'; move speechllm stuff under nemo/collections/multimodal/speechllm
zhehuaichen
committed

Commits on Oct 9, 2023

update copyright; remove workspace/scripts and workspace/tools folders since the main branch has LLaMA support
zhehuaichen
committed