Difficulties in Reproducing CodeFilling

[Ajex777]

Hello, thanks a lot for your excellent work on CodeFilling!
I really appreciate the contribution, but I encountered several difficulties when trying to reproduce the results. Here are some suggestions and questions that may also help other users:

1. Installation instructions
   The installation part in README.md seems inaccurate. Since this repo is built on HEAL, which is further derived from CoAlign, it would be better to explicitly mention that readers may also refer to HEAL for a clearer understanding of the environment setup.

2. Compatibility with HEAL
   CodeFilling is not fully compatible with HEAL. For example, the inference scripts have changed significantly. It would be very helpful if you could provide more detailed guidance on how to properly run inference with CodeFilling.

3. Config files for CodeFilling
   I could not find any dedicated config files for CodeFilling. All the configs in opencood/hypes_yaml seem to be for baselines, and the closest ones appear to be those for HEAL. However, since the codebook introduces many new hyperparameters, I am not sure how to set them properly. Could you please share the recommended config files or examples for CodeFilling?

Effective configuration files are what I need the most, and I believe sharing them will also help CodeFilling gain wider adoption.
Thanks in advance for your time and help!

[Ajex777]

Oh I found the checkpoints in previous issue (#2) , it provides a useful reference.
So I guess, the training phase of CodeFilling includes three parts? That is, single-agent detection training, codebook training (using opencood/tools/codebook_trainer.py?), and fusion network training?

[jeffreychou777]

Oh I found the checkpoints in previous issue (#2) , it provides a useful reference. So I guess, the training phase of CodeFilling includes three parts? That is, single-agent detection training, codebook training (using opencood/tools/codebook_trainer.py?), and fusion network training?

Hi,
I tried to train the codebook part end to end before, but it only worked with multi-scale fusion methods (like those in CoAlign or HEAL). For other fusion methods, such as attention fusion or V2X-ViT, I had to use a two-stage strategy: train the model without the codebook first, and then only train the codebook in the second stage.

[marco-cos]

QuantV2X has provided similar codebook learning process in their released code: https://github.com/ucla-mobility/QuantV2X, you could probably refer to that