factors compression / decompression

[ookey]

Hello,
running your code on scene coffee_martini, I get an error (see below).
My superficial investigation leads me to thinks there's a problem with "factors" parameter compression.

Before compression I've got:
factors: torch.sum(params[:,0]>0)= tensor(3588, device='cuda:0')

After decompression: factors: torch.sum(params[:,0]>0)= tensor(0, device='cuda:0')

I launch the test with ./examples/benchmarks/multigop_gifstream.sh,

SCENE_DIR="<my home>/datasets/Neural3D/"
# Set the directory to store results
RESULT_DIR="<my home>/output/coffee_martini/r2/GIFStream"
# Set the rendering trajectory path
RENDER_TRAJ_PATH="ellipse"
# List of scenes to process
SCENE_LIST="coffee_martini"
# List of entropy lambda values (rate-distortion tradeoff parameter)
ENTROPY_LAMBDA_LIST=(0.0005 0.001 0.002 0.004)
# Data factor for training
DATA_FACTOR=2
# Number of frames per GOP (Group of Pictures)
GOP=60
# The index of the first frame to process
FIRST_FRAME=0
# Total number of frames to process
TOTAL_FRAME=300

Do you have any idea where the problem comes from? Did I do something wrong?

Thanks by advance,
Matthieu Gendrin, Orange Innovation.

PS : the error message I get

runpy.py 196 _run_module_as_main
return _run_code(code, main_globals, None,

runpy.py 86 _run_code
exec(code, run_globals)

__main__.py 71 <module>
cli.main()

cli.py 501 main
run()

cli.py 351 run_file
runpy.run_path(target, run_name="__main__")

pydevd_runpy.py 310 run_path
return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)

pydevd_runpy.py 127 _run_module_code
_run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)

pydevd_runpy.py 118 _run_code
exec(code, run_globals)

simple_trainer_GIFStream.py 1754 <module>
cli(main, cfg, verbose=True)

distributed.py 360 cli
return _distributed_worker(0, 1, fn=fn, args=args)

distributed.py 295 _distributed_worker
fn(local_rank, world_rank, world_size, args)

simple_trainer_GIFStream.py 1635 main
runner.run_compression(step=step)

_contextlib.py 120 decorate_context
return func(*args, **kwargs)

simple_trainer_GIFStream.py 1487 run_compression
splats_c = self.compression_method.decompress(compress_dir, self.entropy_models, self.device)

gifstream_end2end_compression.py 190 decompress
time_features[choose_idx] = splats["time_features"]

RuntimeError:
shape mismatch: value tensor of shape [3588, 60, 4] cannot be broadcast to indexing result of shape [0, 60, 4]

[Hliii]

Thanks for pointing this out.

It seems that there is something wrong with the decompression after training. I am trying to reproduce the error.

Could you please provide more details about the shape of splats["factors"] before and after compression? Have you changed any other options in multigop_gifstream.sh? And in which GOP does it happen?

[ookey]

Thank you for you answer.

The problem happens on all gops.

I attach the patch corresponding to my modifications.
Be warned that this patch also include the modifications I made to fix the problem by myself (my fix enabled to process to finish, but with a PSNR of 7dB :-/, I must have missed something).

The shape of the factors is not modified in the process.

• before compression:

  self.comp_sim_splats["factors"].shape
  [77753,4]
  

• after decompression:

  splats["factors"].shape
  [77753,4]
  

patch.zip

My understanding of the problem is that, during compression, the mask is processed on the not-quantized values of factors.
At decompression time, the mask is processed on the quantized values, which is different.

An equivalent problem is that anchor_features are given reference given for factors compression / decompression. As above, only the quantized values should be used on compression (because only quantized values are available on decompression side)

Regards,

[Hliii]

The concern you raised is addressed in line 1627 of the file ./examples/simple_trainer_GIFStream.py. The simulate_compression function is responsible for handling all quantization.

I was unable to reproduce the error you reported; my experiments show that everything is working correctly.

---

Potential Cause and Suggested Fix

The error you described may be caused by the entropy encoding. Please ensure that the entropy coding is functioning properly.

You can try replacing the existing write_bytes() and read_bytes() functions in gsplat/compression/stream_helper.py with the following implementation:

def write_bytes(fd, values):
    fd.write(values)

def read_bytes(fd, n):
    return fd.read(n)

This specific implementation has successfully resolved issues on other platforms, and I hope it helps you too.