Training GPU Utilization and Other Kind of Dataset Format

[JaaackHongggg]

Thanks for your great work!

I am wondering what is the average gpu utilization when you train lorat on your device? It seems that you use V100 and 4090 in the paper.

When I train lorat by using your code without any modification, the GPU utilization is only about 60 by using H100. I have tested with different num_train_workers and num_io_threads_per_worker, the GPU utilization is always 60 for almost all the time. I use the cpfs (Cloud Parallel File Storage by aliyun) as the file system, which is designed for high performance computing.

I think that maybe the bottleneck is the disk io, but the cpfs is quick enough. Thus, maybe we should turn to other kinds of dataset format, such as lmdb, webdataset, or parquet, which is designed for high performance computing, instead of reading the image frequently from disk.

Do you have any plan to support other kinds of dataset format? I can give my help, if you can develop the support for other dataset format.

If you have any question, plz feel free to contact me.

Best regards.

[LitingLin]

I have cached metadata such as image paths, bounding boxes, and attributes in a custom key-value database, so most I/O operations for the tracking dataset should involve only random image reads. Furthermore, since the tracker’s siamese_training_pair sampling method is completely random, it is hard to optimize I/O through sequential access, so I think the current implementation is nearly optimal. A regular SSD delivers tens of thousands of IOPS, which is not a bottleneck on my platform.

If the bottleneck is on the syscalls like open() and close() (I am not sure if this is the case on cpfs), you could pack the whole dataset into a .tar archive and then modify the code in trackit.miscellanies.image.io to allow it to read files from the tar archive.
If the bottleneck is on the IOPS of the storage, it might be possible to add logic to the data pipeline to save the sampled siamese_training_pair data into the WebDataset format and then read from the resulting WebDataset tar files during subsequent training. However, this would result in a loss of some flexibility.