Incorrect computation of FPs in frames with zero GT annotations

[krishnakanthnakka]

Hi @anDoer,

Thank you for adapting the HOTA for keypoint tracking. I would like to bring to your notice about possible error in computing the False positives in images where there is no GT.

In line

PoseTrack21/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py

Line 127 in 35bd703

res['HOTA_FP'][a] += num_tracker_joints

,
the FP are added by variable num_tracker_joints .

However the variable num_tracker_joints is a vector of size (J,) containing number of all valid keypoints in entire video (which is huge value for long-sequence videos) and not just the detections in current frame.

Please see

PoseTrack21/eval/posetrack21/posetrack21/trackeval/datasets/posetrack.py

Line 349 in 35bd703

num_tracker_joints += count_valid_joints(tracker_dets)

In other words, the FP at time instant t is added by incorrect value and I observed HOTA significantly degrades by few frames (near the end of video) with no GT annotation

I think it should be modified to res['HOTA_FP'][a] += len(tracker_ids_t).

Furthermore, same analogy for FNs in the later lines with zero predictions. res['HOTA_FN'][a] += num_gt_joints to be modified to res['HOTA_FN'][a] += len(gt_ids_t ).

Please correct me if I'm wrong

[anDoer]

Hi @krishnakanthnakka
thanks for your detailed report.

I will investigate this issue and provide a patch soon.

[krishnakanthnakka]

Hello @anDoer,

Thanks for the prompt response.

I want to inform you that the frames with no GT annotations labels, they are already filtered at

PoseTrack21/eval/posetrack21/posetrack21/trackeval/datasets/posetrack.py

Line 444 in 35bd703

raw_gt_data_new = self.remove_empty_frames(raw_gt_data, raw_gt_data['is_labeled'])

using islabelled field.

So, for the PoseTrack21, i think it won't enter this loop

PoseTrack21/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py

Line 127 in 35bd703

res['HOTA_FP'][a] += num_tracker_joints

Please investigate this issue. I found this issue when visual tracked outputs and computed HOTA metrics are not consistent on a private sports dataset.