Clarification about threshold in keypoint HOTA #11
Description
In the distance to similarity computation at
I'm unable to comprehend the reason to clip the similarity to zero if the (head normalized) distance is greater than 0.5.
This has consequences in (incorrectly) tagging each detection as TP/TN/FP/FN for each alpha ranging from 0.05 to 0.45 at
In MOTA calculation, the threshold alpha is fixed at 0.5 i.e., PCKh@0.5. Please refer to
https://github.com/leonid-pishchulin/poseval/blob/4258a1575b9f2ddd0bdb85f74557235ab5df0f52/poseval/evaluatePCKh.py#L56.
However, when we try to estimate HOTA at different confidences (0.05 to 0.99), the clipping of similarity has effect of completely neglecting all the PCKh@0.05, @0.1@0.15, @0.2, @0.25 @0.3 @0.35 @0.4 aand 0.45.
Futher it additionally calculates PCKh@0.52.5, @0575, @0.625, @0.675 @0.725, @0.775 @0.825 @0.875 @0.925 @0.975 to the original PCKh@0.5, @055, @0.6, @0.65 @0.7, @0.75 @0.8 @0.85 @0.9 @0.95
To connect the things into perspective, pascal VOC computes mAP@0.5 whereas COCO computes the mAP@0.05:0.99:0.05. MOTA calculates PCKh@0.5 and hota calculates at all levels @0.05:0.99:0.05.
In short, the distance function may needs to be revisited. I think sim = np.maximum(-1 * distance_mtx + 1, 0) makes it consistent with HOTA metrics based on IoU, MOTA at PCKh@0.5.
Please let me know your thoughts and did i missed something?