UR-Flex seems to have a positional bias

[IDKiro]

During testing, I found that when the two results are relatively close, UR-Flex tends to assign a higher score to Video A.
During training, did UR-Flex enforce score consistency under input-order swapping (i.e., invariance when swapping A and B)?

[CodeGoat24]

Thank you for the feedback!

In our internal analysis, we observed a mild position bias not only in GPT-5.2 but also in Gemini-3-pro.

During training, we mitigated this issue by balancing the proportion of pairs in which A vs. B is the better candidate, so that the model does not systematically associate a particular position with higher scores.

Nevertheless, this bias cannot be fully eliminated in practice, and it tends to be more noticeable in near-tie cases.

Importantly, our RL experiments indicate that despite the residual position bias, UR-Flex still yields a sufficiently well-calibrated preference distribution at the population level, enabling stable and consistent optimization of the generator.

Thank you again for your feedback. we will continue to investigate and address this issue~