Why substract entropy from Q-values? ("min_q_version == 3")

[ikamensh]

Hey,

unlike in the paper, implementation has this part with subtracting the action probabilities from Q:

CQL/d4rl/rlkit/torch/sac/cql.py

Line 253 in d67dbe9

if self.min_q_version == 3:

My guess that the effect would be to have less focus of the loss on a single high-Q action, should policy focus on such. But then we already have temperature parameter. Not sure author will answer, so anybody who knows, I'd appreciate your insights :)

[ikamensh]

It's actually described in Appendix F. Since we approximate distribution ~ softmax(Q(s,a)), but in fact sample from some policy to do so, we do importance sampling - dividing by the probability from the policy.