Performance regression introduced by #2753

[xvrl]

#2753 introduced a performance regression. See @himanshug's comment here:

https://github.com/druid-io/druid/pull/2753/files/1853f36e9f93376943ea14684e1de85adca174fe#r58920723