Particularly slow cavern directory

[Nat1405]

I have been storing a large volume of Gemini processing results in /cavern/projects/Gemini/reductions. There are roughly 6000 sub-directories in this directory. At time of writing, to list the contents of this directory, it is taking 34 minutes:

time ls /cavern/projects/Gemini/reductions/

......

real    34m51.565s
user    0m0.010s
sys     0m0.006s

[exalate-issue-sync]

Brian Major commented: This should be investigated to see if the metadata retrieval in directories with a large number of entries can be improved.

[pdowler]

Since the major rewrite of the libs, cavern directory listing now streams (fast) and no longer supports pagination on the server side, because that was the bit that didn't scale.

However, the above is using ls so presumably in the platform and right on the filesystem, so that's dependent on the underlying filesystem. Would probably have the same behaviour though.

@brianmajor Performance here is not in the control of vospace code, so I'd like to close (or move this ticket to someplace more appropriate).