8 hours to index a dataset with thousands of files

[pdurbin]

At standup this morning we talked about a dataset that takes eight hours to index (as part of "index all"). It has 25 thousand files: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/3CTMKP

More context:

• https://iqss.slack.com/archives/C010LA04BCG/p1631734153147700
• https://iqss.slack.com/archives/CVB2SMDFX/p1624455061061400

[PaulBoon]

The same thing happend to me with a dataset having 33,870 files while upgrading from 4.20 to 5.6 with a new Solr version.
Because the logs do not mention any progress while it is being indexed you start indexing 'manually' for the remaining datasets that you can discover with curl http://localhost:8080/api/admin/index/status.
Otherwise the users won't see those other datasets on the GUI.

Maybe we should change the indexing order and end with the datasets that have many files.