Full re-index of solr data on prod

[cdrini]

This will be an important step into having a more reliable solr environment. Being able to locally create an identical solr environment will get rid of a lot of confusion. It would also allow us a path to move forward on #178 and #599 , since we can spin up a new solr, re-index it with the new settings, and then swap it with the old solr without any downtime.

Subtasks

• Solr Docker image #1055 Create docker image for solr
• Determine data on production solr
  • Why are there type: subject? This looks like it's used for /search/subjects, so these needed to be included.
  • Why are there type: edition? This looks like residuals of dead code for /search/editions (which does appear to work for the measly ~3.5K editions stored in solr)
  • Why isn't there any stats related data? /solr/process_stats.py looks like dead code.
  • Ensure dev's config file is the same as prod's. -> Copied from prod into solrbuilder, so they will be identical.
• Create test solr on server.openjournal.foundation Reindex documents into a new Solr on OJF #2222
• Create Docker-based solr for production use
  • Create solr environment on prod somewhere
  • Pause both solrupdaters
  • Copy OJF solr data to new prod environment
  • Link production to new solr endpoint
  • Destroy old solr endpoint

Notes/Comments

• I believe solr is storing viewage statistics as well as just works/authors themselves
  • @mekarpeles Can you run this query on production solr: NOT(type:work) AND NOT(type:author)?

[cdrini]

These are the other types on production solr. Need to investigate why they're there/if they should be there. Also need to investigate why there wasn't any stats data there (as I previously thought).

type: subject (1510685)
Sample:

<doc>
  <str name="key">/subjects/org:conseil_national_économique_(france)</str>
  <str name="name">Conseil national économique (France)</str>
  <str name="subject_type">org</str>
  <arr name="text">
    <str>Conseil national économique (France)</str>
    <str>/subjects/org:conseil_national_économique_(france)</str>
  </arr>
  <str name="type">subject</str>
  <int name="work_count">1</int>
</doc>

type: edition (3419)
Sample:

<doc>
  <arr name="author_key">
    <str>OL6941607A</str>
  </arr>
  <arr name="author_name">
    <str>Carlos Arturo Jiménez</str>
  </arr>
  <bool name="has_fulltext">false</bool>
  <str name="key">/books/OL25648663M</str>
  <int name="last_modified_i">1419832732</int>
  <arr name="seed">
    <str>/books/OL25648663M</str>
    <str>/works/OL15935579W</str>
    <str>/subjects/politics_and_government</str>
    <str>/subjects/presidents</str>
    <str>/subjects/frente_sandinista_de_liberación_nacional</str>
    <str>/subjects/assassination_attempts</str>
    <str>/subjects/person:daniel_ortega</str>
    <str>/subjects/person:carlos_arturo_jiménez</str>
    <str>/subjects/place:nicaragua</str>
    <str>/subjects/time:1979-1990</str>
    <str>/authors/OL6941607A</str>
  </arr>
  <arr name="text">
    <str>Nosotros no le decíamos presidente</str>
    <str>Carlos Arturo Jiménez</str>
    <str>/books/OL25648663M</str>
    <str>OL6941607A</str>
  </arr>
  <str name="title">Nosotros no le decíamos presidente</str>
  <str name="title_suggest">Nosotros no le decíamos presidente</str>
  <str name="type">edition</str>
</doc>

[tfmorris]

There are a set of official Docker images for Solr that we may want to consider using:
https://hub.docker.com/_/solr

[cdrini]

Unfortunately none of them support our current version of solr :/