Skip to content

WIP - Solr enhancements#2246

Closed
tfmorris wants to merge 102 commits intointernetarchive:masterfrom
tfmorris:solr-builder-performance
Closed

WIP - Solr enhancements#2246
tfmorris wants to merge 102 commits intointernetarchive:masterfrom
tfmorris:solr-builder-performance

Conversation

@tfmorris
Copy link
Copy Markdown
Contributor

@tfmorris tfmorris commented Jul 30, 2019 •
edited
Loading

Description

Technical

This is still a work in progress, but creating a PR for the branch to give it some visibility for review. It is based on @cdrini's PR #1843 which provided the framework.

Things that have been added include:

  • Updated schema for the temp database
  • performance improvements
  • Solr 8.1 support using the official Solr Docker image, updating the schema as necessary (minor)
  • Support for ICUFoldingFilter for author names and subject titles (can be expanded) to normalization plus case and diacritic folding so that Hervé and Herve get indexed the same (as well as more complicated cases)
  • Blacklisting of all fake subjects as candidates for authors' "Top Subjects" list (can be extended to anyplace these subjects are indexed, but this should give us a big usability boost to start with)

Things remaining to be done:

  • Fix delete by query performance issues (~50% time savings?)
  • Fix subject indexing (plus clean up some outstanding bugs like merging subjects that differ only in punctuation Design librarian process & UI for merging duplicate Subjects #65 )
  • Integration of all pieces to be able to download a dump and build an index from scratch in under 12 hours (current time is probably 3x that)

Testing

Evidence

cdrini added 30 commits July 22, 2019 11:58
@cdrini
[solr-builder] create flow for creating a new solr instance from dump
84f3659
@cdrini
[solr-builder] add sql for partial dump from prod
eba69d2
@cdrini
[solr-builder] TMP: fix partial dump query
22dc14e
@cdrini
[solr-builder] TMP: fix partial dump query 2
845f3f6
@cdrini
[solr-builder] m: using set trace instead of manually printing cmds
bffefe2
@cdrini
[solr-builder] update partial dump logic/docs
d14d3a2
@cdrini
[solr-builder] update step 1 docs + add subject placeholder
56ef318
@cdrini
[solr-builder] m: add solr indexing time records (slightly outdated)
4f389ec
@cdrini
[solr-builder] solr-backup for subjects import
b72f75d
@cdrini
[solr-builder] subjects import docs
c4780b4
@cdrini
[solr-builder] fix multiple postgres import at shard boundary
3aa5884
@cdrini
[solr-builder] remove some unused volumes
7f2ad92
@cdrini
[solr-builder] solrconfig import clean default to false
b7a62e6
@cdrini
[solr-builder] add Cython for update_work
4b30ee1
@cdrini
[solr-builder] cython solr_builder script
c758b89
@cdrini
[solr-builder] update gitignores
f3944ef
@cdrini
[solr-builder] update docs w timings
466f32e
@cdrini
[solr-builder] remove partial sync from README
083645b
@cdrini
[solr-builder] move files around + more cython
997a8d2
@cdrini
[solr-builder] move bash in README to bash files
4f799ec
@cdrini
[solr-builder] create separate solrconfigs for dev/prod env
ff581a8
@cdrini
[solr-builder] fix index- bash scripts
20f4000
@cdrini
[solr-builder] fix cython files getting overwritten
17bb04b
@cdrini
[solr-builder] fix solr-backup script
ab39e46
@cdrini
[solr-builder] make subjects import config point to OJF
cf86667
@cdrini
[solr-builder] fix docker compose networks for solr/solr-backup
2fef20b
@cdrini
[solr-builder] solrbuilder log file opt
2cc446c
@cdrini
[solr-builder] add missing db network
6c183b3
@cdrini
[solr-builder] perform only one solr commit for works reindex
c2cfe1e
@cdrini
[solr-builder] update README
0b581b3
@cdrini cdrini mentioned this pull request Sep 9, 2019
Closed
3 tasks
This was referenced Nov 12, 2019
Closed
Open
Closed
Closed
This was referenced Nov 19, 2019
Closed
Closed
Closed
Closed
@cdrini cdrini mentioned this pull request Nov 25, 2019
Closed
@xayhewalo xayhewalo mentioned this pull request Nov 25, 2019
Closed
@tfmorris
Fix typo in author work search. Improve subject name mapping.
a78e4b0
@tfmorris
Fix missing comma. Add to subject blacklist.
018ede0 
Repair missing comma that was merging two entries on the author subject
blacklist. Add "Large type books" to the blacklist (not informative for
an author's "top subjects")
@tfmorris
Update to latest dump
35fec05
This was referenced Mar 4, 2020
Merged
Closed
@cdrini cdrini mentioned this pull request Apr 6, 2020
Closed
31 tasks
@tfmorris tfmorris marked this pull request as draft April 30, 2020 22:40
@cdrini cdrini mentioned this pull request Dec 15, 2020
Merged
@tfmorris
Copy link
Copy Markdown
Contributor Author

tfmorris commented Feb 9, 2023

Abandoned years ago (although some pieces were apparently used in other PRs like #4283)

@tfmorris tfmorris closed this Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

@cdrini cdrini

Labels

Module: Solr Issues related to the configuration or use of the Solr subsystem. [managed] Needs: Review This issue/PR needs to be reviewed in order to be closed or merged (see comments). [managed] State: Work In Progress This issue is being actively worked on. [managed]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tfmorris @cdrini