7940 stop harvest in progress

[landreev]

What this PR does / why we need it:

This provides a mechanism for a dataverse sysadmin to stop a long-running harvesting job in progress.

Which issue(s) this PR closes:

Closes #7940

Special notes for your reviewer:

Suggestions on how to test this:

will update w/ instructions

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

Coverage increased (+0.02%) to 20.0% when pulling 8e70d99 on 7940-stop-harvest-in-progress into 6b1ffa7 on develop.

[mreekie]

This just has the normal review and QA left.
It's hard to know how to size what's left as it's waiting for review and hasn't been reviewed in detail.
It looks like the review may be straightforward.

[qqmyers]

Looks like it will work overall. I think there's a cut/paste error and I suggested a doc update but otherwise looks good to me. There's also some other cleanup that's not directly related to stopping where I had another question.

[qqmyers]

Typo - 'stop stop' . It was also a little unclear - suggested change: 'Note: If the application server is stopped and restarted, any running harvesting jobs will be killed but may remain marked as in progress in the database. We thus recommend using the mechanism here to stop the harvesting prior to a server restart.'

[landreev]

Thank you, I adopted your version almost verbatim.

[qqmyers]

Not sure I understand - is this a typo? the setHarvestSuccess method is using the RunResultType.INTERRUPTED?

[landreev]

That was of course a typo, thank you.
(I'm working on automated tests right now that would catch this)

[qqmyers]

I can see where the join of dataset and dvobject isn't needed, but why the switch to harvestingclient_id? Was the old code just wrong or is this a typo?

[landreev]

(Sorry for the delay)
The new query is simpler, and correct, all harvested datasets are uniquely tied to their specific harvesting configurations (clients).

The old query must have followed the logic from pre-Dataverse 4 times; back then harvested datasets ("studies") were similarly tied to their "harvesting dataverses". (A harvesting dataverse could only contain harvested studies, all harvested from the same place). That old query is still producing the expected results in our prod., because we still have dedicated collections for each harvested source. Strictly speaking, the dvobject-dataset joint in it was useful, although o.dtype='Dataset' would achieve the same result. But I just remembered that this was no longer required. In Dataverse 4+ it is possible to harvest into a collection that has other content - local datasets and/or datasets harvested from other places. So, shorter answer, yes, that query was wrong.

[kcondon]

Test results:

1. This worked when halting a no set harvest from demo but did not seem to halt a no set harvest from prod. I will retest.
   Set up both a no set prod and no set demo harvest, start both. First halt demo, works, then halt prod did not work. Is leaving the demo halt file in place interfering in some way? I removed it but still continued.
   [Kevin] Update: It appears that the demo harvest was not halted but completed.

2. I am also seeing stack trace errors in the log related to this, perhaps we are pulling the rug out from some process and it gets a null ptr?
   This happens some time around touching the halt file, perhaps after chown and the process detects it?
   ImportServiceBean_err.txt

[landreev]

Not sure if all of these problems are related to my PR.
With the "no set" harvest from demo, I suspect it may just mean that the harvest finished too fast, before the stop file was created (would make sense if the default set on demo is not large enough).
The import error is just that - the client failing to import a record it got from the server; whatever the reason for that was, it would not be related to anything in this PR (it didn't touch any of the actual harvesting/importing/etc. functionality). Some records failing to harvest should not by itself be considered a problem, as long as the harvesting job finishes, and the failure is properly recorded.

The part about the long prod. harvest not stopping with the stop file in place does sound like a real problem. So let's run that again on Monday and I'll try to diagnose what's going on.

[kcondon]

Retesting this am, found it was a user error around the stop file name syntax. Should be stopharvest. but I confusingly named my harvest client stop_harvest_demo so the correct file name given the syntax would be stopharvest.stop_harvest_demo. This worked.

Testing rerun behavior, ie pause use case. It works but needed to rerun it twice, will check again.