DANS - 9707/Second fix for #9677

[qqmyers]

What this PR does / why we need it: In removing extra fields from the main json export, the fix in #9677 also removed them from file details where they need to be. This PR leverages the boolean flag introduced in #9695 to just add the extra fields when called to create the file details json export.

Which issue(s) this PR closes:

Closes #9707

Special notes for your reviewer: discussed in https://iqss.slack.com/archives/C03R1E7T4KA/p1689347480349959

Suggestions on how to test this: Verify that DDI exports in the exports menu have variable level metadata and that the main json export does not.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

coverage: 20.391% (-0.001%) from 20.392% when pulling 77260cd on GlobalDataverseCommunityConsortium:DANS-external_exporters into 0661c3a on IQSS:develop.

[pdurbin]

@qqmyers thanks for the PR. Would you be able to add a test so it stays fixed?

It's probably a matter of calling this...

UtilIT.exportDataset(datasetPersistentId, "ddi", apiToken);

...like we do in DatasetsIT.

[qqmyers]

@pdurbin - I think tests like

dataverse/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java

Line 2290 in 0661c3a

public void testRestrictFileExportDdi() throws IOException {

already cover this. It looks like that didn't run for #9695 but has passed for this PR.

[pdurbin]

@qqmyers if existing tests are already catching this bug why aren't they failing in the develop branch? Why did the regression need to be discovered manually?

The latest API test run for develop shows no failures: https://jenkins.dataverse.org/job/IQSS-dataverse-develop/1406/testReport/edu.harvard.iq.dataverse.api/

[qqmyers]

OK - new test now passes. Not sure what is up with the MoveIT test - that doesn't seem related. Perhaps timing related again? Otherwise, I think this is ready for review.

[pdurbin]

Not sure what is up with the MoveIT test

Should be fixed here:

• #9700 increase wait time for indexing on failing test #9703

[pdurbin]

@qqmyers thanks for the test! I'm just recording here the output I see from the new test on develop where I expect it to fail (then I'll go run the test on your branch):

  <fileDscr ID="f6" URI="http://localhost:8080/api/access/datafile/6">
    <fileTxt>
      <fileName>data.tab</fileName>
      <fileType>text/tab-separated-values</fileType>
    </fileTxt>
  </fileDscr>
  <otherMat ID="f6" URI="http://localhost:8080/api/access/datafile/6" level="datafile">
    <labl>data.tab</labl>
    <txt/>
    <notes level="file" type="DATAVERSE:CONTENTTYPE" subject="Content/MIME Type">text/tab-separated-values</notes>
  </otherMat>
</codeBook>
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.11 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DatasetsIT
edu.harvard.iq.dataverse.api.DatasetsIT.testUnrestrictedFileExportDdi  Time elapsed: 9.881 s  <<< FAILURE!
java.lang.AssertionError: 
XML path codeBook.fileDscr.fileTxt.dimensns.caseQnty doesn't match.
Expected: 3
  Actual: 

	at edu.harvard.iq.dataverse.api.DatasetsIT.testUnrestrictedFileExportDdi(DatasetsIT.java:3258)


Results:

Failures: 
  DatasetsIT.testUnrestrictedFileExportDdi:3258 XML path codeBook.fileDscr.fileTxt.dimensns.caseQnty doesn't match.
Expected: 3
  Actual: 


Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

[pdurbin]

I merged develop in and ran the test locally. Works great. Here are the variables being tested:

  <fileDscr ID="f4" URI="http://localhost:8080/api/access/datafile/4">
    <fileTxt>
      <fileName>data.tab</fileName>
      <dimensns>
        <caseQnty>3</caseQnty>
        <varQnty>3</varQnty>
      </dimensns>
      <fileType>text/tab-separated-values</fileType>
    </fileTxt>
    <notes level="file" type="VDC:UNF" subject="Universal Numeric Fingerprint">UNF:6:pZmxR+HCyezWT94Tp3MPxA==</notes>
  </fileDscr>
  <dataDscr>
    <var ID="v2" name="name" intrvl="discrete">
      <location fileid="f4"/>
      <labl level="variable">name</labl>
      <varFormat type="character"/>
      <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:tuxukW1kJmzVCPiI7MR6vg==</notes>
    </var>
    <var ID="v1" name="pounds" intrvl="discrete">
      <location fileid="f4"/>
      <labl level="variable">pounds</labl>
      <sumStat type="vald">3.0</sumStat>
      <sumStat type="invd">0.0</sumStat>
      <sumStat type="min">17.0</sumStat>
      <sumStat type="stdev">12.288205727444508</sumStat>
      <sumStat type="mode">.</sumStat>
      <sumStat type="medn">21.0</sumStat>
      <sumStat type="max">40.0</sumStat>
      <sumStat type="mean">26.0</sumStat>
      <varFormat type="numeric"/>
      <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:E4VJ3RWU5iW8zy7VC9+g5A==</notes>
    </var>
    <var ID="v3" name="species" intrvl="discrete">
      <location fileid="f4"/>
      <labl level="variable">species</labl>
      <varFormat type="character"/>
      <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:8orejrpTyzm42bgLBZKtsw==</notes>
    </var>
  </dataDscr>

[pdurbin]

I'm seeing variables in the DDI HTML Codebook too:

[pdurbin]

Looks good. Thanks, especially for the tests, @qqmyers!