Download: Cannot download tab file but can download original and rdata version on develop

[kcondon]

A regression has snuck into develop branch: cannot download the tab version of a tabular file, 50by1000.dta but can download the original and rdata version. Tested on 2 different machines.

Note that the tabular file is present on disk.
Also note that if I hack the url to look this way, instead of the the one displayed below, it works:

/api/access/datafile/298

There is no log error on download but the browser shows this:

File ingest logs this:

/api/access/datafile/298?format=tab&gbrecs=true

{"status":"ERROR","code":404,"message":"'/api/v1/access/datafile/298' endpoint does not exist on this server. Please check your code for typos, or consult our API guide at http://guides.dataverse.org."}

[2019-10-16T12:29:34.090-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=149 _ThreadName=jk-connector(4)] [timeMillis: 1571243374090] [levelValue: 800] [[
  Ingest of 1 tabular data file(s) is in progress.]]

[2019-10-16T12:29:34.393-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374393] [levelValue: 800] [[
  ***** DTAFileReader: read() start *****]]

[2019-10-16T12:29:34.394-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374394] [levelValue: 800] [[
  release number=113]]

[2019-10-16T12:29:34.395-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374395] [levelValue: 800] [[
  number of variables(nvar)=50]]

[2019-10-16T12:29:34.447-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374447] [levelValue: 800] [[
  ***** DTAFileReader: read() end *****]]

[2019-10-16T12:29:34.448-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374448] [levelValue: 800] [[
  Tabular data successfully ingested; DataTable with 50 variables produced.]]

[2019-10-16T12:29:34.448-0400] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=93 _ThreadName=p: thread-pool-1; w: 1] [timeMillis: 1571243374448] [levelValue: 800] [[
  Tab-delimited file produced: /tmp/tempTabfile.7683848823370224846.tab]]

[landreev]

Also note that if I hack the url to look this way, instead of the the one displayed below, it works: /api/access/datafile/298

This explains why this hasn't been caught by RestAssured - we only test the download API with the default URI, as above. But the UI apparently adds "format=tab" to it, and that's where it appears to be breaking.

My money is on this having been introduced in #6244, because there were some changes specifically to the code that parses the parameters to the download API endpoint. Should be easy to fix too...

(one possible fix is to just stop using "format=tab" - since is the default format... but we probably want to keep it working, for backward compatibility)

[pdurbin]

I took a quick look at this issue but I'm not sure how to fix it.

It's easy to replicate the problem on the phoenix server. Here's the user experience in two screenshots:

As @kcondon and @landreev point out, the URL in the GUI is (for example) http://phoenix.dataverse.org/api/access/datafile/159?format=tab&gbrecs=true with format=tab in it and it fails.

As I dug in a bit I realized that AccessIT is failing (I'm on 7456dda) and is not being exercised by the job at https://jenkins.dataverse.org that I've been talking about. This is because AccessIT is not included in conf/docker-aio/run-test-suite.sh which is what the Jenkins job calls. I assume this should be added. I'd also be happy go ahead and add AccessIT to phoenix to better reflect that there is at least one API test that failing with the following error:

[ERROR]   AccessIT.testDownloadSingleFile:185 expected:<9> but was:<4>

Here's the line that's failing with the error above, which I assume is related:

assertEquals(tabularSizeNoVarHeader, tabFile1SizeConverted);

Here's some of the output that failing test produces:

origSize: 279 | convertSize: 9 | convertNoHeaderSize: 9

[landreev]

In addition to fixing the download issue above, I did a (very minor) cleanup of the API return messages:

• requesting a thumbnail on an image that's too large now generates the same return code, 404, as when requesting a thumbnail on a text file (as implemented in Don't return file when some derived info is requested. #6244); instead of 503 as before. 503/Service unavailable was not the best code for the condition, since 5XX should really be used when service is not available for more catastrophic reasons (server down, unexpected IO errors, etc.)
• added more informative API messages -
  requesting thumbnail on a text file:

  $ curl http://localhost:8080/api/v1/access/datafile/79583?imageThumb=true
  {"status":"ERROR","code":404,"message":"'/api/v1/access/datafile/79583' datafile access error: requested optional service (image scaling, format conversion, etc.) is not supported on this datafile."}
  

  requesting a thumbnail on an image file above the size limit:

  $ curl http://localhost:8080/api/v1/access/datafile/79584?imageThumb=true
  {"status":"ERROR","code":404,"message":"'/api/v1/access/datafile/79584' datafile access error: requested optional service (image scaling, format conversion, etc.) could not be performed on this datafile."}
  

[landreev]

@pdurbin Yes, I believe it will help to enable the AccessIT test on jenkins, once this branch is merged.

[pdurbin]

Yes, I believe it will help to enable the AccessIT test on jenkins, once this branch is merged.

@landreev ok, can you please add it to the list in conf/docker-aio/run-test-suite.sh? (Or let me know if you'd like me to do it.)

[landreev]

Yes, I believe it will help to enable the AccessIT test on jenkins, once this branch is merged.

@landreev ok, can you please add it to the list in conf/docker-aio/run-test-suite.sh? (Or let me know if you'd like me to do it.)

the above is done - but I'm also adding another simple test to AccessIT; will be done shortly.