Inconsistencies in reported size of files

[Xarthisius]

Search API returns size that's inconsistent with the actual size of file being downloaded. E.g.

$ curl -s https://dataverse.harvard.edu/api/search?q=filePersistentId:JKTORG | jq '.data.items[0].size_in_bytes'
12025

whereas

$ curl -s -I https://dataverse.harvard.edu/api/access/datafile/3040230 | grep Content-Length
Content-Length: 12100

[Xarthisius]

Also md5sum is being reported for original format (csv), whereas url points to a tab file

$ curl -s https://dataverse.harvard.edu/api/access/datafile/3040230 | md5sum
2014a12fd1b6d0dd8da17fff93c66d19  -
$ curl -s https://dataverse.harvard.edu/api/access/datafile/3040230?format=original | md5sum
e7dd2f725941b978d45fed3f33ff640c  -
$ curl -s https://dataverse.harvard.edu/api/search?q=filePersistentId:JKTORG | jq '.data.items[0].md5'
"e7dd2f725941b978d45fed3f33ff640c"

[pdurbin]

@Xarthisius if you want the original file please add "format=original" as a query parameter like this:

https://dataverse.harvard.edu/api/access/datafile/3040230?format=original

See http://guides.dataverse.org/en/4.9.4/api/dataaccess.html#basic-file-access for a list of parameters.

Does this help with the file size inconsistency?

[Xarthisius]

@pdurbin no, it doesn't match either of them, for original it's:

$ curl -s -I 'https://dataverse.harvard.edu/api/access/datafile/3040230?format=original' | grep Content-Length
Content-Length: 11684

[pdurbin]

Bummer. Thanks. 😄

[pameyer]

Not sure if it's a factor, but I could see the S3 chunk/block size as being a factor that could be contributing to this inconsistency. It might be interesting to see if similar behaviour was seen with POSIX storage.

[pdurbin]

@pameyer could be. @Xarthisius do you want to try the same test on https://demo.dataverse.org which doesn't use S3?

[Xarthisius]

Sure!

$ curl -s https://demo.dataverse.org/api/search?q=filePersistentId:8BVIX2 | jq '.data.items[0].size_in_bytes'
36843
$ curl -s -I https://demo.dataverse.org/api/access/datafile/300662 | grep Content-Length
Content-Length: 36920
$ curl -s -I https://demo.dataverse.org/api/access/datafile/300662?format=original | grep Content-Length
Content-Length: 26465

[pameyer]

@Xarthisius thanks! It's not just S3.

[pameyer]

Slightly more investigation into https://demo.dataverse.org/api/access/datafile/300662 ; downloaded tab format and original for comparison:

• Content-Length header on access API matches the size of the files (python3 / jython 2.7.0 rb) for both original and reformatted; suggests that the DB results driving the search (and files section of dataset API) are inaccurate.
• Content-Length and files section of native API match for an image file w\ in the same dataverse on demo; suggesting that the problem is related to ingest.

This makes me think that the size of an ingested file is determined based on whatever unified representation gets used for UNF calculation, rather than storage size - but I don't know enough in that area to know if that's reasonable or not.

The checksum swap seems like a bug to me - but wouldn't be surprised if tabular data folks were more concerned about UNF matches than checksum matches.

[landreev]

@Xarthisius et al:
demo.datafverse.org also uses a Posix file system (it just lives on AWS).
This inconsistency in size should only be the case with files ingested as tabular data.
The size reported by the search API is the actual size in bytes, as saved on the filesystem. Here's the physical file for the file id=300662 from the example above:

# ls -l 10.5072/FK2/N7YHEY/167188a391e-300c2dad66c1
-rw-r--r--. 1 root root 36843 Nov 15 18:03 10.5072/FK2/N7YHEY/167188a391e-300c2dad66c1

The extra bytes you get when you download the file through the API are the variable name header the access api adds to tabular files, on the fly, by default:

# curl -s https://demo.dataverse.org/api/access/datafile/300662 | head -1 
metadataBlock	name	dataverseAlias	displayName	E	F	G	H	I	J	K	L	M	N	O	S

# curl -s https://demo.dataverse.org/api/access/datafile/300662 | head -1  | wc
      1      19      77

hence the total size 36920 bytes

[landreev]

(there was some bad formatting in my last comment - should be fixed now)

[landreev]

@Xarthisius et al:
Here's an annoying discovery:
We used to provide a way to download the tabular file as is, without the variable name header, via
/api/access/datafile/[FILEID]?noVarHeader=true
but - unless I'm missing something, it looks like we've lost that, and the noVarHeader argument is no longer being honored...

[landreev]

(just edited a couple of small things above, for clarity)