[R] collect int64, uint32, uint64 as R integer type if not out of bounds

[asfimport]

bit64::integer64 can be awkward to work with in R (one example: #7385). Often in Arrow we get int64 types from compute methods or other translation methods that auto-promote to the largest integer type, but they would fit fine in a 32-bit integer, which is R's native type.

When calling Array__as_vector on an int64, we could first call the minmax function on the array, and if the extrema are within the range of a 32-bit int, return a regular R integer vector. This would add a little bit of ambiguity as to what R type you'll get from an Arrow type, but I wonder if the benefits are worth it since you can't do much with an integer64 in R. (We could also make this optional, similar to ARROW-7657, so you could specify a "strict" mode if you are in a use case where roundtrip fidelity is more important than R usability.)

Likewise, uint32 and uint64 could be kept as integers and prevent the conversion to double that is currently implemented.

Reporter: Neal Richardson / @nealrichardson
Assignee: Neal Richardson / @nealrichardson

Related issues:

• [R] Document mapping of Arrow <-> R types (relates to)
• [C++] Add more general BoundsCheck function that also checks for arbitrary lower limits in integer arrays (is related to)

PRs and other links:

• GitHub Pull Request #7602

_{Note: This issue was originally created as ARROW-9083. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Thoughts @wesm @romainfrancois?

[asfimport]

Wes McKinney / @wesm:
Sounds good to me. We could implement a generalized version of internal::IndexBoundsCheck that checks whether the int64 values fit in the int32 range. Opening a JIRA issue about that

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Note that casting them to smaller width will negate the effect of zero-copy (except for uint32_t if they're all smaller than INT32_MAX).

[asfimport]

Neal Richardson / @nealrichardson:
True, though currently there is no zero-copy at all when pulling data to R. Assuming that there is someday, I could see the value in having some option to control whether to downcast to fit into natural R vector types.

[asfimport]

Wes McKinney / @wesm:
My patch for ARROW-9197 implements the boundschecking you need to do this type narrowing, so this should be doable this week

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 7602
#7602

[asfimport]

Matthew Summersgill / @msummersgill:
Hate to dredge up an issue that was settled last year, but the current behavior is yielding inconsistent result structures depending on the contents of an int64 column in my query. @nealrichardson – Did you ever implement any kind of system option I can set to maintain int64 type regardless of the contents?

If not, would you be open to a PR with that option? I'd imagine something along the lines of checking a user configurable option .Options$arrow.strict.integer64, and if .Options$arrow.strict.integer64==TRUE, skip the step of checking max. In addition, an error would be thrown if this option was enabled and the bit64 package was not enabled.

 

str(collect(SparkR::sql("SELECT bigint('2147483648') AS x;"))$x[[1]])
## integer64 2147483648 
str(collect(SparkR::sql("SELECT bigint('1') AS x;"))$x[[1]])
## int 1

 

[asfimport]

Neal Richardson / @nealrichardson:
Yes, there is an option. See the changelog from where this feature was added: https://arrow.apache.org/docs/r/news/index.html#bug-fixes-and-other-enhancements

It appears that's the only documentation of the feature though, so if you'd like to add a note about it elsewhere (the arrow.Rmd vignette where the type conversion is discussed seems like an appropriate place), please do open a PR/issue.

[asfimport]

Matthew Summersgill / @msummersgill:
Thanks Neal for the prompt response! I opened a pull request to add the comment from the changelog to that vignette (#9634 ) but the place I would normally look first is in the R package reference manual. The "Details" section "data-type" section seems to me like the most sensible place to document this option, would that make sense to you, or is there somewhere else it might fit better?

[asfimport]

Neal Richardson / @nealrichardson:
Feel free to add in more places you think would help. I opened ARROW-11864 and assigned it to you; we can discuss more there or on your PR.