[R] pull() should compute() not collect()

[asfimport]

Currently pull() returns an R vector, but it's the only dplyr verb other than collect() that returns an R data structure. And there's no other natural way to extract a ChunkedArray from the result of an arrow query.

Reporter: Neal Richardson / @nealrichardson
Assignee: Bryce Mecum / @amoeba

PRs and other links:

• GitHub Pull Request #14330

_{Note: This issue was originally created as ARROW-17439. Please see the migration documentation for further details.}

[asfimport]

SHIMA Tatsuya / @eitsupi:
Note that in dbplyr and dtplyr, pull returns a vector in R (as it should).
https://dbplyr.tidyverse.org/reference/pull.tbl_sql.html

I think this change in behavior makes sense, but may confuse users in its current state without references.

[asfimport]

Neal Richardson / @nealrichardson:
Yes, it is an API change, and initially I wrote pull() to return an R vector to match the dbplyr expectation. But because arrow has a meaningful in-memory vector structure (Array/ChunkedArray), it makes sense to use it; it's just not possible in the same way with a remote SQL database with dbplyr.

[asfimport]

SHIMA Tatsuya / @eitsupi:
Since pull() can have additional arguments, why not add an argument that controls whether it should return an arrow structure or an R vector, like the as_data_frame argument that read_csv_arrow() and others have?

[asfimport]

Bryce Mecum / @amoeba:
I submitted a basic PR in #14330 which needs some eyes and little bit of work, I think.

Re: @eitsupi's comment above:

Since pull() can have additional arguments, why not add an argument that controls whether it should return an arrow structure or an R vector, like the as_data_frame argument that read_csv_arrow() and others have?

I think that might work and would give us a way to avoid making a breaking API change if the default was chosen appropriately. Curious for @nealrichardson's thoughts and others'.

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 14330
#14330

[ablack3]

This change is confusing to me as a user. dplyr::pull always returns a vector currently regardless of the type of datastore being used (data.frame, data.table, dbplyr::remote_con). I agree there should be a way to return a chunked array I'm just not sure it should be pull. Currently pull is type-stable in that it always returns a vector which is nice for users. @hadley, @thisisnic what do you think?

[hadley]

@ablack3 I don't have sufficient context to comment. If pull() returns something that you can (e.g.) call mean() on, I think it's fine.

[thisisnic]

@ablack3 Feel free to open a new ticket outlining the issues this is causing you if you want; happy to discuss in more detail there or help think of reasonable workarounds.

[ablack3]

Thanks @thisisnic. As long as there is an option I can set to keep the current behavior I shouldn't have any issues. I have a more pedagogical question about type stability though.

When we say "knowing only the input types" does that mean you know the exact input types or just some of the input types. In S3 an object can have multiple classes and dplyr tbls use several classes.

For example currently pull returns a vector when the first argument is any tbl.
But after this change pull will return a vector when the first argument is tbl_dbi but not for any tbl.

This is purely a learning opportunity for me and perhaps this question belongs on a forum rather than github.

[thisisnic]

It's an interesting question, but it would definitely better in a forum; it's a bit off-topic here (and more people could probably help you elsewhere).

I don't know the exact answer to your question, but if I was posing it I'd be considering pull.data.frame() (i.e. from dplyr) and pull.ArrowTabular() (i.e. from arrow) as entirely different functions to one another - even though they're both S3 methods of the dplyr::pull() generic, they work with entirely differently classes.