[R] Pass various wrapped Arrow objects created in Python into R with zero copy via reticulate

[asfimport]

A user may wish to use some functionality available only in pyarrow using reticulate; it would be useful to be able to construct an R wrapper object to the C++ object inside the corresponding Python type, e.g. pyarrow.Table.

This probably will require some new functions to return the memory address of the shared_ptr/unique_ptr inside the Cython types so that a function on the R side can copy the smart pointer and create the corresponding R wrapper type

cc @pitrou

Reporter: Wes McKinney / @wesm
Assignee: Neal Richardson / @nealrichardson

Related issues:

• [C++][Python][R] C++ implementation of C data interface (relates to)

_{Note: This issue was originally created as ARROW-3750. Please see the migration documentation for further details.}

[asfimport]

Romain Francois / @romainfrancois:
Small example to get started: 

library(reticulate)
pa <- import("pyarrow")
i32 <- pa$int32()
.Internal(inspect(i32))
#> @7fecb9f6caa8 04 ENVSXP g0c0 [OBJ,NAM(3),ATT] <0x7fecb9f6caa8>
#> FRAME:
#> @7fecb9f6c958 02 LISTSXP g0c0 [] 
#> TAG: @7fecae9a1b08 01 SYMSXP g1c0 [MARK] "convert"
#> @7fecb9f538b0 10 LGLSXP g0c1 [] (len=1, tl=0) 1
#> TAG: @7fecba1a1f00 01 SYMSXP g0c0 [] "pyobj"
#> @7fecb9f6ca00 22 EXTPTRSXP g0c0 [] 
#> ENCLOS:
#> @7fecad01e2a8 04 ENVSXP g1c0 [MARK,NAM(3)] <R_EmptyEnv>
#> ATTRIB:
#> @7fecb9f711f0 02 LISTSXP g0c0 [] 
#> TAG: @7fecad01dfd0 01 SYMSXP g1c0 [MARK,NAM(3),LCK,gp=0x6000] "class" (has value)
#> @7fecb9f05ec8 16 STRSXP g0c2 [NAM(3)] (len=2, tl=0)
#> @7fecb9eeb458 09 CHARSXP g0c3 [gp=0x60,ATT] [ASCII] [cached] "pyarrow.lib.DataType"
#> @7fecaeb87cb8 09 CHARSXP g1c3 [MARK,gp=0x60] [ASCII] [cached] "python.builtin.object"
get("pyobj", as.environment(i32))
#> <pointer: 0x1157fae40>

 

Then pyobj is an external pointer to some pyObject I suppose, we would need to figure out how to go from this to the underlying shared_ptr or unique_ptr

 

[asfimport]

Wes McKinney / @wesm:
Methods would need to be added to the Cython extension types to give the memory address of the smart pointer object they contain

[asfimport]

Wes McKinney / @wesm:
@pitrou seems like this is a potential case for the C API

[asfimport]

Neal Richardson / @nealrichardson:
See also ARROW-5956 for possibly relevant discussion

[asfimport]

Neal Richardson / @nealrichardson:
pitrou#5 is our proof-of-concept using the C API. Once the protocol is approved we can move ahead with it.