Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
12ee483
Add extra headings, bullet points, and labelling for signposting
thisisnic Oct 22, 2021
ca4d396
Fix typo
thisisnic Oct 22, 2021
a1bc636
Remove duplicated line!
thisisnic Nov 22, 2021
0a8ba1e
Remove references to compression libraries not being on
thisisnic Nov 22, 2021
d7c2cfe
Remove line added back in from previous merge
thisisnic Nov 22, 2021
57765e6
Remove accidentally reintroduced line
thisisnic Nov 22, 2021
331f33b
Add in description of NOT_CRAN env var
thisisnic Nov 29, 2021
74c5ee1
Spacing tweaks
thisisnic Nov 29, 2021
80eb937
Put env vars in table
thisisnic Nov 29, 2021
3881d4d
Add spacing so bullet points work
thisisnic Nov 29, 2021
4dfa450
Remove duplicated content and move developer content to developer sec…
thisisnic Nov 29, 2021
8dbcdf7
Rename file and trim out duplication
thisisnic Nov 29, 2021
e2b0639
Move dependency resolution to dev guide
thisisnic Nov 29, 2021
302c510
Add to pkgdown
thisisnic Nov 29, 2021
f6a6a97
Add diagram
thisisnic Nov 29, 2021
f96d481
Change image size
thisisnic Nov 29, 2021
830d974
Promote heading level of the Build env vars section
thisisnic Nov 29, 2021
bd2c904
Update LIBARROW_MINIMAL description
thisisnic Nov 29, 2021
c646fdf
Add images, rephrase hugely, reorder everything, divide source build …
thisisnic Dec 2, 2021
232c4bf
Tell users that here be dragons (well, unnecessarily complex stuff)
thisisnic Dec 2, 2021
6dfa05c
Link to distro map and move ARROW_USE_PKG_CONFIG further down
thisisnic Dec 2, 2021
17fef94
Snark free solaris
thisisnic Dec 2, 2021
a3bbe34
Whitespace fixes
thisisnic Dec 2, 2021
ef01b51
Update r/vignettes/developers/install_details.Rmd
thisisnic Dec 2, 2021
945e435
Update r/vignettes/developers/install_details.Rmd
thisisnic Dec 2, 2021
bbf8da0
Remove backticks and call libarrow libarrow; add description of when …
thisisnic Dec 2, 2021
8f2b424
Merge branch 'ARROW-13398_install.Rmd' of github.com:thisisnic/arrow …
thisisnic Dec 2, 2021
8de3933
Rearrange so heading match bullets
thisisnic Dec 2, 2021
95f8c35
Put graphics in the right place
thisisnic Dec 2, 2021
e009c02
Add back in "on Linux"
thisisnic Dec 2, 2021
f56f96f
Add in why we care
thisisnic Dec 2, 2021
f54c4c1
typo
thisisnic Dec 2, 2021
c4f07a7
Update r/vignettes/install.Rmd
thisisnic Dec 2, 2021
7b6c332
Add RSPM link
thisisnic Dec 3, 2021
c0481c8
Split conda command onto new line
thisisnic Dec 3, 2021
e6f22dc
Merge branch 'ARROW-13398_install.Rmd' of github.com:thisisnic/arrow …
thisisnic Dec 3, 2021
89dfe69
Add mention of winlibs and diagram
thisisnic Dec 3, 2021
9d7facb
Update r/vignettes/developers/install_details.Rmd
thisisnic Dec 6, 2021
798f9b6
Rearrange so flows better
thisisnic Dec 6, 2021
988e137
Move dependencies to main install doc, grab an extra top-level headin…
thisisnic Dec 7, 2021
4d08f7d
Move images to directories
thisisnic Dec 7, 2021
7fee133
Make image backgrounds white, remove incorrect dependencies info
thisisnic Dec 7, 2021
39ed38d
Add content about downloading dependencies
thisisnic Dec 7, 2021
da109cd
Explicitly mention FALSE
thisisnic Dec 7, 2021
07e2046
Be more explicit about features being disabled
thisisnic Dec 7, 2021
42686cd
Soften recommendation
thisisnic Dec 7, 2021
30a3141
Move dependencies to the top
thisisnic Dec 7, 2021
893825b
A bit of shuffling
jonkeane Dec 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions r/_pkgdown.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ navbar:
href: articles/developers/workflow.html
- text: Debugging
href: articles/developers/debugging.html
- text: Package Installation Details
href: articles/developers/install_details.html
reference:
- title: Multi-file datasets
contents:
Expand Down
124 changes: 124 additions & 0 deletions r/vignettes/developers/install_details.Rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
title: "How the R package is installed - advanced"
---

This document is intended specifically for arrow _developers_ who wish to know
more about these scripts. If you are an arrow _user_ looking for help with
installing arrow, please see [the installation guide](../install.html)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️


The arrow R package requires that Arrow C++ library (also known as libarrow) to
be installed in order to work properly. There are a number of different ways
in which libarrow could be installed:

* as part of the R package installation process
* a system package
* a library you've built yourself outside of the context of installing the R package

Below, we discuss each of these setups in turn.

# Installing libarrow during R package installation

There are a number of scripts that are triggered
when `R CMD INSTALL .` is run and for Arrow users, these should all just work
without configuration and pull in the most complete pieces (e.g. official
binaries that we host). One of the jobs of these scripts is to work out
if libarrow is installed, and if not, install it.

An overview of these scripts is shown below:

* `configure` and `configure.win` - these scripts are triggered during
`R CMD INSTALL .` on non-Windows and Windows platforms, respectively. They
handle finding the libarrow, setting up the build variables necessary, and
writing the package Makevars file that is used to compile the C++ code in the R
package.

* `tools/nixlibs.R` - this script is sometimes called by `configure` on Linux
(or on any non-windows OS with the environment variable
`FORCE_BUNDLED_BUILD=true`) if an existing libarrow installation cannot be found.
This sets up the build process for our bundled builds (which is the default on
linux) and checks for binaries or downloads libarrow from source depending on
dependency availability and build configuration.

* `tools/winlibs.R` - this script is sometimes called by `configure.win` on Windows
when environment variable `ARROW_HOME` is not set. It looks for an existing libarrow
installation, and if it can't find one downloads an appropriate libarrow binary.

* `inst/build_arrow_static.sh` - called by `tools/nixlibs.R` when libarrow
needs to be built. It builds libarrow for a bundled, static build, and
mirrors the steps described in the ["Arrow R Developer Guide" vignette](./setup.html)
This build script is also what is used to generate our prebuilt binaries.

The actions taken by these scripts to resolve dependencies and install the
correct components are described below.

## How the R package finds libarrow

### Windows

The diagram below shows how the R package finds a libarrow installation on Windows.

```{r, echo=FALSE, out.width="70%"}
knitr::include_graphics("./install_diagram_windows.png")
```

### Linux

The diagram below shows how the R package finds a libarrow installation on non-Windows systems.

```{r, echo=FALSE, out.width="70%"}
knitr::include_graphics("./install_nix.png")
```

More information about these steps can be found below.

#### Using pkg-config

When you install the arrow R package on Linux, if no environment variables
relating to the location of an existing libarrow installation have already by
set, the installation code will attempt to find libarrow on
your system using the `pkg-config` command.

This will find either installed system packages or libraries you've built yourself.
In order for `install.packages("arrow")` to work with these system packages,
you'll need to install them before installing the R package.

#### Prebuilt binaries

If libarrow is not found on the system, the R package installation
script will next attempt to download prebuilt libarrow binaries
that match your both your local operating system and arrow R package version.
The libarrow binaries will only be retrieved if you have set the environment variable
`LIBARROW_BINARY` or `NOT_CRAN`.

If found, they will be downloaded and bundled when your R package compiles.
For a list of supported distributions and versions,
see the [arrow-r-nightly](https://github.com/ursa-labs/arrow-r-nightly/blob/master/README.md) project.

#### Building from source

If no libarrow binary is found, it will attempt to build it locally.
First, it will also look to see if you are in a checkout of the `apache/arrow`
git repository and thus have the libarrow source files there.
Otherwise, it builds from the source files included in the package.
Depending on your system, building libarrow from source may be slow. If
libarrow is built from source, `inst/build_arrow_static.sh` is executed.

# Using the R package with libarrow installed as a system package

If you are authorized to install system packages and you're installing a CRAN release,
you may want to use the official Apache Arrow release packages corresponding to
the R package version via software distribution tools such as `apt` or `yum`
(though there are some drawbacks: see the
["Troubleshooting" section in the main installation docs]("../install.html)).
See the [Arrow project installation page](https://arrow.apache.org/install/)
to find pre-compiled binary packages for some common Linux distributions,
including Debian, Ubuntu, and CentOS.

Generally, we do not recommend this method of working with libarrow with the R
package unless you have a specific reason to do so.

# Using the R package with an existing libarrow build

This setup is much more common for arrow developers, who may be needing to make
changes to both the R package and libarrow source code. See
the [developer setup docs](./setup.html) for more information.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added r/vignettes/developers/install_nix.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading