Static Vignettes

[JustinMShea]

There are a number of advantages for hosting static vignettes in all packages, namely
stability on build. For this project, static vignettes will solve a few of our issues ( #16 #18 #13 #15 #17 ) related to the many data sources we have. We can create these vignettes with .Rmd files and store them in the inst directory, so others have access to the code locally for reproduction, or for use as a template to modify accordingly.

• For data stored in excel files at the source, this approach gives users the choice to accept the risks associated with downloading updated excel files to their machine, instead of forcing this on package build (we shouldn't do that).

• For users with access to the data subscriptions we have, they can use their credentials to download new data and run accordingly.

• packages used in the vignette, don't have to be defined in the suggests, depends, or imports fields of the DESCRIPTION file. light weight is the right weight

@markvanderloo (author of tinytest we are also using) has a nice quick guide here

[vi-to]

cc: @braverock @jaymon0703 @erolbicero

Hey @JustinMShea, let me premise I'm new to the requested vignette-based paper-specific development flow as opposed to the scripts-based sftwr dev I'm more familiar with. Although IMHO the former adds some overhead, it can be worked on. Please, a bit of patience with the following questions, your technical support will help get through the issue. To help you help me I'll be a little extra verbose.

At the time of writing what R CMD check produces is one vignette (re-)building WARNING. At present, what I think is going on under the hood is: sandbox is .Rbuildignored and, since (some) vignettes need objects constructed from data series we are locally storing in sandbox/data/, vignettes building and rebuilding do not match (the two routines have different roots and "consider differently" other directories). So:

• un.Rbuildignoreing sandbox and
• reverting to %\VignetteEngine{knitr::rmarkdown} in vignettes

solves the problem for me and only returns a workable WARNING on PDFs compression.

Also, as a consequence, sandbox contents end up in check results, but to my knowledge so far no trace of local data sets can be found in the installed package. After all, data sets are not even in the repository people can download (and install) from us. Our local usage so far it's simply a convention, fictionally based on sandbox/data/ in order to have a common path. Hence I think we are not (and will not) be redisseminating any sourced/proprietary data file, but be able to agilely work with disclosed (unmodified) series and share rendered vignettes. Because this is a delicate matter it's very important to make sure, can you please double check?
Of course, the folder where each of us is locally storing these data sets needs not to be sandbox/data/. This folder was used because it seemed to be the practice in our repo at the time. In fact, since sandbox it's commonly used to store other files too – and provided we do not clash with R building reserved directories – should we use another directory for this specific purpose?

Now, a useR can retrieve installed package docs with ?RShowDoc. To my knowledge, doc/ is written by standard R build mechanisms and includes all compiled vignettes' *.Rmd, *.R, and *.pdf files. But anyways, the usual download from GitHub will give people the whole source. In practice, to share rendered PDFs just requires us to periodically push them to, say, inst/docs?

On other points:

• Agreed, it shouldn't happen in the final distro. Some files, those from GetFactors(), are currently left just in order to give us a way to check the import is correct. In relation to e.g. Create FamaFrench.Rdata file, documenting accordingly  #10 we can remove them already. However it's worth noting they only produce NOTEs in build/check processes, so may be worth keep our ability to double check until the package will be effectively delivered?
• Shouldn't be one already able to do so through *.Rmds? Unfortunately, I'm not able to provide specific support in this direction because I’m not directly working with databases at this time. I'm working with data sets we are disclosing and providing web references, minimal details, and tools to work with (parsers/). So far all data sets used are freely accessible for everyone. We can give users instructions to where to store the files in order to run vignettes as they are (what we're doing internally), but they would ultimately be in charge of deciding if they would like to download needed files and reproduce results on their machines.
• In my understanding so far, we want to share with users the same code/vignettes we are working with. Without any, sometimes helpful, separation. On our side, in order to deliver rendered vignettes, I think appropriate DESCRIPTION components/fields are needed. Wouldn't they need them too in order to reproduce vignettes?

[JustinMShea]

Hey @vi-to ,you raise many good questions related to this issue, and others in the realm of design choices for R package development.

Fortunately, a few of these answers will reveal themselves as you work through creating static vignettes. I'd get started converting #17, currently in process. Just render the .Rmd file from it's current location, save the pdf in the vignettes folder, and use Mark's guide to write the .Rnw file, which will point the package build process to the PDF, and not build the .Rmd file. Work out bugs that arise in the process, and ping us when stuck.

The entirety of your comments above, will require a longer discussion which we can have in the near future. The Saturday call is an obvious good venue for a more real time dialogue on the topic(s).

[vi-to]

Thank you @JustinMShea for your responsiveness. I've tried the .Rnw based solution, unfortunately without much success so far.

I understand there are circumstances where one doesn't want to build vignettes upon users' install and indeed I think it's the default setting when installing a package from GitHub. I also understand one may want to avoid that the build creates vignettes each and every time. But although I haven't anything against .Rnw based static vignettes per se, even assuming we'll get it working, I am unsure about which specific advantages would they bring in our situation (as opposed to when no underlying *.Rmd exists) and would really like to clarify this aspect to myself.

I think we are dealing with two overlapping problems: one problem is that we of course want to control the usual R CMD build/R CMD check flow for package development purposes, the other is that we would like to distribute our knitted PDFs and keep them as updated as possible in order to show latest results achieved. Since our results are heavily based on external data sources (web-based or not), both the problems must have the sourcing aspect working in order to be dealt with smoothly. This last aspect is what I'm particularly paying attention to and should be addressed by the solution proposed above, I look forward for your review.

Concisely, some feasible solutions I see applying are:

• If our goal is to avoid triggering vignettes building/rebuilding (requiring), couldn't we just set --no-build-vignettes (as our Travis config)?
• If our goal is to redirect a given vignette's rendered *.pdf in a given easily accessible folder, couldn't we simply take advantage of rmarkdown::render(..., output_dir = file.path("our-pdfs-dir")) upon knitting?

To my knowledge, none of these should impair the usage of R CMD build/R CMD check any further, require additional files, limit us from knitting locally and pushing to the desired folder every time we'd like to (indeed, the last setting would make the process even easier for us)?

[JustinMShea]

I don't think Github issues are the best venue for this conversation, given the depth it may require.

For now, lets table this issue and revisit at a team meeting.