From e31869b5ec59a49fe061389547f40b6a138720b9 Mon Sep 17 00:00:00 2001 From: jangorecki Date: Tue, 5 Dec 2023 16:11:23 +0100 Subject: [PATCH] glci rework --- .ci/README.md | 64 +++----- .ci/deploy.sh | 30 ---- .ci/publish.R | 82 +++++++--- .gitlab-ci.yml | 363 ++++++++++++++++++------------------------ R/devel.R | 6 +- _pkgdown.yml | 35 ++-- man/update_dev_pkg.Rd | 15 +- 7 files changed, 271 insertions(+), 324 deletions(-) delete mode 100644 .ci/deploy.sh diff --git a/.ci/README.md b/.ci/README.md index 3f303e34ac..d684a598e3 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -1,72 +1,50 @@ # data.table continuous integration and deployment -On each Pull Request opened in GitHub we run Travis CI and Appveyor to provide prompt feedback about the status of PR. Our main CI pipeline runs on GitLab CI. GitLab repository automatically mirrors our GitHub repository and runs pipeline on `master` branch. It tests more environments and different configurations. It publish variety of artifacts. +On each Pull Request opened in GitHub we run GitHub Actions test jobs to provide prompt feedback about the status of PR. Our main CI pipeline runs on GitLab CI nightly. GitLab repository automatically mirrors our GitHub repository and runs pipeline on `master` branch every night. It tests more environments and different configurations. It publish variety of artifacts. ## Environments ### [GitLab CI](./../.gitlab-ci.yml) Test jobs: -- `test-rel-lin` - `r-release` on Linux, most comprehensive test environment, `-O3 -flto -fno-common -Wunused-result`, extra check for no compilation warnings, includes testing [_with other packages_](./../inst/tests/other.Rraw) -- `test-rel-cran-lin` - `--as-cran` on Linux, `-g0`, extra check for final status of `R CMD check` where we allow one NOTE (_size of tarball_). -- `test-dev-cran-lin` - `r-devel` and `--as-cran` on Linux, `--with-recommended-packages --enable-strict-barrier --disable-long-double`, tests for compilation warnings in pkg install and new NOTEs/Warnings in pkg check, and because it is R-devel it is marked as allow_failure -- `test-rel-vanilla-lin` - `r-release` on Linux, no suggested deps, no OpenMP, `-O0`, tracks memory usage during tests -- `test-310-cran-lin` - R 3.1.0 on Linux -- `test-344-cran-lin` - R 3.4.4 on Linux -- `test-350-cran-lin` - R 3.5.0 on Linux, no `r-recommended` -- `test-rel-win` - `r-release` on Windows -- `test-dev-win` - `r-devel` on Windows -- `test-old-win` - `r-oldrel` on Windows -- `test-rel-osx` - MacOSX build not yet deployed, see [#3326](https://github.com/Rdatatable/data.table/issues/3326) for status +- `test-lin-rel` - `r-release` on Linux, most comprehensive test environment, force all suggests, `-O3 -flto=auto -fno-common -Wunused-result`, test for no compilation warnings. +- `test-lin-rel-vanilla` - `r-release` on Linux, no suggested deps, no zlib, no OpenMP, flags `-g -O0 -fno-openmp`, skip manual and vignettes. +- `test-lin-rel-cran` - `--as-cran` on Linux, strict test for final status of `R CMD check`. +- `test-lin-dev-gcc-strict-cran` - `--as-cran` on Linux, `r-devel` built with `-enable-strict-barrier --disable-long-double`, test for compilation warnings, test for new NOTEs/WARNINGs from `R CMD check`. +- `test-lin-dev-clang-cran` - same as `gcc-strict` job but R built with `clang` and no `--enable-strict-barrier --disable-long-double` flags. +- `test-lin-310-cran` - R 3.1.0 on Linux, stated R dependency version. +- `test-win-rel` - `r-release` on Windows. +- `test-win-dev` - `r-devel` on Windows. +- `test-win-old` - `r-oldrel` on Windows. +- `test-mac-rel` - macOS build not yet available, see [#3326](https://github.com/Rdatatable/data.table/issues/3326) for status + +Tests jobs are allowed to fail, summary and logs of test jobs are later published at _CRAN-like checks_ page, see artifacts below. Artifacts: - [homepage](https://rdatatable.gitlab.io/data.table) - made with [pkgdown](https://github.com/r-lib/pkgdown) - [html manual](https://rdatatable.gitlab.io/data.table/library/data.table/html/00Index.html) - [pdf manual](https://rdatatable.gitlab.io/data.table/web/packages/data.table/data.table.pdf) - [html vignettes](https://rdatatable.gitlab.io/data.table/library/data.table/doc/index.html) -- R packages repository for `data.table` and all _Suggests_ dependencies, url: `https://Rdatatable.gitlab.io/data.table` +- R packages repository for `data.table` and all _Suggests_ dependencies, url: `https://rdatatable.gitlab.io/data.table` - sources - Windows binaries for `r-release`, `r-devel` and `r-oldrel` - [CRAN-like homepage](https://rdatatable.gitlab.io/data.table/web/packages/data.table/index.html) -- [CRAN-like checks results](https://rdatatable.gitlab.io/data.table/web/checks/check_results_data.table.html) - note that all artifacts, including check results page, are being published only when all test jobs successfully pass, thus one will not see an _ERROR_ status there (unless error happened on a job marked as `allow_failure`). -- [docker images](https://gitlab.com/Rdatatable/data.table/container_registry) - copy/paste-able `docker pull` commands can be found at the bottom of our [CRAN-like homepage](https://rdatatable.gitlab.io/data.table/web/packages/data.table/index.html) +- [CRAN-like checks results](https://rdatatable.gitlab.io/data.table/web/checks/check_results_data.table.html) -### [Travis CI](./../.travis.yml) +### [GitHub Actions](./../.github/workflows) -Test jobs: -- `r-release` on Linux, includes code coverage check -- _(might be disabled)_ `r-release` on OSX - -Artifacts: -- R packages repository having `data.table` sources only, url: `https://Rdatatable.github.io/data.table` -- code coverage stats pushed to [codecov.io/gh/Rdatatable/data.table](https://codecov.io/gh/Rdatatable/data.table) +TODO document ### [Appveyor](./../.appveyor.yml) -Test jobs: -- Windows `r-release` -- _(might be disabled)_ Windows `r-devel` - -Artifacts: -- Windows `r-release` binaries accessed only via web UI +TODO document -## Tools +## CI tools ### [`ci.R`](./ci.R) -Base R implemented helper script, [originally proposed to R](https://svn.r-project.org/R/branches/tools4pkgs/src/library/tools/R/packages.R), that ease the process of extracting dependency information from description files, also to mirror packages and their recursive dependencies from CRAN to local CRAN-like directory. It is widely used in our [GitLab CI pipeline](./../.gitlab-ci.yml). +Base R implemented helper script, [originally proposed to base R](https://svn.r-project.org/R/branches/tools4pkgs/src/library/tools/R/packages.R), that ease the process of extracting dependency information from description files, and to mirror packages and their recursive dependencies from CRAN to local CRAN-like directory. It is used in [GitLab CI pipeline](./../.gitlab-ci.yml). ### [`publish.R`](./publish.R) -Base R implemented helper script to orchestrate generation of most artifacts. It is being used only in [_integration_ stage in GitLab CI pipeline](./../.gitlab-ci.yml). - -### [`Dockerfile.in`](./Dockerfile.in) - -Template file to produce `Dockerfile` for, as of now, three docker images. Docker images are being built and published in [_deploy_ stage in GitLab CI pipeline](./../.gitlab-ci.yml). -- `r-base-dev` using `r-release`: publish docker image of `data.table` on R-release -- `r-builder` using `r-release`: publish on R-release and OS dependencies for building Rmarkdown vignettes -- `r-devel`: publish docker image of `data.table` on R-devel built with `--with-recommended-packages --enable-strict-barrier --disable-long-double` - -### [`deploy.sh`](./deploy.sh) - -Script used on Travis CI to publish CRAN-like repository of `data.table` sources. It publishes to `gh-pages` branch in GitHub repository. It depends on a token, which is provided based on `secure` environment variable in [.travis.yml](./../.travis.yml). It has been generated by @jangorecki. +Base R implemented helper script to orchestrate generation of most artifacts and to arrange them nicely. It is being used only in [_integration_ stage in GitLab CI pipeline](./../.gitlab-ci.yml). diff --git a/.ci/deploy.sh b/.ci/deploy.sh deleted file mode 100644 index 6f01ef136f..0000000000 --- a/.ci/deploy.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash - -set -o errexit -o nounset -PKG_REPO=$PWD -PKG_TARBALL=$(ls -1t *.tar.gz | head -n 1) -cd .. - -addToDrat(){ - mkdir drat; cd drat - - ## Set up Repo parameters - git init - git config user.name "addToDrat" - git config user.email "addToDrat@travis.ci" - - ## Get drat repo - git remote add upstream "https://$GH_TOKEN@github.com/Rdatatable/data.table.git" 2>err.txt - git fetch upstream gh-pages 2>err.txt - git checkout gh-pages 2>err.txt - git reset --hard "88000defd316538c37af4c8dc842e73e7953f4e2" 2>err.txt - - Rscript -e "drat::insertPackage('$PKG_REPO/$PKG_TARBALL', \ - repodir = '.', \ - commit='Travis publish data.table: build $TRAVIS_COMMIT', \ - addFiles=TRUE, fields='Revision')" - git push --force upstream gh-pages 2>err.txt - -} - -addToDrat diff --git a/.ci/publish.R b/.ci/publish.R index ec35fe43f3..923b89b5d1 100644 --- a/.ci/publish.R +++ b/.ci/publish.R @@ -91,16 +91,16 @@ package.index <- function(package, lib.loc, repodir="bus/integration/cran") { ) vign = tools::getVignetteInfo(pkg, lib.loc=lib.loc) r_rel_ver = Sys.getenv("R_REL_VERSION") - r_devel_ver = Sys.getenv("R_DEVEL_VERSION") - r_oldrel_ver = Sys.getenv("R_OLDREL_VERSION") - stopifnot(nzchar(r_rel_ver), nzchar(r_devel_ver), nzchar(r_oldrel_ver)) + r_dev_ver = Sys.getenv("R_DEV_VERSION") + r_old_ver = Sys.getenv("R_OLD_VERSION") + stopifnot(nzchar(r_rel_ver), nzchar(r_dev_ver), nzchar(r_old_ver)) cran.home = "../../.." tbl.dl = c( sprintf(" Reference manual: %s.pdf, 00Index.html ", pkg, pkg, cran.home, pkg), if (nrow(vign)) sprintf("Vignettes:%s", paste(sprintf("%s
", cran.home, vign[,"PDF"], vign[,"Title"]), collapse="\n")), # location unline cran web/pkg/vignettes to not duplicate content, documentation is in ../../../library sprintf(" Package source: %s_%s.tar.gz ", cran.home,pkg, version, pkg, version), - sprintf(" Windows binaries: %s ", format.bins(ver=c("r-devel","r-release","r-oldrel"), bin_ver=c(r_devel_ver, r_rel_ver, r_oldrel_ver), cran.home=cran.home, os.type="windows", pkg=pkg, version=version, repodir=repodir)), - sprintf(" macOS binaries: %s ", format.bins(ver=c("r-release","r-oldrel"), bin_ver=c(r_rel_ver, r_oldrel_ver), cran.home=cran.home, os.type="macosx", pkg=pkg, version=version, repodir=repodir)) + sprintf(" Windows binaries: %s ", format.bins(ver=c("r-devel","r-release","r-oldrel"), bin_ver=c(r_dev_ver, r_rel_ver, r_old_ver), cran.home=cran.home, os.type="windows", pkg=pkg, version=version, repodir=repodir)), + sprintf(" macOS binaries: %s ", format.bins(ver=c("r-release","r-oldrel"), bin_ver=c(r_rel_ver, r_old_ver), cran.home=cran.home, os.type="macosx", pkg=pkg, version=version, repodir=repodir)) ) index.file = file.path(repodir, "web/packages", pkg, "index.html") if (!dir.exists(dirname(index.file))) dir.create(dirname(index.file), recursive=TRUE) @@ -155,24 +155,30 @@ plat <- function(x) if (grepl("^.*win", x)) "Windows" else if (grepl("^.*mac", x r.ver <- function(x) { tmp = strsplit(x, "-", fixed=TRUE)[[1L]] - if (length(tmp) < 2L) stop("test job names must be test-[r.version]-...") - v = tmp[2L] + if (length(tmp) < 3L) stop("test job names must be test-[lin|win|mac]-[r.version]-...") + v = tmp[3L] if (identical(v, "rel")) "r-release" else if (identical(v, "dev")) "r-devel" else if (identical(v, "old")) "r-oldrel" else { - if (grepl("\\D", v)) stop("second word in test job name must be rel/dev/old or numbers of R version") + if (grepl("\\D", v)) stop("third word in test job name must be rel/dev/old or numbers of R version") paste0("r-", paste(strsplit(v, "")[[1L]], collapse=".")) } } # this for now is constant but when we move to independent pipelines (commit, daily, weekly) those values can be different pkg.version <- function(job, pkg) { - dcf = read.dcf(file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "DESCRIPTION")) + Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep=".")) + if (!dir.exists(Rcheck)) + return(NA_character_) + dcf = read.dcf(file.path(Rcheck, "00_pkg_src", pkg, "DESCRIPTION")) dcf[,"Version"] } pkg.revision <- function(job, pkg) { - dcf = read.dcf(file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "DESCRIPTION")) + Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep=".")) + if (!dir.exists(Rcheck)) + return(NA_character_) + dcf = read.dcf(file.path(Rcheck, "00_pkg_src", pkg, "DESCRIPTION")) if ("Revision" %in% colnames(dcf)) { proj.url = Sys.getenv("CI_PROJECT_URL", "") if (!nzchar(proj.url)) { @@ -184,7 +190,10 @@ pkg.revision <- function(job, pkg) { } else "" } pkg.flags <- function(job, pkg) { - cc = file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "cc") ## data.table style cc file + Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep=".")) + if (!dir.exists(Rcheck)) + return(NA_character_) + cc = file.path(Rcheck, pkg, "cc") ## data.table style cc file if (file.exists(cc)) { d = readLines(cc) w.cflags = substr(d, 1, 7)=="CFLAGS=" @@ -254,6 +263,34 @@ check.flavors <- function(jobs, repodir="bus/integration/cran") { setNames(file.exists(file), file) } +log.copy <- function(job, repodir="bus/integration/cran") { + dir.create(job.checks<-file.path(repodir, "web", "checks", pkg<-"data.table", job), recursive=TRUE, showWarnings=FALSE) + to = file.path(job.checks, "log") + if (!file.exists(job_id_file <- file.path("bus", job, "id"))) + return(setNames(file.exists(to), "log")) + job_id = readLines(job_id_file, warn=FALSE)[1L] + from = sprintf("https://gitlab.com/Rdatatable/data.table/-/jobs/%s/raw", job_id) + download.file(from, to, method="wget", quiet=TRUE) + Sys.sleep(0.1) ## to not get ban from gitlab.com + setNames(file.exists(to), "log") +} + +ci.status <- function(job) { + if (!file.exists(status_file <- file.path("bus", job, "status"))) + return(NA_character_) + readLines(status_file, warn=FALSE)[1L] +} + +ci.log <- function(jobs, repodir="bus/integration/cran") { + pkg = "data.table" + ans = vector("character", length(jobs)) + logs = sapply(jobs, log.copy, repodir=repodir) + statuses = sapply(jobs, ci.status) + ans[!logs] = statuses[!logs] + ans[logs] = sprintf('%s', pkg[any(logs)], jobs[logs], statuses[logs]) + ans +} + check.index <- function(pkg, jobs, repodir="bus/integration/cran") { status = function(x) if (grepl("^.*ERROR", x)) "ERROR" else if (grepl("^.*WARNING", x)) "WARNING" else if (grepl("^.*NOTE", x)) "NOTE" else if (grepl("^.*OK", x)) "OK" else NA_character_ test.files = function(job, files, trim.name=FALSE, trim.exts=0L, pkg="data.table") { @@ -294,17 +331,18 @@ check.index <- function(pkg, jobs, repodir="bus/integration/cran") { } memouts }) - th = "FlavorVersionRevisionInstallStatusFlagsRout.failMemtest" + th = "FlavorVersionRevisionInstallStatusFlagsRout.failLogMemtest" tbl = sprintf( - "%s%s%sout%s%s%s%s", - sub("test-", "", jobs, fixed=TRUE), - sapply(jobs, pkg.version, pkg), - sapply(jobs, pkg.revision, pkg), - pkg, jobs, ## install - pkg, jobs, sapply(sapply(jobs, check.test, pkg="data.table"), status), ## check - sapply(jobs, pkg.flags, pkg), - mapply(test.files, jobs, routs, trim.exts=2L), # 1st fail, 2nd Rout, keep just: tests_x64/main - mapply(test.files, jobs, memouts, trim.name=TRUE) + "%s%s%sout%s%s%s%s%s", + sub("test-", "", jobs, fixed=TRUE), ## Flavor + sapply(jobs, pkg.version, pkg), ## Version + sapply(jobs, pkg.revision, pkg), ## Revision + pkg, jobs, ## Install + pkg, jobs, sapply(sapply(jobs, check.test, pkg="data.table"), status), ## Status + sapply(jobs, pkg.flags, pkg), ## Flags + mapply(test.files, jobs, routs, trim.exts=2L), ## Rout.fail: 1st fail, 2nd Rout, keep just: tests_x64/main + ci.log(jobs), ## CI job logs + mapply(test.files, jobs, memouts, trim.name=TRUE) ## Memtest // currently not used ) file = file.path(repodir, "web/checks", sprintf("check_results_%s.html", pkg)) writeLines(c( @@ -340,7 +378,7 @@ check.test <- function(job, pkg) { check[length(check)] } -move.bin <- function(job, bin.version, os.type, file="DESCRIPTION", silent=FALSE) { +move.bin <- function(job, bin.version, os.type, file="DESCRIPTION", silent=TRUE) { if (os.type=="unix") { stop("publish of linux binaries not supported") } else if (os.type=="windows") { diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 80fa5d00a7..60cf09bb55 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,9 @@ +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE=="schedule" && $CI_COMMIT_REF_NAME=="master"' ## nightly scheduled pipeline at 4:15 UTC + - if: '$CI_PIPELINE_SOURCE=="web"' ## manually started from web UI + - if: '$CI_PIPELINE_SOURCE=="push" && $CI_COMMIT_REF_NAME!="master"' ## branches pushed to GL directly, mirror is set for master branch only + variables: CRAN_MIRROR: "https://cloud.r-project.org" _R_CHECK_FORCE_SUGGESTS_: "false" @@ -6,9 +12,18 @@ variables: TZ: "UTC" ## to avoid 'Failed to create bus connection' from timedatectl via Sys.timezone() on Docker with R 3.4. ## Setting TZ for all GLCI jobs to isolate them from timezone. We could have a new GLCI job to test under ## a non-UTC timezone, although, that's what we do routinely in dev. - R_REL_VERSION: "4.3" - R_DEVEL_VERSION: "4.4" - R_OLDREL_VERSION: "4.2" + R_REL_VERSION: "4.3" + R_REL_WIN_BIN: "https://cloud.r-project.org/bin/windows/base/old/4.3.2/R-4.3.2-win.exe" + RTOOLS_REL_BIN: "https://cloud.r-project.org/bin/windows/Rtools/rtools43/files/rtools43-5863-5818.exe" + RTOOLS43_HOME: "/c/rtools" + R_DEV_VERSION: "4.4" + R_DEV_WIN_BIN: "https://cloud.r-project.org/bin/windows/base/R-devel-win.exe" + RTOOLS_DEV_BIN: "https://cloud.r-project.org/bin/windows/Rtools/rtools43/files/rtools43-5863-5818.exe" + RTOOLS44_HOME: "" ## in case R-devel will use new Rtools toolchain, now it uses 4.3 env var + R_OLD_VERSION: "4.2" + R_OLD_WIN_BIN: "https://cloud.r-project.org/bin/windows/base/old/4.2.3/R-4.2.3-win.exe" + RTOOLS_OLD_BIN: "https://cloud.r-project.org/bin/windows/Rtools/rtools42/files/rtools42-5355-5357.exe" + RTOOLS42_HOME: "/c/rtools" stages: - dependencies @@ -22,7 +37,7 @@ stages: expire_in: 2 weeks when: always paths: - - bus + - bus/$CI_JOB_NAME ## mirror packages # download all recursive dependencies once to be used across multiple test jobs @@ -40,9 +55,13 @@ mirror-packages: - echo 'source(".ci/ci.R")' >> .Rprofile - mkdir -p bus/$CI_JOB_NAME/cran/src/contrib - Rscript -e 'mirror.packages(dcf.dependencies("DESCRIPTION", "all"), repos=Sys.getenv("CRAN_MIRROR"), repodir="bus/mirror-packages/cran")' - - Rscript -e 'sapply(simplify=FALSE, setNames(nm=Sys.getenv(c("R_REL_VERSION","R_DEVEL_VERSION","R_OLDREL_VERSION"))), function(binary.ver) mirror.packages(type="win.binary", dcf.dependencies("DESCRIPTION", "all"), repos=Sys.getenv("CRAN_MIRROR"), repodir="bus/mirror-packages/cran", binary.ver=binary.ver))' + - Rscript -e 'sapply(simplify=FALSE, setNames(nm=Sys.getenv(c("R_REL_VERSION","R_DEV_VERSION","R_OLD_VERSION"))), function(binary.ver) mirror.packages(type="win.binary", dcf.dependencies("DESCRIPTION", "all"), repos=Sys.getenv("CRAN_MIRROR"), repodir="bus/mirror-packages/cran", binary.ver=binary.ver))' <<: *artifacts +## install deps aliases +.test-install-deps: &install-deps + - Rscript -e 'source(".ci/ci.R"); install.packages(dcf.dependencies("DESCRIPTION", which="all"), repos=file.path("file:", normalizePath("bus/mirror-packages/cran", mustWork=FALSE)), quiet=FALSE)' + ## build # sources as tar.gz archive # build vignettes @@ -50,137 +69,78 @@ build: stage: build tags: - linux - image: registry.gitlab.com/jangorecki/dockerfiles/r-base ## r-base-gcc after rstudio/markdown#108 + image: registry.gitlab.com/jangorecki/dockerfiles/r-base-gcc needs: ["mirror-packages"] before_script: - - Rscript -e 'install.packages(c("knitr","rmarkdown"), repos=file.path("file:",normalizePath("bus/mirror-packages/cran")), quiet=TRUE)' + - *install-deps - rm -r bus + script: - sed -i '/^[[:space:]]*$/d' ./DESCRIPTION ## make last line end abruptly; i.e. without a final \n - echo "Revision:" $CI_COMMIT_SHA >> ./DESCRIPTION - script: - R CMD build . - - mkdir -p bus/$CI_JOB_NAME/cran/src/contrib - - mv $(ls -1t data.table_*.tar.gz | head -n 1) bus/$CI_JOB_NAME/cran/src/contrib/. - - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/build/cran"), fields="Revision", addFiles=TRUE)' + - mkdir -p bus/$CI_JOB_NAME/ + - mv $(ls -1t data.table_*.tar.gz | head -n 1) bus/$CI_JOB_NAME/ <<: *artifacts -## install deps aliases -.test-install-deps: &install-deps - - Rscript -e 'source(".ci/ci.R"); install.packages(dcf.dependencies("DESCRIPTION", which="all"), repos=file.path("file:", normalizePath("bus/mirror-packages/cran", mustWork=FALSE)), quiet=TRUE)' -.test-install-deps-win: &install-deps-win - - Rscript.exe -e "source('.ci/ci.R'); install.packages(dcf.dependencies('DESCRIPTION', which='all'), repos=file.path('file://',getwd(),'bus/mirror-packages/cran'), quiet=TRUE)" - -## copy data.table tar.gz from bus R repo to current directory -.test-cp-src: &cp-src - - cp $(ls -1t bus/build/cran/src/contrib/data.table_*.tar.gz | head -n 1) . -.test-cp-src-win: &cp-src-win - - cp.exe $(ls.exe -1t bus/build/cran/src/contrib/data.table_*.tar.gz | head.exe -n 1) . - -## move data.table tar.gz to bus -.test-mv-src: &mv-src - - mkdir -p bus/$CI_JOB_NAME && mv $(ls -1t data.table_*.tar.gz | head -n 1) bus/$CI_JOB_NAME -.test-mv-src-win: &mv-src-win - - mkdir.exe -p bus/$CI_JOB_NAME; mv.exe $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) bus/$CI_JOB_NAME - -## move data.table binaries to bus R repo -.test-mv-bin-win: &mv-bin-win - - mkdir.exe -p cran/bin/windows/contrib/$R_VERSION; mv.exe $(ls.exe -1t data.table_*.zip | head.exe -n 1) cran/bin/windows/contrib/$R_VERSION - -## remove data.table tar.gz -.test-rm-src: &rm-src - - rm $(ls -1t data.table_*.tar.gz | head -n 1) -.test-rm-src-win: &rm-src-win - - rm.exe $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - -## install R on windows -.test-install-r-rel-win: &install-r-rel-win - - curl.exe -s -o ../R-rel.exe https://cloud.r-project.org/bin/windows/base/old/4.3.2/R-4.3.2-win.exe; Start-Process -FilePath ..\R-rel.exe -ArgumentList "/VERYSILENT /DIR=C:\R" -NoNewWindow -Wait -.test-install-r-devel-win: &install-r-devel-win - - curl.exe -s -o ../R-devel.exe https://cloud.r-project.org/bin/windows/base/R-devel-win.exe; Start-Process -FilePath ..\R-devel.exe -ArgumentList "/VERYSILENT /DIR=C:\R" -NoNewWindow -Wait -.test-install-r-oldrel-win: &install-r-oldrel-win - - curl.exe -s -o ../R-oldrel.exe https://cloud.r-project.org/bin/windows/base/old/4.2.3/R-4.2.3-win.exe; Start-Process -FilePath ..\R-oldrel.exe -ArgumentList "/VERYSILENT /DIR=C:\R" -NoNewWindow -Wait - -## install Rtools on windows -.test-install-rtools42-win: &install-rtools42-win - - curl.exe -s -o ../rtools.exe https://cloud.r-project.org/bin/windows/Rtools/rtools42/files/rtools42-5355-5357.exe; Start-Process -FilePath ..\rtools.exe -ArgumentList "/VERYSILENT /DIR=C:\rtools42" -NoNewWindow -Wait -.test-install-rtools43-win: &install-rtools43-win - - curl.exe -s -o ../rtools.exe https://cloud.r-project.org/bin/windows/Rtools/rtools43/files/rtools43-5863-5818.exe; Start-Process -FilePath ..\rtools.exe -ArgumentList "/VERYSILENT /DIR=C:\rtools43" -NoNewWindow -Wait - .test-template: &test stage: test needs: ["mirror-packages","build"] + allow_failure: true <<: *artifacts .test-lin-template: &test-lin <<: *test tags: - linux - -.test-win-template: &test-win - <<: *test - tags: - - windows - - shared-windows - -#.test-mac-template: &test-mac -# <<: *test -# tags: -# - macosx + before_script: + - cp $(ls -1t bus/build/data.table_*.tar.gz | head -n 1) . + - mkdir -p ~/.R + after_script: + - mkdir -p bus/$CI_JOB_NAME + - echo $CI_JOB_ID > bus/$CI_JOB_NAME/id + - echo $CI_JOB_STATUS > bus/$CI_JOB_NAME/status + - echo $CI_JOB_IMAGE > bus/$CI_JOB_NAME/image + - '[ -d data.table.Rcheck ] && mv data.table.Rcheck bus/$CI_JOB_NAME/' ## most comprehensive tests # force all suggests # flags: gcc -O3 -flto=auto -fno-common -Wunused-result # tests for compilation warnings -test-rel-lin: +test-lin-rel: <<: *test-lin image: registry.gitlab.com/jangorecki/dockerfiles/r-data.table - needs: ["mirror-packages","build"] variables: _R_CHECK_CRAN_INCOMING_: "FALSE" _R_CHECK_CRAN_INCOMING_REMOTE_: "FALSE" _R_CHECK_FORCE_SUGGESTS_: "TRUE" _R_CHECK_TESTS_NLINES_: "0" OPENBLAS_MAIN_FREE: "1" - before_script: + script: - apt-get update -q && apt-get install -y --no-install-recommends libcurl4-openssl-dev ## remove after #5749 - *install-deps - - *cp-src - - rm -r bus - - mkdir -p ~/.R - - echo 'CFLAGS=-g -O3 -flto=auto -fno-common -Wunused-result -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars - - echo 'CXXFLAGS=-g -O3 -flto=auto -fno-common -Wunused-result -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars - script: - - *mv-src - - cd bus/$CI_JOB_NAME + - echo 'CFLAGS=-g -O3 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars + - echo 'CXXFLAGS=-g -O3 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars - R CMD check $(ls -1t data.table_*.tar.gz | head -n 1) - - *rm-src - (! grep "warning:" data.table.Rcheck/00install.out) ## vanilla minimal +# no zlib # no suggested deps # no vignettes or manuals # no openmp # flags: gcc -O0 -fno-openmp -test-rel-vanilla-lin: +test-lin-rel-vanilla: <<: *test-lin image: registry.gitlab.com/jangorecki/dockerfiles/r-base-gcc - allow_failure: true ## temp workaround #5484 - before_script: - - *cp-src - - rm -r bus - - mkdir -p ~/.R + script: - echo 'CFLAGS=-g -O0 -fno-openmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars - echo 'CXXFLAGS=-g -O0 -fno-openmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars - script: - - *mv-src - - cd bus/$CI_JOB_NAME - R CMD check --no-manual --ignore-vignettes $(ls -1t data.table_*.tar.gz | head -n 1) - - *rm-src ## R-release on Linux # strict checks for 0 NOTEs # extra NOTEs check and build pdf manual thus not from cran-lin template -test-rel-cran-lin: +test-lin-rel-cran: <<: *test-lin image: registry.gitlab.com/jangorecki/dockerfiles/r-base variables: @@ -188,147 +148,139 @@ test-rel-cran-lin: _R_CHECK_CRAN_INCOMING_REMOTE_: "FALSE" ## Other than no URL checking (takes many minutes) or 'Days since last update 0' NOTEs needed, #3284 _R_CHECK_CRAN_INCOMING_TARBALL_THRESHOLD_: "7500000" ## bytes _R_CHECK_PKG_SIZES_THRESHOLD_: "7" ## MB 'checking installed package size' NOTE - before_script: - - apt-get update -q && apt-get install -y --no-install-recommends libcurl4-openssl-dev ## temp workaround #5749 + script: + - apt-get -qq update && apt-get install -y --no-install-recommends libcurl4-openssl-dev ## temp workaround for curl dep #5749 - *install-deps - - *cp-src - - rm -r bus - - mkdir -p ~/.R - echo 'CFLAGS=-g -O2 -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars - echo 'CXXFLAGS=-g -O2 -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars - script: - - *mv-src - - cd bus/$CI_JOB_NAME - R CMD check --as-cran $(ls -1t data.table_*.tar.gz | head -n 1) - - *rm-src - >- Rscript -e 'l=tail(readLines("data.table.Rcheck/00check.log"), 1L); if (!identical(l, "Status: OK")) stop("Last line of ", shQuote("00check.log"), " is not ", shQuote("Status: OK"), " but ", shQuote(l)) else q("no")' -## R-devel on Linux -# TODO: --enable-strict-barrier --disable-long-double +## R-devel on Linux gcc strict +# R built with --enable-strict-barrier --disable-long-double # tests for compilation warnings # tests for new notes -# thus allow_failure -test-dev-cran-lin: +test-lin-dev-gcc-strict-cran: <<: *test-lin - image: registry.gitlab.com/jangorecki/dockerfiles/r-devel - allow_failure: true ## to not be blocked by changes in r-devel + image: registry.gitlab.com/jangorecki/dockerfiles/r-devel-gcc-strict variables: _R_CHECK_CRAN_INCOMING_: "TRUE" _R_CHECK_CRAN_INCOMING_REMOTE_: "FALSE" _R_S3_METHOD_LOOKUP_BASEENV_AFTER_GLOBALENV_: "FALSE" ## detects S3 method lookup found on search path #4777 _R_S3_METHOD_LOOKUP_REPORT_SEARCH_PATH_USES_: "TRUE" - before_script: + script: + - echo 'CFLAGS=-g -O2 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars + - echo 'CXXFLAGS=-g -O2 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars - *install-deps - - *cp-src - - rm -r bus + - R CMD check --as-cran $(ls -1t data.table_*.tar.gz | head -n 1) + - (! grep "warning:" data.table.Rcheck/00install.out) + - >- + Rscript -e 'l=tail(readLines("data.table.Rcheck/00check.log"), 1L); if (!identical(l, "Status: 3 NOTEs")) stop("Last line of ", shQuote("00check.log"), " is not ", shQuote("Status: 3 NOTEs"), " (size of tarball, suggested but not available for checking: 'curl', installed package size) but ", shQuote(l)) else q("no")' + +## R-devel on Linux clang +# R compiled with clang +# tests for compilation warnings +# tests for new notes +test-lin-dev-clang-cran: + <<: *test-lin + image: registry.gitlab.com/jangorecki/dockerfiles/r-devel-clang + variables: + _R_CHECK_CRAN_INCOMING_: "TRUE" + _R_CHECK_CRAN_INCOMING_REMOTE_: "FALSE" + _R_S3_METHOD_LOOKUP_BASEENV_AFTER_GLOBALENV_: "FALSE" + _R_S3_METHOD_LOOKUP_REPORT_SEARCH_PATH_USES_: "TRUE" script: - - *mv-src - - cd bus/$CI_JOB_NAME - - R CMD check --as-cran --no-manual $(ls -1t data.table_*.tar.gz | head -n 1) - - *rm-src + - echo 'CFLAGS=-g -O2 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' > ~/.R/Makevars + - echo 'CXXFLAGS=-g -O2 -flto=auto -fno-common -fopenmp -Wall -pedantic -fstack-protector-strong -D_FORTIFY_SOURCE=2' >> ~/.R/Makevars + - *install-deps + - R CMD check --as-cran $(ls -1t data.table_*.tar.gz | head -n 1) - (! grep "warning:" data.table.Rcheck/00install.out) - - >- ## this likely need an update but check fails now on complex NA so CI is not reaching here anyway - Rscript -e 'l=tail(readLines("data.table.Rcheck/00check.log"), 1L); if (!identical(l, "Status: 3 NOTEs")) stop("Last line of ", shQuote("00check.log"), " is not ", shQuote("Status: 3 NOTEs"), " (size of tarball, installed package size, top-level files) but ", shQuote(l)) else q("no")' + - >- + Rscript -e 'l=tail(readLines("data.table.Rcheck/00check.log"), 1L); if (!identical(l, "Status: 3 NOTEs")) stop("Last line of ", shQuote("00check.log"), " is not ", shQuote("Status: 3 NOTEs"), " (size of tarball, suggested but not available for checking: 'curl', installed package size) but ", shQuote(l)) else q("no")' ## R 3.1.0 # stated dependency on R -test-310-cran-lin: +test-lin-310-cran: image: registry.gitlab.com/jangorecki/dockerfiles/r-3.1.0 <<: *test-lin - before_script: - - *install-deps - - *cp-src - - rm -r bus script: - - *mv-src - - cd bus/$CI_JOB_NAME + - *install-deps - R CMD check --no-manual $(ls -1t data.table_*.tar.gz | head -n 1) - - *rm-src + +.test-win-template: &test-win + <<: *test + tags: + - shared-windows + before_script: + - curl.exe -s -o ../R-win.exe $R_BIN; Start-Process -FilePath ..\R-win.exe -ArgumentList "/VERYSILENT /DIR=C:\R" -NoNewWindow -Wait + - curl.exe -s -o ../rtools.exe $RTOOLS_BIN; Start-Process -FilePath ..\rtools.exe -ArgumentList "/VERYSILENT /DIR=C:\rtools" -NoNewWindow -Wait + - $env:PATH = "C:\R\bin;C:\rtools\usr\bin;$env:PATH" + - Rscript.exe -e "source('.ci/ci.R'); install.packages(dcf.dependencies('DESCRIPTION', which='all'), repos=file.path('file://',getwd(),'bus/mirror-packages/cran'), quiet=TRUE)" + - cp.exe $(ls.exe -1t bus/build/data.table_*.tar.gz | head.exe -n 1) . + script: + - R.exe CMD check --no-manual $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) + - R.exe CMD INSTALL --build $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) + after_script: + - $env:PATH = "C:\R\bin;C:\rtools\usr\bin;$env:PATH" + - mkdir.exe -p bus/$CI_JOB_NAME + - Rscript.exe -e "cat(Sys.getenv('CI_JOB_ID'), file=file.path('bus', Sys.getenv('CI_JOB_NAME'), 'id'))" + - Rscript.exe -e "cat(Sys.getenv('CI_JOB_STATUS'), file=file.path('bus', Sys.getenv('CI_JOB_NAME'), 'status'))" + - Rscript.exe -e "cat(Sys.getenv('CI_JOB_IMAGE'), file=file.path('bus', Sys.getenv('CI_JOB_NAME'), 'image'))" + - Rscript.exe -e "if (dir.exists(from<-'data.table.Rcheck')) file.rename(from, file.path('bus', Sys.getenv('CI_JOB_NAME'), 'data.table.Rcheck'))" + - Rscript.exe -e "if (length(from<-tail(list.files('^data\\.table_.*\\.zip$'), 1L))) file.rename(from, file.path('bus', Sys.getenv('CI_JOB_NAME'), from))" ## R-release on Windows # test and build binaries -test-rel-win: +test-win-rel: <<: *test-win variables: R_VERSION: "$R_REL_VERSION" - before_script: - - *install-r-rel-win - - *install-rtools43-win - - $ENV:PATH = "C:\R\bin;C:\rtools43\usr\bin;$ENV:PATH" - - *install-deps-win - - *cp-src-win - - rm.exe -r bus - script: - - *mv-src-win - - cd bus/$CI_JOB_NAME - - R.exe CMD check --no-manual $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - R.exe CMD INSTALL --build $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - *rm-src-win - - *mv-bin-win + R_BIN: "$R_REL_WIN_BIN" + RTOOLS_BIN: "$RTOOLS_REL_BIN" ## R-devel on Windows # test and build binaries -test-dev-win: +test-win-dev: <<: *test-win variables: - R_VERSION: "$R_DEVEL_VERSION" - allow_failure: true ## temp workaround #5748 - before_script: - - *install-r-devel-win - - *install-rtools43-win - - $ENV:PATH = "C:\R\bin;C:\rtools43\usr\bin;$ENV:PATH" - - *install-deps-win - - *cp-src-win - - rm.exe -r bus - script: - - *mv-src-win - - cd bus/$CI_JOB_NAME - - R.exe CMD check --no-manual --ignore-vignettes $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - R.exe CMD INSTALL --build $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - *rm-src-win - - *mv-bin-win + R_VERSION: "$R_DEV_VERSION" + R_BIN: "$R_DEV_WIN_BIN" + RTOOLS_BIN: "$RTOOLS_DEV_BIN" ## R-oldrel on Windows # test and build binaries -test-old-win: +test-win-old: <<: *test-win variables: - R_VERSION: "$R_OLDREL_VERSION" + R_VERSION: "$R_OLD_VERSION" + R_BIN: "$R_OLD_WIN_BIN" + RTOOLS_BIN: "$RTOOLS_OLD_BIN" + +.test-mac-template: &test-mac + <<: *test + tags: + - saas-macos-medium-m1 before_script: - - *install-r-oldrel-win - - *install-rtools42-win - - $ENV:PATH = "C:\R\bin;C:\rtools42\usr\bin;$ENV:PATH" - - *install-deps-win - - *cp-src-win - - rm.exe -r bus - script: - - *mv-src-win - - cd bus/$CI_JOB_NAME - - R.exe CMD check --no-manual --ignore-vignettes $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - R.exe CMD INSTALL --build $(ls.exe -1t data.table_*.tar.gz | head.exe -n 1) - - *rm-src-win - - *mv-bin-win + - cp $(ls -1t bus/build/data.table_*.tar.gz | head -n 1) . + after_script: + - mkdir -p bus/$CI_JOB_NAME + - '[ -d data.table.Rcheck ] && mv data.table.Rcheck bus/$CI_JOB_NAME/' + #- '[ -f data.table_*.tgz ] && mv $(ls -1t data.table_*.tgz | head -n 1) bus/$CI_JOB_NAME/' + - echo $CI_JOB_ID > bus/$CI_JOB_NAME/id + - echo $CI_JOB_STATUS > bus/$CI_JOB_NAME/status + - echo $CI_JOB_IMAGE > bus/$CI_JOB_NAME/image ## R-release on MacOS # no macosx runner set yet -#test-rel-mac: -# <<: *test-mac -# variables: -# R_VERSION: "$R_REL_VERSION" -# before_script: -# - *install-deps -# - *cp-src -# - rm -r bus -# script: -# - *mv-src -# - cd bus/$CI_JOB_NAME -# - R CMD check $(ls -1t data.table_*.tar.gz | head -n 1) -# - R CMD INSTALL --build $(ls -1t data.table_*.tar.gz | head -n 1) -# - mkdir -p cran/bin/macosx/el-capitan/contrib/$R_VERSION -# - mv $(ls -1t data.table_*.tgz | head -n 1) cran/bin/macosx/el-capitan/contrib/$R_VERSION -# - *rm-src -# - *mv-bin-mac +.test-mac-rel: + <<: *test-mac + variables: + R_VERSION: "$R_REL_VERSION" + script: + - *install-deps + - R CMD check $(ls -1t data.table_*.tar.gz | head -n 1) + - R CMD INSTALL --build $(ls -1t data.table_*.tar.gz | head -n 1) ## integrate artifacts # merging package tarballs and binaries into single R repository @@ -342,10 +294,13 @@ integration: - linux only: - master - needs: ["mirror-packages","build","test-rel-lin","test-rel-cran-lin","test-dev-cran-lin","test-rel-vanilla-lin","test-310-cran-lin","test-rel-win","test-dev-win","test-old-win"] + needs: ["mirror-packages","build","test-lin-rel","test-lin-rel-cran","test-lin-dev-gcc-strict-cran","test-lin-dev-clang-cran","test-lin-rel-vanilla","test-lin-310-cran","test-win-rel","test-win-dev" ,"test-win-old"] script: - R --version + - *install-deps ## markdown pkg not present in r-pkgdown image + - rm -rf ./vignettes ## r-lib/pkgdown#2383 - Rscript -e 'pkgdown::build_site(override=list(destination="./pkgdown"))' + - sed -i 's!!!g' pkgdown/index.html ## html manual, vignettes, repos, cran_web, cran_checks - echo 'source(".ci/ci.R"); source(".ci/publish.R")' >> .Rprofile ## list of available test-* jobs dynamically based on bus/test-* directories @@ -355,30 +310,33 @@ integration: ## delete any existing non-dev version of data.table - rm -f bus/mirror-packages/cran/src/contrib/data.table_*.tar.gz - rm -f bus/mirror-packages/cran/bin/windows/contrib/$R_REL_VERSION/data.table_*.zip - - rm -f bus/mirror-packages/cran/bin/windows/contrib/$R_DEVEL_VERSION/data.table_*.zip - - rm -f bus/mirror-packages/cran/bin/windows/contrib/$R_OLDREL_VERSION/data.table_*.zip + - rm -f bus/mirror-packages/cran/bin/windows/contrib/$R_DEV_VERSION/data.table_*.zip + - rm -f bus/mirror-packages/cran/bin/windows/contrib/$R_OLD_VERSION/data.table_*.zip #- rm -f bus/mirror-packages/cran/bin/macosx/el-capitan/contrib/$R_REL_VERSION/data.table_*.tgz - #- rm -f bus/mirror-packages/cran/bin/macosx/el-capitan/contrib/$R_DEVEL_VERSION/data.table_*.tgz - #- rm -f bus/mirror-packages/cran/bin/macosx/el-capitan/contrib/$R_OLDREL_VERSION/data.table_*.tgz + #- rm -f bus/mirror-packages/cran/bin/macosx/el-capitan/contrib/$R_DEV_VERSION/data.table_*.tgz + #- rm -f bus/mirror-packages/cran/bin/macosx/el-capitan/contrib/$R_OLD_VERSION/data.table_*.tgz ## merge mirror-packages and R devel packages - mv bus/mirror-packages/cran bus/$CI_JOB_NAME/ ## publish package sources - mkdir -p bus/$CI_JOB_NAME/cran/library bus/$CI_JOB_NAME/cran/doc - - mv $(ls -1t bus/build/cran/src/contrib/data.table_*.tar.gz | head -n 1) bus/$CI_JOB_NAME/cran/src/contrib + - mv $(ls -1t bus/build/data.table_*.tar.gz | head -n 1) bus/$CI_JOB_NAME/cran/src/contrib - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="source"), type="source", fields="Revision", addFiles=TRUE)' ## publish binaries - - Rscript -e 'move.bin("test-rel-win", Sys.getenv("R_REL_VERSION"), os.type="windows")' - - Rscript -e 'move.bin("test-dev-win", Sys.getenv("R_DEVEL_VERSION"), os.type="windows", silent=TRUE)' - - Rscript -e 'move.bin("test-old-win", Sys.getenv("R_OLDREL_VERSION"), os.type="windows")' + - mkdir -p bus/integration/cran/bin/windows/contrib/$R_REL_VERSION/ + - mkdir -p bus/integration/cran/bin/windows/contrib/$R_DEV_VERSION/ + - mkdir -p bus/integration/cran/bin/windows/contrib/$R_OLD_VERSION/ + - '[ -f bus/test-win-rel/data.table_*.zip ] && cp bus/test-win-rel/data.table_*.zip bus/integration/cran/bin/windows/contrib/$R_REL_VERSION/' + - '[ -f bus/test-win-dev/data.table_*.zip ] && cp bus/test-win-dev/data.table_*.zip bus/integration/cran/bin/windows/contrib/$R_DEV_VERSION/' + - '[ -f bus/test-win-old/data.table_*.zip ] && cp bus/test-win-old/data.table_*.zip bus/integration/cran/bin/windows/contrib/$R_OLD_VERSION/' - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="win.binary", ver=Sys.getenv("R_REL_VERSION")), type="win.binary", fields="Revision", addFiles=TRUE)' - - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="win.binary", ver=Sys.getenv("R_DEVEL_VERSION")), type="win.binary", fields="Revision", addFiles=TRUE)' - - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="win.binary", ver=Sys.getenv("R_OLDREL_VERSION")), type="win.binary", fields="Revision", addFiles=TRUE)' - #- Rscript -e 'move.bin("test-rel-mac", Sys.getenv("R_REL_VERSION"), os.type="macosx")' - #- Rscript -e 'move.bin("test-dev-mac", Sys.getenv("R_DEVEL_VERSION"), os.type="macosx")' - #- Rscript -e 'move.bin("test-old-mac", Sys.getenv("R_OLDREL_VERSION"), os.type="macosx")' + - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="win.binary", ver=Sys.getenv("R_DEV_VERSION")), type="win.binary", fields="Revision", addFiles=TRUE)' + - Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="win.binary", ver=Sys.getenv("R_OLD_VERSION")), type="win.binary", fields="Revision", addFiles=TRUE)' + #- Rscript -e 'move.bin("test-mac-rel", Sys.getenv("R_REL_VERSION"), os.type="macosx")' + #- Rscript -e 'move.bin("test-mac-dev", Sys.getenv("R_DEV_VERSION"), os.type="macosx")' + #- Rscript -e 'move.bin("test-mac-old", Sys.getenv("R_OLD_VERSION"), os.type="macosx")' #- Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="mac.binary.el-capitan", ver=Sys.getenv("R_REL_VERSION")), type="mac.binary.el-capitan", fields="Revision", addFiles=TRUE)' - #- Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="mac.binary.el-capitan", ver=Sys.getenv("R_DEVEL_VERSION")), type="mac.binary.el-capitan", fields="Revision", addFiles=TRUE)' - #- Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="mac.binary.el-capitan", ver=Sys.getenv("R_OLDREL_VERSION")), type="mac.binary.el-capitan", fields="Revision", addFiles=TRUE)' + #- Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="mac.binary.el-capitan", ver=Sys.getenv("R_DEV_VERSION")), type="mac.binary.el-capitan", fields="Revision", addFiles=TRUE)' + #- Rscript -e 'tools::write_PACKAGES(contrib.url("bus/integration/cran", type="mac.binary.el-capitan", ver=Sys.getenv("R_OLD_VERSION")), type="mac.binary.el-capitan", fields="Revision", addFiles=TRUE)' ## install all pkgs to render html and double check successful installation of all devel packages - mkdir -p /tmp/opencran/library /tmp/opencran/doc/html ## reset R_LIBS_USER to re-install all with html because pkgdown image has pre installed curl knitr - R_LIBS_USER="" Rscript -e 'install.packages("data.table", dependencies=TRUE, lib="/tmp/opencran/library", repos=file.path("file:",normalizePath("bus/integration/cran")), INSTALL_opts="--html", quiet=TRUE)' @@ -397,7 +355,7 @@ integration: ## web/checks/$pkg/$job 00install.out, 00check.log, *.Rout, memtest.csv, memtest.png ## memtest not available for now #5764 - Rscript -e 'sapply(names(test.jobs), check.copy, simplify=FALSE)' ## web/packages/$pkg/$pkg.pdf - - Rscript -e 'pdf.copy("data.table", "test-rel-lin")' + - Rscript -e 'pdf.copy("data.table", "test-lin-rel")' ## web/checks/check_results_$pkg.html - Rscript -e 'check.index("data.table", names(test.jobs))' ## web/checks/check_flavors.html @@ -405,11 +363,6 @@ integration: ## pkgdown merge - Rscript -e 'common_files<-function(path1, path2) intersect(list.files(path1, all.files=TRUE, no..=TRUE), list.files(path2, all.files=TRUE, no..=TRUE)); msg = if (length(f<-common_files("pkgdown","bus/integration/cran"))) paste(c("Following artifacts will be overwritten by pkgdown artifacts:", paste0(" ", f)), collapse="\n") else "No overlapping files from pkgdown artifacts"; message(msg); q("no")' - mv pkgdown/* bus/integration/cran/ - ## cleanup artifacts from other jobs - - mkdir tmpbus - - mv bus/$CI_JOB_NAME tmpbus - - rm -r bus - - mv tmpbus bus <<: *artifacts ## publish diff --git a/R/devel.R b/R/devel.R index 8bd7a1466a..3aed1017f8 100644 --- a/R/devel.R +++ b/R/devel.R @@ -17,9 +17,8 @@ dcf.repo = function(pkg, repo, field, type) { dcf[dcf[,"Package"]==pkg, field][[1L]] } -update_dev_pkg = function(object="data.table", repo="https://Rdatatable.gitlab.io/data.table", field="Revision", type=getOption("pkgType"), lib=NULL, ...) { +update_dev_pkg = function(pkg="data.table", repo="https://Rdatatable.gitlab.io/data.table", field="Revision", type=getOption("pkgType"), lib=NULL, ...) { # this works for any package, not just data.table - pkg = object # perform package upgrade when new Revision present stopifnot(is.character(pkg), length(pkg)==1L, !is.na(pkg), is.character(repo), length(repo)==1L, !is.na(repo), @@ -28,7 +27,7 @@ update_dev_pkg = function(object="data.table", repo="https://Rdatatable.gitlab.i # get Revision field from remote repository PACKAGES file una = is.na(ups<-dcf.repo(pkg, repo, field, type)) if (una) - catf("No revision information found in DESCRIPTION file for %s package. Unsure '%s' is correct field in PACKAGES file in your package repository '%s'. Otherwise package will be re-installed every time, proceeding to installation.\n", + catf("No revision information found in DESCRIPTION file for %s package. Make sure that '%s' is correct field in PACKAGES file in your package repository '%s'. Otherwise package will be re-installed every time, proceeding to installation.\n", pkg, field, contrib.url(repo, type=type)) # see if Revision is different then currently installed Revision, note that installed package will have Revision info only when it was installed from remote devel repo upg = una || !identical(ups, dcf.lib(pkg, field, lib.loc=lib)) @@ -44,6 +43,7 @@ update_dev_pkg = function(object="data.table", repo="https://Rdatatable.gitlab.i unname(read.dcf(system.file("DESCRIPTION", package=pkg, lib.loc=lib, mustWork=TRUE), fields=field)[, field]), utils::packageVersion(pkg, lib.loc=lib))) }) + invisible(upg) } # non-exported utility when using devel version #3272: data.table:::.git() diff --git a/_pkgdown.yml b/_pkgdown.yml index 4b02b39491..117ec29574 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -1,8 +1,4 @@ -url: https://Rdatatable.gitlab.io/data.table - -template: - params: - ganalytics: UA-129166154-2 +url: https://rdatatable.gitlab.io/data.table development: version_tooltip: "Development version" @@ -11,6 +7,8 @@ home: links: - text: CRAN-like website href: web/packages/data.table/index.html + - text: CRAN-like checks + href: web/checks/check_results_data.table.html navbar: structure: @@ -22,30 +20,30 @@ navbar: href: index.html introduction: text: Introduction - href: articles/datatable-intro.html + href: library/data.table/doc/datatable-intro.html articles: text: Vignettes menu: - text: "Introduction to data.table" - href: articles/datatable-intro.html + href: library/data.table/doc/datatable-intro.html - text: "Reference semantics" - href: articles/datatable-reference-semantics.html + href: library/data.table/doc/datatable-reference-semantics.html - text: "Using .SD for Data Analysis" - href: articles/datatable-sd-usage.html + href: library/data.table/doc/datatable-sd-usage.html - text: "Keys and fast binary search based subset" - href: articles/datatable-keys-fast-subset.html + href: library/data.table/doc/datatable-keys-fast-subset.html - text: "Secondary indices and auto indexing" - href: articles/datatable-secondary-indices-and-auto-indexing.html + href: library/data.table/doc/datatable-secondary-indices-and-auto-indexing.html - text: "Efficient reshaping using data.table" - href: articles/datatable-reshape.html + href: library/data.table/doc/datatable-reshape.html - text: "Programming on data.table" - href: articles/datatable-programming.html + href: library/data.table/doc/datatable-programming.html - text: "Frequently asked questions" - href: articles/datatable-faq.html + href: library/data.table/doc/datatable-faq.html - text: "Importing data.table" - href: articles/datatable-importing.html + href: library/data.table/doc/datatable-importing.html - text: "Benchmarking data.table" - href: articles/datatable-benchmarking.html + href: library/data.table/doc/datatable-benchmarking.html news: text: News href: news/index.html @@ -64,3 +62,8 @@ navbar: github: icon: fab fa-github fa-lg href: https://github.com/Rdatatable/data.table + +templates: + includes: + in_header: | + diff --git a/man/update_dev_pkg.Rd b/man/update_dev_pkg.Rd index 3db5b98316..66fff0422d 100644 --- a/man/update_dev_pkg.Rd +++ b/man/update_dev_pkg.Rd @@ -2,14 +2,14 @@ \alias{update_dev_pkg} \title{Perform update of development version of a package} \description{ - Downloads and installs latest development version only when a new commit is available which has also passed all tests. Defaults are set to update \code{data.table}, other packages can be used as well. Their repository has to include git commit information in PACKAGES file. + Downloads and installs latest development version, only when a new commit is available. Defaults are set to update \code{data.table}, other packages can be used as well. Repository of a package has to include git commit SHA information in PACKAGES file. } -\usage{update_dev_pkg(object="data.table", +\usage{update_dev_pkg(pkg="data.table", repo="https://Rdatatable.gitlab.io/data.table", field="Revision", type=getOption("pkgType"), lib=NULL, \dots) } \arguments{ - \item{object}{ character scalar, package name. } + \item{pkg}{ character scalar, package name. } \item{repo}{ character scalar, url of package devel repository. } \item{field}{ character scalar, metadata field to use in PACKAGES file and DESCRIPTION file, default \code{"Revision"}. } @@ -20,13 +20,18 @@ \item{\dots}{ passed to \code{\link[utils]{install.packages}}. } } \details{ - In case if a devel repository does not provide binaries user will need development tools installed for package compilation, like \emph{Rtools} on Windows, and eventually set \code{type="source"}. + In case if a devel repository does not provide binaries user will need development tools installed for package compilation, like \emph{Rtools} on Windows, or alternatively eventually set \code{type="source"}. +} +\section{data.table repositories}{ + By default the function uses our GitLab-hosted R repository at \code{https://Rdatatable.gitlab.io/data.table}. This repository is updated nightly. It runs multiple test jobs (on top of GitHub tests jobs run upstream) and publish the package (sources and binaries), even if GitLab test jobs are failing. Status of GitLab test jobs can be checked at \href{https://rdatatable.gitlab.io/data.table/web/checks/check_results_data.table.html}{Package Check Results}.\cr + We also publish bleeding edge version of the package on GitHub-hosted R repository at \code{https://Rdatatable.gitlab.io/data.table} (just minor change in url from \emph{lab} to \emph{hub}). GitHub version should be considered less stable than GitLab one. It publishes only package sources.\cr + There are also other repositories maintained by R community, for example \code{https://rdatatable.r-universe.dev}. Those can be used as well, but as they are unlikely to provide git commit SHA, the function will install the package even if latest version is already installed. } \note{ Package namespace is unloaded before attempting to install newer version. } \value{ - NULL. + Invisibly \code{TRUE} if package was updated, otherwise \code{FALSE}. } \examples{ \dontshow{ # using if(FALSE) because \dontrun could still be run by --run-dontrun; #5421 }