Use fallback mirrors in the install script by wilzbach · Pull Request #291 · dlang/installer

wilzbach · 2018-01-07T13:19:21Z

Failures are getting more frequent, see e.g. travis-ci/travis-build#1282
This is a first step at adding more mirrors.

dlang-bot · 2018-01-07T13:19:21Z

Thanks for your pull request, @wilzbach!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

wilzbach · 2018-01-10T09:56:19Z

script/install.sh

-    retry curl2 -sS "$url" -o "$path"
+    try_all_mirrors() {
+        for mirror in "${mirrors[@]}" ; do
+            if curl2 -sS "$mirror" -o "$path" ; then


This is copy due to curl2 being used here and curl in download. Is there any reason not to use curl2 in download?

wilzbach · 2018-01-10T09:59:24Z

script/install.sh

+    mirrors=(
+        "https://dlang.org/install.sh"
+        "https://downloads.dlang.org/other/install.sh"
+        "https://nightlies.dlang.org/install.sh"


I would add https://github.com/dlang/installer/raw/stable/script/install.sh to the list too, but I wasn't sure whether you would be okay with it.

Let's not automatically deploy this, it's not worth the security implications.
Even without sudo (which often is cached), rm -rf ~ can do a lot of damage.

wilzbach · 2018-01-10T10:00:22Z

script/install.sh

 verify() {
+    local keyring_mirrors=(
+        "https://dlang.org/d-keyring.gpg"
+        "https://github.com/dlang/dlang.org/raw/846269e913114fd77af0cc1aff67269046b70db8/d-keyring.gpg"


I couldn't find another mirror for this.

There you go, https://nightlies.dlang.org/d-keyring.gpg.
In fact you can upload files to sebastian@nightlies.dlang.org:/var/www/builds yourself.
But you cannot move or delete my entries in that folder (sticky bit). Allowing any nginx group member to mess with the install script and the signature key sounds like disaster.

wilzbach · 2018-01-10T11:19:55Z

FWIW for gdc downloads (e.g. at dlang/dmd) I still see failures related to the dub-registry being down often, e.g. https://semaphoreci.com/cybershadow/dmd/branches/pull-request-7659/builds/2, as gdc doesn't ship with dub.

So it would be cool to add a mirror for the DUB binaries too (see #273)

@ibuclaw

code.dlang.org is often down and there's no mirror for http://code.dlang.org/download/LATEST yet See also: - dlang/dub#1290 - dlang/installer#273 - dlang/installer#291 CC @ibuclaw

MartinNowak · 2018-01-16T05:05:58Z

script/install.sh

+                "http://ftp.digitalmars.com/LATEST_BETA"
+            )
+            logV "Determing latest dmd-beta version (${mirrors[0]})."
+            COMPILER="dmd-$(fetch "${mirrors[*]}")"


Think this should be [@] expanded, so fetch really gets 2 arguments.

cnt() { echo $#; } args=('entry with' 'spaces') cnt "${args[*]}" # 1 cnt "${args[@]}" # 2 cnt ${args[*]} # 3 cnt ${args[@]} # 3

As with "$@" you always want to pass arguments in an array using `"${ary[@]}".

MartinNowak · 2018-01-16T05:09:42Z

script/install.sh

-    local url path
-    url=$1
+    local mirrors path
+    mirrors=($1)


This does IFS/whitespace splitting. Rather pass all the arguments separately on the call-side and use ("$@") here.

cnt() { echo $#; } bug() { ary=($@); cnt "${ary[@]}"; } bug 1 2 # -> 2 bug '1 1' 2 # -> 3 :/ ok() { ary=("$@"); cnt "${ary[@]}"; } ok 1 2 # -> 2 ok '1 1' 2 # -> 2 :)

I'm aware (I did it on purpose after all) - urls aren't supposed to contain spaces 😉
I did have this, s.t. I can pack the urls for mirrors and their signatures as individual arguments for the download function.

MartinNowak · 2018-01-16T05:11:03Z

script/install.sh

 }
-
 # url, path, [gpg signature url to verify]
+# url and the gpg signature can be an array of urls


Let's change it to path, verify, url... then, and always require 0 or 1 for verify.

The signature can be assumed to be url+.sig.

MartinNowak · 2018-01-16T05:16:03Z

script/install.sh

-    log "Downloading $url"
-    download "$url" "$tmp/install.sh" "$url.sig"
+    log "Downloading ${mirrors[0]}"
+    download "${mirrors[*]}" "$tmp/install.sh" "${mirrors[*]/%/.sig}"


Again .sig is implicit when verify=1, so the call would be.

download "$tmp/install.sh" 1 "${mirrors[@]}"

Or for better readability, use download_and_verify and plain download.

MartinNowak · 2018-01-16T05:54:32Z

LMostlyGTM

…t the end

wilzbach · 2018-01-29T11:29:13Z

Ping @MartinNowak - are we good here?
People are still seeing random failures on their CI jobs (e.g. https://travis-ci.org/d-widget-toolkit/dwt/builds/334451034). Note in that case the DUB binary failed, but I will revive #273 as soon as the mirroring functionality has been merged. with this PR #273 will be a simple two-line diff.

CC @jacob-carlborg

wilzbach · 2018-01-29T11:31:28Z

script/install.sh

+# path, urls...
+download_without_verify() {
+    download "$1" 0 "${@:2}"
+}


I added these to functions to avoid the need of passing around a magic 1 or 0

wilzbach · 2018-01-29T11:33:18Z

script/install.sh

+
+# path, urls...
+download_and_unpack_without_verify() {
+    download_and_unpack "$1" 0 "${@:2}"


These two functions were added to avoid passing around magic 0 or 1

wilzbach · 2018-02-06T09:27:07Z

Ping @MartinNowak - just saw another random failure on Travis :/

- no https on s3 - gh url for keyring needs to be updated in install.sh script (we already have a distribution/update problem for d-keyring.gpg, so let's not add another one) (also shattered.io to a tiny degree)

wilzbach mentioned this pull request Jan 7, 2018

Add more mirrors for the D install script travis-ci/travis-build#1282

Merged

wilzbach force-pushed the dmd-mirror branch 3 times, most recently from 71fae3c to 81b1fa2 Compare January 10, 2018 03:56

wilzbach commented Jan 10, 2018

View reviewed changes

This was referenced Jan 12, 2018

Cache D installer download on SemaphoreCI dlang/dmd#7685

Closed

auto-merge should obey combined CI status dlang/dlang-bot#69

Merged

MartinNowak reviewed Jan 16, 2018

View reviewed changes

wilzbach force-pushed the dmd-mirror branch 2 times, most recently from 72c54ec to 24dffbf Compare January 16, 2018 19:11

wilzbach added 2 commits January 16, 2018 20:12

Use fallback mirrors in the install script

cbf70b0

Change fetch,download,download_unpack,verify signatures to put urls a…

dff4677

…t the end

wilzbach force-pushed the dmd-mirror branch from 24dffbf to 30022e2 Compare January 16, 2018 19:12

wilzbach closed this Jan 17, 2018

wilzbach reopened this Jan 17, 2018

wilzbach closed this Jan 17, 2018

wilzbach reopened this Jan 17, 2018

Use download_with_verify aliases for better readability

c42ccff

wilzbach force-pushed the dmd-mirror branch from 30022e2 to c42ccff Compare January 17, 2018 01:18

wilzbach closed this Jan 17, 2018

wilzbach reopened this Jan 17, 2018

Fix ShellCheck

6081ccc

wilzbach mentioned this pull request Jan 29, 2018

Use a fallback mirror for code.dlang.org #273

Merged

wilzbach commented Jan 29, 2018

View reviewed changes

remove problematic mirrors

d24bcb6

- no https on s3 - gh url for keyring needs to be updated in install.sh script (we already have a distribution/update problem for d-keyring.gpg, so let's not add another one) (also shattered.io to a tiny degree)

MartinNowak merged commit 3e8e46a into dlang:master Feb 17, 2018

Uh oh!

Comments

Conversation

wilzbach commented Jan 7, 2018

Uh oh!

dlang-bot commented Jan 7, 2018

Bugzilla references

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wilzbach commented Jan 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak commented Jan 16, 2018

Uh oh!

wilzbach commented Jan 29, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wilzbach commented Feb 6, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MartinNowak Jan 16, 2018 •

edited

Loading

MartinNowak Jan 16, 2018 •

edited

Loading