Add buildkite config script

[wilzbach]

Preview: https://buildkite.com/d-lang/dmd-test/builds
TODO:

• create pipeline for each repository and add webhook for dmd,druntime,phobos,tools,dub (probably a good idea to start with one repo initially)

if [ [ "$BUILDKITE_PULL_REQUEST" == "false" ] && [ "$BUILDKITE_PULL_REQUEST_REPO" != "git://github.com/dlang/ci.git" ] ] ; then
   wget https://github.com/dlang/ci/raw/master/buildkite.sh
fi

bash buildkite.sh | buildkite-agent pipeline upload

• setup more agents (it seems like some projects do need quite a lot of memory, so we probably can't run too many agents on a single machine)

sudo apt-get update
sudo apt-get install -y apt-transport-https sudo
sudo sh -c 'echo deb https://apt.buildkite.com/buildkite-agent stable main > /etc/apt/sources.list.d/buildkite-agent.list'
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 32A37959C2FA5C3C99EFBC32A79206696452D198
sudo apt-get update && sudo apt-get install -y buildkite-agent
# add token (see buildkite)
sudo systemctl enable buildkite-agent@1 && systemctl start buildkite-agent@1
sudo systemctl enable buildkite-agent@2 && systemctl start buildkite-agent@2
# ...
# install global dependencies for the build
sudo apt-get install python python-yaml make gdb g++ gcc libcurl3-dev libssl1.0-dev

• copy all project configuration
• make sure that all projects pass (currently the errors only seem to be due to missing memory)
• figure out how to make the build logs publicly accessible
• figure out why the CI notifications aren't displayed here

Other notes:

• dub doesn't build in release mode with dmd master (to be investigated)

[dlang-bot]

Thanks for your pull request, @wilzbach!

[wilzbach]

Preview:

[wilzbach]

CC @CyberShadow. I'm not entirely sure why extended file attributes aren't supported under OpenVZ, but do you have a better/more generic idea than this patch?

[CyberShadow]

What was the failing error message?

[wilzbach]

https://buildkite.com/d-lang/dmd-test/builds/119#ec6df065-b730-4db4-b2d2-a899fb29dd0f (needs credentials until it's made open source which should hopefully happen soon - it depends on their support to approve your status as open source org.)

ae 0.0.2231: building configuration "ae-test-library"...
Linking...
Running ./ae-test-library 
Current filesystem is , not btrfs, skipping ae.sys.btrfs.clone_range test.
Current filesystem is , not btrfs, skipping ae.sys.btrfs.extent_same test.
Initialized empty Git repository in /var/lib/buildkite-agent/builds/srv-3/d-lang/dmd-test/ae/test-cache/cache/.git/
fatal: 'refs/ae-sys-d-cache/test-key' - not a valid ref
a6a0e35a7ca8f1c00290fc6a463cc42daed0b185 refs/ae-sys-d-cache/test-key
fatal: 'refs/ae-sys-d-cache/test-key' - not a valid ref
notice: HEAD points to an unborn branch (master)
Checking object directories:  10% (26/256)   
Checking object directories:  32% (82/256)   
Checking object directories:  39% (100/256)   
Checking object directories:  51% (132/256)   
Checking object directories:  54% (140/256)   
Checking object directories:  56% (145/256)   
Checking object directories:  58% (149/256)   
Checking object directories:  59% (153/256)   
Checking object directories:  65% (167/256)   
Checking object directories:  98% (253/256)   
Checking object directories: 100% (256/256)   
Checking object directories: 100% (256/256), done.
dangling commit a6a0e35a7ca8f1c00290fc6a463cc42daed0b185
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects:  50% (1/2)   
Compressing objects: 100% (2/2)   
Compressing objects: 100% (2/2), done.
Writing objects:  20% (1/5)   
Writing objects:  40% (2/5)   
Writing objects:  60% (3/5)   
Writing objects:  80% (4/5)   
Writing objects: 100% (5/5)   
Writing objects: 100% (5/5), done.
Total 5 (delta 0), reused 0 (delta 0)
std.exception.ErrnoException@sys/file.d(853): Error reading file xattrs (Operation not supported)
----------------
/var/lib/buildkite-agent/builds/srv-3/d-lang/dmd-test/distribution/bin/../imports/std/exception.d:515 @safe void std.exception.bailOut!(std.exception.ErrnoException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x8ed87cf1]
/var/lib/buildkite-agent/builds/srv-3/d-lang/dmd-test/distribution/bin/../imports/std/exception.d:436 @safe bool std.exception.enforce!(std.exception.ErrnoException).enforce!(bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8edc1166]
sys/file.d:853 bool ae.sys.file.XAttrs!(const(char)*, "").XAttrs.opIn_r(immutable(char)[]) [0x8ee9de5c]
sys/file.d:914 void ae.sys.file.__unittest_L906_C2() [0x8ee9ae67]
??:? void ae.sys.file.__modtest() [0x8eea00dd]
??:? int core.runtime.runModuleUnitTests().__foreachbody2(object.ModuleInfo*) [0x8f01d297]
??:? int object.ModuleInfo.opApply(scope int delegate(object.ModuleInfo*)).__lambda2(immutable(object.ModuleInfo*)) [0x8efb3f32]
??:? int rt.minfo.moduleinfos_apply(scope int delegate(immutable(object.ModuleInfo*))).__foreachbody2(ref rt.sections_elf_shared.DSO) [0x8efc1b89]
??:? int rt.sections_elf_shared.DSO.opApply(scope int delegate(ref rt.sections_elf_shared.DSO)) [0x8efc1fec]
??:? int rt.minfo.moduleinfos_apply(scope int delegate(immutable(object.ModuleInfo*))) [0x8efc1b15]
??:? int object.ModuleInfo.opApply(scope int delegate(object.ModuleInfo*)) [0x8efb3f09]
??:? runModuleUnitTests [0x8f01d06d]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x8efbbfe0]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x8efbbf67]
??:? _d_run_main [0x8efbbed2]
??:? main [0x8ed5ef5d]
??:? __libc_start_main [0x8d83b2e0]
1/76 unittests FAILED
Program exited with code 1
�[31m🚨 Error: The command exited with status 2�[0m
^^^ +++
^^^ +++

[MartinNowak]

I already reserved https://buildkite.com/dlang and have successfully applied as open source organisation, other than that great and thanks for helping with this.

[MartinNowak]

What agents are you currently using, I plan to use on-demand Hetzner KVMs (http://hetzner.de/cloud). Scaling those to the demand will require a bit of an algorithm.

[MartinNowak]

They (buildkite) told me via Mail, that they're working on publically visible build results for open source projects atm. We will likely participate in the beta of that feature. Hopefully won't take too long, as this seems like a hard requirement to replace Jenkins.

[wilzbach]

What agents are you currently using

A VPS from VPSDime that I had running around anyhow.
As it only has 6 GB of RAM, it can run only two agents in parallel. Otherwise DMD's greedy memory consumption kicks in and results in random failures.

Scaling those to the demand will require a bit of an algorithm.

Might be easier to just be a bit more excessive with agents and do a bit of retry logic. Actually only four projects failed when I was running this with four agents (the machine has four cores).
My guess is that - even if we have to restart 4/40 project for a build - we might be faster when we can run 2x more in parallel.

[MartinNowak]

Hopefully won't take too long, as this seems like a hard requirement to replace Jenkins.

Mail was from mid May.

[wilzbach]

I already reserved https://buildkite.com/dlang

Oh wow. Cool!
I already transferred over the agents and created https://buildkite.com/dlang/ci.
So basically all we need is to configure the webhook for this repo and we can fully start playing with this.

[wilzbach]

What agents are you currently using, I plan to use on-demand Hetzner KVMs (http://hetzner.de/cloud). Scaling those to the demand will require a bit of an algorithm.

I found a decent offer at https://lowendbox.com
tl;dr: the yearly fee was about the same as two months for an equivalent box at Hetzners (6 GB RAM, 4vCPU) and they have a free three day refund, so I was willing to give it a shot and so far it looks like it would be a solution that's already faster than the existing Jenkins setup:

[wilzbach]

I'm tired of fixing Jenkins (it's currently having issues again) and will rather tell people why buildkite failed or preferably make it, s.t. it doesn't. Also we can invite most contributors to buildkite as a temporary workaround.