Updates sometimes don't propogate

[wilzbach]

PR: dlang/dmd#7553

was fixed in #108 and even though it was repeatedly restarted afterwards, e.g.

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7553/7/pipeline/183

It still showed;

Checking out Revision 2427af70404bbc8878da7295d0e662a458a6a93e (v4.0.0-alpha.2)

Found by @JinShil

[wilzbach]

Rebasing the PR to upstream/master fixed the problem: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/PR-7553/8/pipeline/186 for this case.

[ibuclaw]

It looks like its more of a timing issue, and just pot luck depending on which runner you hit.

Looking at the build history of the last 2 hours (Sunday, Jan 7, 12pm UTC)
https://ci.dlang.io/view/all/builds

Timeline is this:

Time (approx)	Change	Runner	Result
10:25	D Programming Language » ci » PR-115 # 1	runner-01.ci.lxd	Pass
10:45	D Programming Language » ci » master # 66	runner-02.ci.lxd	Pass
10:51	D Programming Language » dmd » PR-7625 # 2	runner-01.ci.lxd	Pass
10:52	D Programming Language » dmd » PR-7624 # 2	runner-02.ci.lxd	Fail
10:54	D Programming Language » dmd » PR-7622 # 2	runner-02.ci.lxd	Fail
10:54	D Programming Language » dmd » PR-7620 # 2	runner-02.ci.lxd	Fail
10:55	D Programming Language » dmd » PR-7619 # 2	runner-02.ci.lxd	Fail
11:03	D Programming Language » dmd » PR-7617 # 13	runner-01-gce	Pass
11:24	D Programming Language » dmd » PR-7619 # 3	runner-01.ci.lxd	Pass
11:25	D Programming Language » dmd » PR-7620 # 3	runner-02.ci.lxd	Pass

The first two are for a commit that should have fixed all builds.

The first successful build on runner-02.ci.lxd was nearly an hour later.

Looking at the logs of the failed builds, i.e: https://ci.dlang.io/job/dlang-org/job/dmd/job/PR-7619/2/consoleFull

Early on, there is a message that doesn't appear in the successful builds:

Replacing contents of vars/runPipeline.groovy

Which I can only assume is related to the "fix" in #115.

From this, it would appear that it takes a while for changes to propagate to all runners, which may not be desirable if something requires immediate feedback for a problem that needs fixing asap.

[MartinNowak]

No, Jenkins should always fetch vars/runPipeline.groovy from master of the ci repo.

[MartinNowak]

was fixed in #108 and even though it was repeatedly restarted afterwards, e.g.

Because restarting, replays with the same parameters. That extends to anything checked out by Jenkins itself (dlang/ci@master, and the testee), but not the repos where we manually fetch a branch (instead of a tag) in the script.
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/activity?branch=PR-7553

[MartinNowak]

FYI @JinShil

[wilzbach]

Because restarting, replays with the same parameters. That extends to anything checked out by Jenkins itself (dlang/ci@master, and the testee), but not the repos where we manually fetch a branch (instead of a tag) in the script.

I observed otherwise. Have a look at these logs. If replays would be with the same parameters, then "replays" shouldn't pass right?

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/activity/?branch=PR-2015
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/activity/?branch=PR-2017
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/activity/?branch=PR-2018
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/activity/?branch=PR-2020

The second last "Replay" was made after the fix for dlang/ci was merged which imho clearly indicates that there's a cache.

[MartinNowak]

As I said, for the projects checked out by Jenkins itself, that's dlang/ci@master (referenced in the Jenkinsfile) and the testee (the one that triggered the build) checked-out here.
Those two are replayed at the same state/commit, not even merging into an updated base branch (master) IIRC.

For the rest the script fetches references itself (see here), thus you'll get some updates even in replays.

[wilzbach]

not even merging into an updated base branch (master) IIRC.

Hehe - Jenkins doesn't checkout the testee branch, it always checks out master:

#120 (I found out about this the hard way - see below).

For the rest the script fetches references itself (see here), thus you'll get some updates even in replays.

The only thing that broke Jenkins yesterday was dlang/druntime#2036 (it passed there, hence #120). I consequently added a temporary fix to the groovy script at dlang/ci (#121).
Now I retriggered the druntime PRs and they still failed with the same error message (e.g. https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdruntime/detail/PR-2015/8/pipeline/209), so I'm pretty sure that there's a cache. As I said replaying after a couple of hours worked. My assumption is that dlang/ci only gets cloned once for a worker and doesn't get updated on a new PR job, but only due to the worker being de-provisioned and re-provisioned.

[MartinNowak]

Mmh, dlang/ci is included directly in Jenkinsfile with library identifier: "dlang@master". The branch is hardcoded in the admin settings. So indeed it might be that Jenkins is updating that master ref globally.