release-osuosl-aix61-ppc64_be-1 disk full

[rvagg]

today's builds are failing

13:07:32 as: There is not enough space in the file system.

@nodejs/platform-aix

[richardlau]

Possibly the same issue (#1951) that was seen on test-osuosl-aix61-ppc64_be-2 recently where /tmp needed clearing out?

[rvagg]

ran that command @sam-github used and it went from

/dev/hd3          262144    205936   22%      984     4% /tmp

to

/dev/hd3          262144    247592    6%       25     1% /tmp

not really much space being used there. / seems like a more realistic culprit:

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         1310720         0  100%    17703    92% /
/dev/hd2         5898240    131096   98%    52089    70% /usr
/dev/hd9var      1179648    276192   77%     6769    18% /var
/dev/hd3          262144    247592    6%       25     1% /tmp
/dev/hd1          131072    130320    1%        7     1% /home2
/dev/hd11admin     262144    261384    1%        5     1% /admin
/proc                  -         -    -         -     -  /proc
/dev/hd10opt     3538944   2109096   41%    18556     8% /opt
/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump
/dev/fslv00    117178368  97295512   17%   176808     2% /home
/aha                   -         -    -        16     1% /aha

but I'm not sure how that would get involved in as, maybe there's another temp directory in there somewhere?

[sam-github]

-rw-r----- 1 root system 382222336 Oct 14 00:52 failedlogin /etc/security was huge, I cleared that out.

[richardlau]

Is this the reason we've not had an AIX 13.0.0-nightly since https://nodejs.org/download/nightly/v13.0.0-nightly20191010768287489a/?

It's missing from https://nodejs.org/download/nightly/v13.0.0-nightly2019101181bc7b3ba5/ onwards and looks to be the reason why node-test-node-addon-api-LTS versions has been failing.

[sam-github]

I don't know how the nightlies are built, but the disk still has lots of space,

[mhdawson]

@sam-github which disk di you check? This is the output on the release machine:

../../deps/v8/src/builtins/typed-array-filter.tq ../../deps/v8/src/builtins/typed-array-find.tq ../../deps/v8/src/builtins/typed-array-findindex.tq ../../deps/v8/src/builtins/typed-array-foreach.tq ../../deps/v8/src/builtins/typed-array-reduce.tq ../../deps/v8/src/builtins/typed-array-reduceright.tq ../../deps/v8/src/builtins/typed-array-slice.tq ../../deps/v8/src/builtins/typed-array-some.tq ../../deps/v8/src/builtins/typed-array-subarray.tq ../../deps/v8/src/builtins/typed-array.tq ../../deps/v8/third_party/v8/builtins/array-sort.tq ../../deps/v8/test/torque/test-torque.tq ../../deps/v8/src/objects/intl-objects.tq -o "/home/iojs/build/ws/out/Release/obj/gen/torque-output-root/torque-generated" -v8-root ../../deps/v8
02:10:43 as: There is not enough space in the file system.
02:10:43 Assembler:
02:10:43 /tmp//ccaK8Mqk.s: line 119128: 1252-161 A system problem occurred while processing file /tmp/asR70aaa
02:10:45 gmake[2]: *** [/home/iojs/build/ws/out/Release/obj.target/icudata/gen/icudt64_dat.o] Error 1
02:10:45 gmake[2]: *** Waiting for unfinished jobs....

[sam-github]

https://ci-release.nodejs.org/job/iojs+release/nodes=aix61-ppc64/4769/console

So, this is failing on icudt64_dat.o, which I suspect is not a coincidence.

Its probably not so much that cruft has accumulated and filled up space, but that the full ICU has significantly increased demand for /tmp space.

release has significantly less capacity:

% ssh release-osuosl-aix61-ppc64_be-1 df /tmp
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3          262144    236176   10%       28     1% /tmp
% ssh test-osuosl-aix61-ppc64_be-1 df /tmp   
/dev/hd3          917504    708368   23%      690     1% /tmp
% ssh test-osuosl-aix61-ppc64_be-2 df /tmp
/dev/hd3          917504    775296   16%      287     1% /tmp
% ssh test-osuosl-aix61-ppc64_be-3 df /tmp
/dev/hd3         6291456   4503184   29%     9176     2% /tmp

I'll have to figure out if its possible to increase the tmp space, or to convince the AIX as to use a different tmpfile.

[sam-github]

I resized /tmp:

# chfs -a size=917504 /tmp                                  
Filesystem size changed to 917504

Its now the same as the 2 smaller test machines:

~ % ssh release-osuosl-aix61-ppc64_be-1 df /tmp
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3          917504    891432    3%       28     1% /tmp
~ % ssh test-osuosl-aix61-ppc64_be-1 df /tmp   
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3          917504    708368   23%      690     1% /tmp

[sam-github]

https://ci-release.nodejs.org/job/iojs+release/4770/nodes=aix61-ppc64/ is building, we'll see if it works.

And if we get nightly tomorrow.

[sam-github]

https://ci-release.nodejs.org/job/iojs+release/4770/nodes=aix61-ppc64/ was green, so hopefully nightlies will occur.

[sam-github]

For the record, this was not "simply" /tmp filling up, its fallout from big-ICU, because massive .S icu data files need a lot more disk space during assembly. Big ICU has stressed various tools. I was getting ccache OOM errors on AIX7.2 because it loads full files into memory to hash them, for example.

The fix wasn't cleaning /tmp, it was resizing it to be identical to the test- machines, because they obviously have enough /tmp space to compile ICU.

[sam-github]

AFAICT, no nightlies were run last night at all: https://ci-release.nodejs.org/job/iojs+release/ shows the build @mhdawson and I started yesterday, green, and a master build targos did by hand, and that's it.

I don't see a nightly job, either, or a periodic trigger for the iojs+release job. :headscratching: