thread pool management - really high numbers of threads and memory usage per buildkite-agent #3302
Description
Hello,
We have self hosted hosts servers with multiple containers running a single instance of buildkite-agent. These servers are unclustered agents in Buildkite (not sure if relevant, but wanted to add that detail too).
I have noticed that the buildkite-agent process spawns a high number of threads that are not properly cleaned up/pruned. I imagine this is a thread pool, but it grows a lot, with some containers agents' reaching 150 threads.
We are running buildkite-agent with a single agent spawned per container (i.e, not using the --spawn option), so why do we have so many threads on that process? Each agent should only run one job at a time, or have a cap on the thread pool per agent.
At the same time, these agents and threads are using an enormous amount of memory. This looks like a memory leak, but I havent' had the chance to monitor it over a couple days. I see that processes go back and forth in used memory, but they will never return to the initial value if they are idle for a while. (Processes are sleeping, not zombie)
eg, an idle buildkite agent process might have 100+ spawned thread and allocate 60gb in memory to those threads.
These bots have been running for 6 days, the numbers: 👇
Number of threads per server (agg)
running ps -L -C buildkite-agent | wc -l | sort -k2,2nr across our containers. You can see that we have agents that have 150, but most of them have between 80 and 100 processes.
(the left column is the number of servers who share the number of threads on the right column)
1 152
1 147
1 145
1 144
3 140
1 138
4 137
2 136
1 135
1 134
1 129
1 127
3 126
2 125
1 124
3 122
4 120
2 119
4 118
2 117
3 116
4 115
4 114
7 113
8 112
9 111
8 110
8 109
11 108
10 107
4 106
6 105
6 104
15 103
8 102
10 101
9 100
29 99
20 98
12 97
16 96
13 95
15 94
16 93
25 92
19 91
15 90
16 89
11 88
11 87
17 86
7 85
6 84
8 83
5 82
2 81
2 80
1 78
1 77
1 76
2 74
1 73
3 72
2 70
4 69
1 68
2 67
3 66
1 65
1 64
2 63
5 62
2 61
3 60
2 59
3 58
1 57
1 56
7 55
8 54
4 53
7 52
2 51
7 50
3 49
1 48
7 47
4 46
3 45
1 43
2 42
1 41
1 40
1 36
memory usage numbers
top usage is 92.8gb 😳
sc status buildbot.service | grep -i memory
[0] [SUCCESS] (2s) buildbot100
Memory: 22.7G
[1] [SUCCESS] (2s) buildbot101
Memory: 62.3G
[2] [SUCCESS] (2s) buildbot102
Memory: 48.3G
[3] [SUCCESS] (2s) buildbot103
Memory: 16.1G
[4] [SUCCESS] (2s) buildbot104
Memory: 27.5G
[5] [SUCCESS] (2s) buildbot105
Memory: 67.8G
[6] [SUCCESS] (2s) buildbot106
Memory: 37.9G
[7] [SUCCESS] (2s) buildbot107
Memory: 43.4G
[8] [SUCCESS] (2s) buildbot108
Memory: 62.0G
[9] [SUCCESS] (2s) buildbot109
Memory: 33.9G
[10] [SUCCESS] (2s) buildbot110
Memory: 72.5G
[11] [SUCCESS] (2s) buildbot111
Memory: 40.5G
[12] [SUCCESS] (2s) buildbot112
Memory: 31.5G
[13] [SUCCESS] (2s) buildbot113
Memory: 31.2G
[14] [SUCCESS] (2s) buildbot114
Memory: 32.3G
[15] [SUCCESS] (2s) buildbot115
Memory: 23.5G
[16] [SUCCESS] (2s) buildbot116
Memory: 75.2G
[17] [SUCCESS] (2s) buildbot117
Memory: 60.8G
[18] [SUCCESS] (2s) buildbot118
Memory: 33.9G
[19] [SUCCESS] (2s) buildbot119
Memory: 37.2G
[20] [SUCCESS] (2s) buildbot120
Memory: 39.1G
[21] [SUCCESS] (2s) buildbot121
Memory: 24.2G
[22] [SUCCESS] (2s) buildbot122
Memory: 13.2G
[23] [SUCCESS] (2s) buildbot123
Memory: 20.0G
[24] [SUCCESS] (2s) buildbot124
Memory: 38.6G
[25] [SUCCESS] (2s) buildbot125
Memory: 40.1G
[26] [SUCCESS] (2s) buildbot126
Memory: 43.8G
[27] [SUCCESS] (2s) buildbot127
Memory: 18.8G
[28] [SUCCESS] (2s) buildbot128
Memory: 50.0G
[29] [SUCCESS] (2s) buildbot129
Memory: 30.3G
[30] [SUCCESS] (2s) buildbot130
Memory: 28.7G
[31] [SUCCESS] (2s) buildbot131
Memory: 20.1G
[32] [SUCCESS] (2s) buildbot132
Memory: 12.9G
[33] [SUCCESS] (2s) buildbot133
Memory: 13.2G
[34] [SUCCESS] (2s) buildbot134
Memory: 25.5G
[35] [SUCCESS] (2s) buildbot135
Memory: 76.5G
[36] [SUCCESS] (2s) buildbot136
Memory: 39.7G
[37] [SUCCESS] (2s) buildbot137
Memory: 62.2G
[38] [SUCCESS] (2s) buildbot138
Memory: 21.7G
[39] [SUCCESS] (2s) buildbot139
Memory: 59.8G
[40] [SUCCESS] (2s) buildbot200
Memory: 4.6G
[41] [SUCCESS] (2s) buildbot201
Memory: 35.9G
[42] [SUCCESS] (2s) buildbot202
Memory: 5.0G
[43] [SUCCESS] (2s) buildbot203
Memory: 24.4G
[44] [SUCCESS] (2s) buildbot204
Memory: 1.8G
[45] [SUCCESS] (2s) buildbot205
Memory: 39.0G
[46] [SUCCESS] (2s) buildbot206
Memory: 4.9G
[47] [SUCCESS] (2s) buildbot207
Memory: 1.2G
[48] [SUCCESS] (2s) buildbot208
Memory: 1.8G
[49] [SUCCESS] (2s) buildbot209
Memory: 11.2G
[50] [SUCCESS] (2s) buildbot210
Memory: 7.7G
[51] [SUCCESS] (2s) buildbot211
Memory: 9.8G
[52] [SUCCESS] (2s) buildbot212
Memory: 10.2G
[53] [SUCCESS] (2s) buildbot213
Memory: 16.7G
[54] [SUCCESS] (2s) buildbot214
Memory: 13.6G
[55] [SUCCESS] (2s) buildbot215
Memory: 5.1G
[56] [SUCCESS] (2s) buildbot216
Memory: 33.7G
[57] [SUCCESS] (2s) buildbot217
Memory: 12.7G
[58] [SUCCESS] (2s) buildbot218
Memory: 14.4G
[59] [SUCCESS] (2s) buildbot219
Memory: 15.9G
[60] [SUCCESS] (2s) buildbot220
Memory: 20.5G
[61] [SUCCESS] (2s) buildbot221
Memory: 14.5G
[62] [SUCCESS] (2s) buildbot222
Memory: 80.4G
[63] [SUCCESS] (2s) buildbot223
Memory: 17.7G
[64] [SUCCESS] (2s) buildbot224
Memory: 20.9G
[65] [SUCCESS] (2s) buildbot225
Memory: 6.6G
[66] [SUCCESS] (2s) buildbot226
Memory: 9.5G
[67] [SUCCESS] (2s) buildbot227
Memory: 17.0G
[68] [SUCCESS] (2s) buildbot228
Memory: 22.5G
[69] [SUCCESS] (2s) buildbot229
Memory: 19.5G
[70] [SUCCESS] (2s) buildbot230
Memory: 8.8G
[71] [SUCCESS] (2s) buildbot231
Memory: 13.5G
[72] [SUCCESS] (2s) buildbot232
Memory: 21.7G
[73] [SUCCESS] (2s) buildbot233
Memory: 16.1G
[74] [SUCCESS] (2s) buildbot234
Memory: 11.5G
[75] [SUCCESS] (2s) buildbot235
Memory: 13.4G
[76] [SUCCESS] (2s) buildbot236
Memory: 14.0G
[77] [SUCCESS] (2s) buildbot237
Memory: 15.5G
[78] [SUCCESS] (2s) buildbot238
Memory: 9.6G
[79] [SUCCESS] (2s) buildbot239
Memory: 8.1G
[80] [SUCCESS] (2s) buildbot240
Memory: 27.9G
[81] [SUCCESS] (2s) buildbot241
Memory: 6.1G
[82] [SUCCESS] (2s) buildbot242
Memory: 4.9G
[83] [SUCCESS] (2s) buildbot243
Memory: 11.3G
[84] [SUCCESS] (2s) buildbot244
Memory: 19.3G
[85] [SUCCESS] (2s) buildbot245
Memory: 9.9G
[86] [SUCCESS] (2s) buildbot246
Memory: 37.7G
[87] [SUCCESS] (2s) buildbot247
Memory: 7.7G
[88] [SUCCESS] (2s) buildbot248
Memory: 5.7G
[89] [SUCCESS] (2s) buildbot249
Memory: 13.7G
[90] [SUCCESS] (2s) buildbot250
Memory: 11.8G
[91] [SUCCESS] (2s) buildbot251
Memory: 25.3G
[92] [SUCCESS] (2s) buildbot252
Memory: 8.7G
[93] [SUCCESS] (2s) buildbot253
Memory: 1.4G
[94] [SUCCESS] (2s) buildbot254
Memory: 5.4G
[95] [SUCCESS] (2s) buildbot255
Memory: 1.2G
[96] [SUCCESS] (2s) buildbot256
Memory: 4.5G
[97] [SUCCESS] (2s) buildbot257
Memory: 14.4G
[98] [SUCCESS] (2s) buildbot258
Memory: 8.8G
[99] [SUCCESS] (2s) buildbot259
Memory: 8.6G
stderr: Control socket connect(/home/me/.ssh/ahrefs/controls/team@168.100.147.94:22061): Connection refused
[100] [SUCCESS] (2s) buildbot260
Memory: 2.1G
[101] [SUCCESS] (2s) buildbot261
Memory: 13.0G
[102] [SUCCESS] (2s) buildbot262
Memory: 4.7G
[103] [SUCCESS] (2s) buildbot263
Memory: 24.9G
[104] [SUCCESS] (2s) buildbot264
Memory: 59.2G
[105] [SUCCESS] (2s) buildbot265
Memory: 17.5G
[106] [SUCCESS] (2s) buildbot266
Memory: 9.2G
[107] [SUCCESS] (2s) buildbot267
Memory: 70.1G
[108] [SUCCESS] (2s) buildbot268
Memory: 5.2G
[109] [SUCCESS] (2s) buildbot269
Memory: 2.8G
[110] [SUCCESS] (2s) buildbot400
Memory: 17.6G
[111] [SUCCESS] (2s) buildbot401
Memory: 9.3G
[112] [SUCCESS] (2s) buildbot402
Memory: 18.1G
[113] [SUCCESS] (2s) buildbot403
Memory: 13.8G
[114] [SUCCESS] (2s) buildbot404
Memory: 16.3G
[115] [SUCCESS] (2s) buildbot405
Memory: 11.2G
[116] [SUCCESS] (2s) buildbot406
Memory: 34.4G
[117] [SUCCESS] (2s) buildbot407
Memory: 23.8G
[118] [SUCCESS] (2s) buildbot408
Memory: 13.8G
[119] [SUCCESS] (2s) buildbot409
Memory: 23.0G
[120] [SUCCESS] (2s) buildbot410
Memory: 33.1G
[121] [SUCCESS] (2s) buildbot411
Memory: 8.6G
[122] [SUCCESS] (2s) buildbot412
Memory: 11.5G
[123] [SUCCESS] (2s) buildbot413
Memory: 28.9G
[124] [SUCCESS] (2s) buildbot414
Memory: 27.3G
[125] [SUCCESS] (2s) buildbot415
Memory: 20.3G
[126] [SUCCESS] (2s) buildbot416
Memory: 33.1G
[127] [SUCCESS] (2s) buildbot417
Memory: 10.5G
[128] [SUCCESS] (2s) buildbot418
Memory: 19.5G
[129] [SUCCESS] (2s) buildbot419
Memory: 36.0G
[130] [SUCCESS] (2s) buildbot420
Memory: 14.9G
[131] [SUCCESS] (2s) buildbot421
Memory: 14.9G
[132] [SUCCESS] (2s) buildbot422
Memory: 18.1G
[133] [SUCCESS] (2s) buildbot423
Memory: 59.9G
[134] [SUCCESS] (2s) buildbot424
Memory: 24.8G
[135] [SUCCESS] (2s) buildbot425
Memory: 27.6G
[136] [SUCCESS] (2s) buildbot426
Memory: 10.2G
[137] [SUCCESS] (2s) buildbot427
Memory: 36.5G
[138] [SUCCESS] (2s) buildbot428
Memory: 33.0G
[139] [SUCCESS] (2s) buildbot429
Memory: 23.0G
[140] [SUCCESS] (2s) buildbot430
Memory: 22.0G
[141] [SUCCESS] (2s) buildbot431
Memory: 23.0G
[142] [SUCCESS] (2s) buildbot432
Memory: 36.1G
[143] [SUCCESS] (2s) buildbot433
Memory: 10.1G
[144] [SUCCESS] (2s) buildbot434
Memory: 15.1G
[145] [SUCCESS] (2s) buildbot435
Memory: 25.4G
[146] [SUCCESS] (2s) buildbot436
Memory: 7.2G
[147] [SUCCESS] (2s) buildbot437
Memory: 11.8G
[148] [SUCCESS] (2s) buildbot438
Memory: 36.7G
[149] [SUCCESS] (2s) buildbot439
Memory: 13.1G
[150] [SUCCESS] (2s) buildbot440
Memory: 16.0G
[151] [SUCCESS] (2s) buildbot441
Memory: 49.2G
[152] [SUCCESS] (2s) buildbot442
Memory: 12.2G
[153] [SUCCESS] (2s) buildbot443
Memory: 34.9G
[154] [SUCCESS] (2s) buildbot444
Memory: 13.3G
[155] [SUCCESS] (2s) buildbot445
Memory: 17.4G
[156] [SUCCESS] (2s) buildbot446
Memory: 8.6G
[157] [SUCCESS] (2s) buildbot447
Memory: 32.2G
[158] [SUCCESS] (2s) buildbot448
Memory: 15.2G
[159] [SUCCESS] (2s) buildbot449
Memory: 5.3G
[160] [SUCCESS] (2s) buildbot450
Memory: 9.7G
[161] [SUCCESS] (2s) buildbot451
Memory: 8.5G
[162] [SUCCESS] (2s) buildbot452
Memory: 31.0G
[163] [SUCCESS] (2s) buildbot453
Memory: 13.8G
[164] [SUCCESS] (2s) buildbot454
Memory: 27.3G
[165] [SUCCESS] (2s) buildbot455
Memory: 9.9G
[166] [SUCCESS] (2s) buildbot456
Memory: 11.5G
[167] [SUCCESS] (2s) buildbot457
Memory: 42.0G
[168] [SUCCESS] (2s) buildbot458
Memory: 12.7G
[169] [SUCCESS] (2s) buildbot459
Memory: 40.3G
[170] [SUCCESS] (2s) buildbot460
Memory: 12.9G
[171] [SUCCESS] (2s) buildbot461
Memory: 28.6G
[172] [SUCCESS] (2s) buildbot462
Memory: 10.8G
[173] [SUCCESS] (2s) buildbot463
Memory: 33.0G
[174] [SUCCESS] (2s) buildbot464
Memory: 7.2G
[175] [SUCCESS] (2s) buildbot465
Memory: 65.1G
[176] [SUCCESS] (2s) buildbot466
Memory: 33.6G
[177] [SUCCESS] (2s) buildbot467
Memory: 10.5G
[178] [SUCCESS] (2s) buildbot468
Memory: 28.7G
[179] [SUCCESS] (2s) buildbot469
Memory: 22.2G
[180] [SUCCESS] (2s) buildbot500
Memory: 13.0G
[181] [SUCCESS] (2s) buildbot501
Memory: 8.5G
[182] [SUCCESS] (2s) buildbot502
Memory: 21.7G
[183] [SUCCESS] (2s) buildbot503
Memory: 20.0G
[184] [SUCCESS] (2s) buildbot504
Memory: 33.0G
[185] [SUCCESS] (2s) buildbot505
Memory: 26.0G
[186] [SUCCESS] (2s) buildbot506
Memory: 10.6G
stderr: Control socket connect(/home/me/.ssh/ahrefs/controls/team@168.100.148.115:22008): Connection refused
[187] [SUCCESS] (2s) buildbot507
Memory: 35.8G
[188] [SUCCESS] (2s) buildbot508
Memory: 6.6G
[189] [SUCCESS] (2s) buildbot509
Memory: 68.1G
[190] [SUCCESS] (2s) buildbot510
Memory: 11.6G
[191] [SUCCESS] (2s) buildbot511
Memory: 27.0G
[192] [SUCCESS] (2s) buildbot512
Memory: 40.4G
[193] [SUCCESS] (2s) buildbot513
Memory: 10.0G
[194] [SUCCESS] (2s) buildbot514
Memory: 55.9G
[195] [SUCCESS] (2s) buildbot515
Memory: 8.6G
[196] [SUCCESS] (2s) buildbot516
Memory: 34.4G
[197] [SUCCESS] (2s) buildbot517
Memory: 55.3G
[198] [SUCCESS] (2s) buildbot518
Memory: 21.1G
[199] [SUCCESS] (2s) buildbot519
Memory: 64.6G
[200] [SUCCESS] (2s) buildbot520
Memory: 46.5G
[201] [SUCCESS] (2s) buildbot521
Memory: 30.7G
[202] [SUCCESS] (2s) buildbot522
Memory: 9.8G
[203] [SUCCESS] (2s) buildbot523
Memory: 13.0G
[204] [SUCCESS] (2s) buildbot524
Memory: 33.0G
[205] [SUCCESS] (2s) buildbot525
Memory: 11.7G
[206] [SUCCESS] (2s) buildbot526
Memory: 34.7G
[207] [SUCCESS] (2s) buildbot527
Memory: 16.4G
[208] [SUCCESS] (2s) buildbot528
Memory: 67.7G
[209] [SUCCESS] (2s) buildbot529
Memory: 13.3G
[210] [SUCCESS] (2s) buildbot530
Memory: 12.4G
[211] [SUCCESS] (2s) buildbot531
Memory: 8.6G
[212] [SUCCESS] (2s) buildbot533
Memory: 7.7G
[213] [SUCCESS] (2s) buildbot535
Memory: 17.3G
[214] [SUCCESS] (2s) buildbot538
Memory: 42.0G
[215] [SUCCESS] (2s) buildbot541
Memory: 81.9G
stderr: Control socket connect(/home/me/.ssh/ahrefs/controls/team@168.100.148.115:22043): Connection refused
[216] [SUCCESS] (3s) buildbot532
Memory: 39.4G
[217] [SUCCESS] (3s) buildbot534
Memory: 10.8G
[218] [SUCCESS] (3s) buildbot536
Memory: 12.7G
[219] [SUCCESS] (3s) buildbot537
Memory: 55.4G
[220] [SUCCESS] (3s) buildbot539
Memory: 48.7G
[221] [SUCCESS] (3s) buildbot540
Memory: 26.2G
[222] [SUCCESS] (3s) buildbot542
Memory: 7.8G
stderr: Control socket connect(/home/me/.ssh/ahrefs/controls/team@168.100.148.115:22044): Connection refused
[223] [SUCCESS] (3s) buildbot543
Memory: 27.7G
[224] [SUCCESS] (3s) buildbot544
Memory: 30.4G
[225] [SUCCESS] (3s) buildbot545
Memory: 26.5G
[226] [SUCCESS] (3s) buildbot546
Memory: 11.2G
[227] [SUCCESS] (3s) buildbot547
Memory: 28.7G
[228] [SUCCESS] (3s) buildbot548
Memory: 15.8G
[229] [SUCCESS] (3s) buildbot549
Memory: 25.5G
[230] [SUCCESS] (3s) buildbot550
Memory: 9.9G
[231] [SUCCESS] (3s) buildbot551
Memory: 7.7G
[232] [SUCCESS] (3s) buildbot552
Memory: 52.3G
[233] [SUCCESS] (3s) buildbot553
Memory: 50.9G
[234] [SUCCESS] (3s) buildbot554
Memory: 11.7G
[235] [SUCCESS] (3s) buildbot555
Memory: 8.7G
[236] [SUCCESS] (3s) buildbot556
Memory: 29.7G
[237] [SUCCESS] (3s) buildbot557
Memory: 11.4G
[238] [SUCCESS] (3s) buildbot558
Memory: 49.4G
[239] [SUCCESS] (3s) buildbot559
Memory: 13.9G
[240] [SUCCESS] (3s) buildbot560
Memory: 10.1G
[241] [SUCCESS] (3s) buildbot561
Memory: 9.8G
[242] [SUCCESS] (3s) buildbot562
Memory: 7.0G
[243] [SUCCESS] (3s) buildbot563
Memory: 6.3G
[244] [SUCCESS] (3s) buildbot564
Memory: 11.8G
[245] [SUCCESS] (3s) buildbot565
Memory: 5.4G
[246] [SUCCESS] (3s) buildbot566
Memory: 55.5G
[247] [SUCCESS] (3s) buildbot567
Memory: 9.3G
[248] [SUCCESS] (3s) buildbot568
Memory: 43.2G
[249] [SUCCESS] (3s) buildbot569
Memory: 77.5G
[250] [SUCCESS] (3s) buildbot798
Memory: 17.4G
[251] [SUCCESS] (3s) buildbot996
Memory: 1.6G
[252] [SUCCESS] (4s) buildbot600
Memory: 9.6G
[253] [SUCCESS] (4s) buildbot601
Memory: 11.4G
[254] [SUCCESS] (4s) buildbot602
Memory: 7.6G
[255] [SUCCESS] (4s) buildbot603
Memory: 32.9G
[256] [SUCCESS] (4s) buildbot604
Memory: 49.5G
[257] [SUCCESS] (4s) buildbot605
Memory: 7.2G
[258] [SUCCESS] (4s) buildbot606
Memory: 6.3G
[259] [SUCCESS] (4s) buildbot607
Memory: 10.6G
[260] [SUCCESS] (4s) buildbot608
Memory: 33.0G
[261] [SUCCESS] (4s) buildbot609
Memory: 11.9G
[262] [SUCCESS] (4s) buildbot610
Memory: 68.1G
[263] [SUCCESS] (4s) buildbot611
Memory: 10.6G
[264] [SUCCESS] (4s) buildbot612
Memory: 25.3G
[265] [SUCCESS] (4s) buildbot613
Memory: 34.2G
[266] [SUCCESS] (4s) buildbot614
Memory: 43.7G
[267] [SUCCESS] (4s) buildbot615
Memory: 50.7G
[268] [SUCCESS] (4s) buildbot616
Memory: 27.2G
[269] [SUCCESS] (4s) buildbot617
Memory: 78.8G
[270] [SUCCESS] (4s) buildbot618
Memory: 18.4G
[271] [SUCCESS] (4s) buildbot619
Memory: 15.6G
[272] [SUCCESS] (4s) buildbot620
Memory: 22.2G
[273] [SUCCESS] (4s) buildbot621
Memory: 41.0G
[274] [SUCCESS] (4s) buildbot622
Memory: 20.4G
[275] [SUCCESS] (4s) buildbot623
Memory: 44.0G
[276] [SUCCESS] (4s) buildbot624
Memory: 16.7G
[277] [SUCCESS] (4s) buildbot625
Memory: 48.8G
[278] [SUCCESS] (4s) buildbot626
Memory: 16.4G
[279] [SUCCESS] (4s) buildbot627
Memory: 13.2G
[280] [SUCCESS] (4s) buildbot628
Memory: 20.0G
[281] [SUCCESS] (4s) buildbot629
Memory: 48.4G
[282] [SUCCESS] (4s) buildbot630
Memory: 31.2G
[283] [SUCCESS] (4s) buildbot631
Memory: 64.4G
[284] [SUCCESS] (4s) buildbot632
Memory: 10.0G
[285] [SUCCESS] (4s) buildbot633
Memory: 45.0G
[286] [SUCCESS] (4s) buildbot634
Memory: 33.8G
[287] [SUCCESS] (4s) buildbot635
Memory: 14.7G
[288] [SUCCESS] (4s) buildbot636
Memory: 10.8G
[289] [SUCCESS] (4s) buildbot637
Memory: 48.7G
[290] [SUCCESS] (4s) buildbot638
Memory: 30.5G
[291] [SUCCESS] (4s) buildbot639
Memory: 10.1G
[292] [SUCCESS] (4s) buildbot640
Memory: 13.3G
[293] [SUCCESS] (4s) buildbot641
Memory: 9.7G
[294] [SUCCESS] (4s) buildbot642
Memory: 47.6G
[295] [SUCCESS] (4s) buildbot643
Memory: 24.6G
[296] [SUCCESS] (4s) buildbot644
Memory: 77.7G
[297] [SUCCESS] (4s) buildbot645
Memory: 16.7G
[298] [SUCCESS] (4s) buildbot646
Memory: 57.4G
[299] [SUCCESS] (4s) buildbot647
Memory: 38.3G
[300] [SUCCESS] (4s) buildbot648
Memory: 7.9G
[301] [SUCCESS] (4s) buildbot649
Memory: 17.4G
[302] [SUCCESS] (4s) buildbot650
Memory: 52.6G
[303] [SUCCESS] (4s) buildbot651
Memory: 7.2G
[304] [SUCCESS] (4s) buildbot652
Memory: 59.4G
[305] [SUCCESS] (4s) buildbot653
Memory: 27.3G
[306] [SUCCESS] (4s) buildbot654
Memory: 6.7G
[307] [SUCCESS] (4s) buildbot655
Memory: 10.8G
[308] [SUCCESS] (4s) buildbot656
Memory: 38.5G
[309] [SUCCESS] (4s) buildbot657
Memory: 36.9G
[310] [SUCCESS] (4s) buildbot658
Memory: 18.9G
[311] [SUCCESS] (4s) buildbot659
Memory: 28.3G
[312] [SUCCESS] (4s) buildbot660
Memory: 7.7G
[313] [SUCCESS] (4s) buildbot661
Memory: 9.4G
[314] [SUCCESS] (4s) buildbot662
Memory: 12.9G
[315] [SUCCESS] (4s) buildbot663
Memory: 14.3G
[316] [SUCCESS] (4s) buildbot664
Memory: 10.6G
[317] [SUCCESS] (4s) buildbot665
Memory: 17.1G
[318] [SUCCESS] (4s) buildbot666
Memory: 6.6G
[319] [SUCCESS] (4s) buildbot667
Memory: 25.8G
[320] [SUCCESS] (4s) buildbot668
Memory: 7.9G
[321] [SUCCESS] (4s) buildbot669
Memory: 35.7G
[322] [SUCCESS] (4s) buildbot700
Memory: 3.5G
[323] [SUCCESS] (4s) buildbot705
Memory: 44.0G
[324] [SUCCESS] (4s) buildbot995
Memory: 1.4G
[325] [SUCCESS] (4s) buildbot997
Memory: 1.5G
[326] [SUCCESS] (4s) buildbot998
Memory: 2.0G
[327] [SUCCESS] (4s) buildbot999
Memory: 1.0G
[328] [SUCCESS] (5s) buildbot709
Memory: 21.6G
[329] [SUCCESS] (5s) buildbot747
Memory: 16.0G
[330] [SUCCESS] (5s) buildbot757
Memory: 37.4G
[331] [SUCCESS] (5s) buildbot762
Memory: 7.2G
[332] [SUCCESS] (5s) buildbot785
Memory: 24.2G
[333] [SUCCESS] (5s) buildbot799
Memory: 5.5G
[334] [SUCCESS] (6s) buildbot701
Memory: 4.4G
[335] [SUCCESS] (6s) buildbot714
Memory: 24.0G
[336] [SUCCESS] (6s) buildbot719
Memory: 52.7G
[337] [SUCCESS] (6s) buildbot722
Memory: 8.5G
[338] [SUCCESS] (6s) buildbot733
Memory: 35.5G
[339] [SUCCESS] (6s) buildbot740
Memory: 20.1G
[340] [SUCCESS] (6s) buildbot745
Memory: 26.1G
[341] [SUCCESS] (6s) buildbot751
Memory: 9.0G
[342] [SUCCESS] (6s) buildbot752
Memory: 11.9G
[343] [SUCCESS] (6s) buildbot772
Memory: 24.6G
[344] [SUCCESS] (7s) buildbot702
Memory: 40.2G
[345] [SUCCESS] (7s) buildbot704
Memory: 9.5G
[346] [SUCCESS] (7s) buildbot706
Memory: 22.2G
[347] [SUCCESS] (7s) buildbot707
Memory: 5.1G
[348] [SUCCESS] (7s) buildbot708
Memory: 73.6G
[349] [SUCCESS] (7s) buildbot710
Memory: 7.2G
[350] [SUCCESS] (7s) buildbot711
Memory: 6.6G
[351] [SUCCESS] (7s) buildbot712
Memory: 3.6G
[352] [SUCCESS] (7s) buildbot713
Memory: 2.8G
[353] [SUCCESS] (7s) buildbot715
Memory: 166.4M
[354] [SUCCESS] (7s) buildbot716
Memory: 3.7G
[355] [SUCCESS] (7s) buildbot717
Memory: 13.4G
[356] [SUCCESS] (7s) buildbot718
Memory: 110.1G
[357] [SUCCESS] (7s) buildbot720
Memory: 17.1G
[358] [SUCCESS] (7s) buildbot723
Memory: 23.6G
[359] [SUCCESS] (7s) buildbot725
Memory: 24.1G
[360] [SUCCESS] (7s) buildbot726
Memory: 9.1G
[361] [SUCCESS] (7s) buildbot728
Memory: 12.5G
[362] [SUCCESS] (7s) buildbot729
Memory: 14.9G
[363] [SUCCESS] (7s) buildbot731
Memory: 12.7G
[364] [SUCCESS] (7s) buildbot732
Memory: 7.6G
[365] [SUCCESS] (7s) buildbot735
Memory: 6.9G
[366] [SUCCESS] (7s) buildbot736
Memory: 24.4G
[367] [SUCCESS] (7s) buildbot737
Memory: 11.3G
[368] [SUCCESS] (7s) buildbot738
Memory: 38.6G
[369] [SUCCESS] (7s) buildbot739
Memory: 9.8G
[370] [SUCCESS] (7s) buildbot741
Memory: 12.1G
[371] [SUCCESS] (7s) buildbot742
Memory: 14.1G
[372] [SUCCESS] (7s) buildbot743
Memory: 14.2G
[373] [SUCCESS] (7s) buildbot744
Memory: 45.8G
[374] [SUCCESS] (7s) buildbot746
Memory: 18.3G
[375] [SUCCESS] (7s) buildbot750
Memory: 22.1G
[376] [SUCCESS] (7s) buildbot753
Memory: 8.8G
[377] [SUCCESS] (7s) buildbot754
Memory: 3.7G
[378] [SUCCESS] (7s) buildbot755
Memory: 6.0G
[379] [SUCCESS] (7s) buildbot756
Memory: 6.6G
[380] [SUCCESS] (7s) buildbot758
Memory: 31.4G
[381] [SUCCESS] (7s) buildbot759
Memory: 8.1G
[382] [SUCCESS] (7s) buildbot761
Memory: 8.5G
[383] [SUCCESS] (7s) buildbot763
Memory: 5.8G
[384] [SUCCESS] (7s) buildbot765
Memory: 10.0G
[385] [SUCCESS] (7s) buildbot767
Memory: 6.1G
[386] [SUCCESS] (7s) buildbot769
Memory: 2.8G
[387] [SUCCESS] (7s) buildbot770
Memory: 30.5G
[388] [SUCCESS] (7s) buildbot771
Memory: 7.2G
[389] [SUCCESS] (7s) buildbot773
Memory: 6.8G
[390] [SUCCESS] (7s) buildbot774
Memory: 3.0G
[391] [SUCCESS] (7s) buildbot775
Memory: 2.8G
[392] [SUCCESS] (7s) buildbot776
Memory: 8.3G
[393] [SUCCESS] (7s) buildbot777
Memory: 8.9G
[394] [SUCCESS] (7s) buildbot778
Memory: 8.0G
[395] [SUCCESS] (7s) buildbot779
Memory: 4.0G
[396] [SUCCESS] (7s) buildbot780
Memory: 4.6G
[397] [SUCCESS] (7s) buildbot781
Memory: 5.0G
[398] [SUCCESS] (7s) buildbot782
Memory: 3.5G
[399] [SUCCESS] (7s) buildbot783
Memory: 4.4G
[400] [SUCCESS] (7s) buildbot784
Memory: 8.2G
[401] [SUCCESS] (7s) buildbot786
Memory: 4.1G
[402] [SUCCESS] (7s) buildbot787
Memory: 15.0G
[403] [SUCCESS] (7s) buildbot788
Memory: 4.4G
[404] [SUCCESS] (7s) buildbot789
Memory: 8.8G
[405] [SUCCESS] (7s) buildbot790
Memory: 14.8G
[406] [SUCCESS] (7s) buildbot791
Memory: 6.2G
[407] [SUCCESS] (7s) buildbot792
Memory: 31.1G
[408] [SUCCESS] (7s) buildbot793
Memory: 14.5G
[409] [SUCCESS] (7s) buildbot794
Memory: 25.8G
[410] [SUCCESS] (7s) buildbot795
Memory: 9.4G
[411] [SUCCESS] (7s) buildbot796
Memory: 9.6G
[412] [SUCCESS] (7s) buildbot797
Memory: 10.9G
[413] [SUCCESS] (8s) buildbot703
Memory: 10.2G
[414] [SUCCESS] (8s) buildbot721
Memory: 22.6G
[415] [SUCCESS] (8s) buildbot724
Memory: 10.7G
[416] [SUCCESS] (8s) buildbot727
Memory: 30.0G
[417] [SUCCESS] (8s) buildbot730
Memory: 6.8G
[418] [SUCCESS] (8s) buildbot734
Memory: 12.2G
[419] [SUCCESS] (8s) buildbot748
Memory: 66.7G
[420] [SUCCESS] (8s) buildbot749
Memory: 43.3G
[421] [SUCCESS] (8s) buildbot760
Memory: 8.2G
[422] [SUCCESS] (8s) buildbot764
Memory: 5.6G
[423] [SUCCESS] (8s) buildbot766
Memory: 9.5G
[424] [SUCCESS] (8s) buildbot768
Memory: 1.9G
(8s) Done 425 out of 425
Is this a known issue? i didn't find any mentions to this, so I'm curious to know if this is expected or if there is a different way to handle these self hosted clusters.
Thanks in advance for your help