Add step in SNP pipeline to run CTest

[DomAyre]

Contributes to #4190

Separate test fixes which have been spun out into their own PR's

• Handle race between index construction and e2e test #4803
• Run join_old_snapshot test outside after other network cleanup #4805
• Fix running SNP pipeline against PR manually #4808

Other changes which I decided to keep in this PR because they didn't make sense outside of it:

• Support running tests on a files-share
  • Make workspace configurable by environment variable to facilitate changing it for the SNP testing run
  • Change some symlinking operations to copies as running on a files-share doesn't support symlinks
• Increase timeouts in places where slow IO was triggering IO errors that I could see in runs
  • connections_test create_connections_until_exhaustion from 1 second to 3
  • get_all_entries timeout in e2e_logging_cft test from 5 seconds to 10
  • Election timeout from 4 seconds to 10

[ghost]

run-snp-ctest@60174 aka 20230111.9 vs main ewma over 20 builds from 59503 to 60167

Click to see table

main

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	ls_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_virtual_cft^	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	pi_ls_jwt_virtual_cft^	ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_js_jwt_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	hist_sgx_cft^	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
59503	20230103.14	0.807001	17176.2	41838.3	6322.06	8.24484e+07	15654.4	1.53396e+07	46279.6	15621.7	9.31027e+06	13081.9	12221.1	4529.19	3530.7	3399.76	5928.6	6.68883e+06	5844.27	1.50774e+07	22258.1	1571.32	7.73741e+06	1293.52	7.47526e+06	1297.74	6.95098e+06	835260	1.17714e+06	8.17151e+06	3.01989e+07
59552	20230103.25	0.793474	17049.5	43539.9	6274.88	8.1662e+07	15453	1.50774e+07	45301.4	15583.5	8.52384e+06	13366.6	12521.6	4665.5	3550.42	3420.06	5845.5	6.68883e+06	5841.39	1.50774e+07	24220.9	1570.79	7.73741e+06	1288.84	7.73741e+06	1300.04	6.95098e+06	838063	1.17793e+06	8.04055e+06	3.07351e+07
59578	20230103.30	0.787714	16790.8	45826.5	6317.76	8.21863e+07	15729.5	1.53396e+07	45660	15625.5	9.31027e+06	13368.6	12472.2	4653.88	3659.56	3324.89	5808.9	6.68883e+06	5816.03	1.50774e+07	23206.6	1573.11	7.73741e+06	1289.07	7.21312e+06	1301.95	6.95098e+06	843043	1.18213e+06	8.15092e+06	3.08243e+07
59588	20230104.3	0.791543	17095	45701.8	6282.06	8.1662e+07	15431.4	1.50774e+07	47455.6	15613.9	8.78598e+06	13374.6	12556.1	4670.34	3701.22	3325.31	5853.7	6.68883e+06	5852	1.48153e+07	24152.6	1571.33	7.73741e+06	1283.97	7.73741e+06	1296.71	6.95098e+06	831776	1.18216e+06	8.17317e+06	3.16059e+07
59639	20230104.19	0.793442	17099	43700.9	6317.48	8.19242e+07	15711.9	1.53396e+07	47658.5	15654	8.78598e+06	13311.4	12386.3	4631.54	3538.75	3445.45	5888.9	6.95098e+06	5834.29	1.50774e+07	27936.5	1565.38	7.73741e+06	1289.07	7.73741e+06	1297.73	6.95098e+06	825987	1.18309e+06	8.15566e+06	3.10906e+07
59665	20230105.2	0.79042	17170.6	43827.9	6325.77	8.21863e+07	15559.4	1.53396e+07	47798.4	15682.3	8.78598e+06	13342.3	14174.5	4513.65	3622.04	3476.38	5861.5	6.68883e+06	5834.47	1.50774e+07	22620.3	1570.57	7.73741e+06	1284.7	7.73741e+06	1315.74	6.95098e+06	837979	1.17992e+06	8.17323e+06	3.07162e+07
59678	20230105.6	0.79786	17180.4	43679.2	6287.89	8.24484e+07	15447.8	1.53396e+07	44918.6	15613.9	8.78598e+06	13323.5	12415.2	4590.03	3582.73	3448.03	5818	6.68883e+06	5834.47	1.50774e+07	27185.3	1571.65	7.73741e+06	1284.57	7.73741e+06	1300.41	6.95098e+06	818802	1.15908e+06	8.14269e+06	3.08531e+07
59719	20230105.16	0.816841	17292	43799.1	6305.07	8.24484e+07	15717.1	1.53396e+07	47855.8	15978.5	8.2617e+06	13712.7	12652	4681.08	3583.49	3465.9	5810.9	4.59168e+06	5800.21	1.50774e+07	23815.7	1572.4	7.73741e+06	1293.32	7.47526e+06	1309.53	6.95098e+06	832333	1.17719e+06	8.15582e+06	3.13903e+07
59747	20230105.22	0.794968	17255.8	43731.7	6282.56	8.21863e+07	15541.7	1.50774e+07	48926.1	15864.1	8.2617e+06	12965.6	12297.4	4606.8	3539.16	3455.56	5731.4	4.59168e+06	5853.15	1.4291e+07	27253	1568	7.73741e+06	1270.88	7.47526e+06	1290.39	6.95098e+06	830778	1.17671e+06	8.15459e+06	3.08021e+07
59781	20230106.3	0.766149	17131.4	45865.6	6304.56	8.21863e+07	15485.8	1.53396e+07	47560.6	15996.4	8.2617e+06	13760.5	12520	4556.31	3545.8	3482.61	5767.3	4.59168e+06	5836.68	1.50774e+07	29148.2	1571.69	7.73741e+06	1285.69	7.21312e+06	1301.23	6.95098e+06	830718	1.18073e+06	8.13347e+06	3.12057e+07
59796	20230106.7	0.761166	17319.1	43805.5	6393.56	8.19242e+07	15526.1	1.50774e+07	49223.5	15994	8.2617e+06	13222.1	14194.6	4558.98	3539.1	3405.32	5776.3	4.59168e+06	5856.54	1.50774e+07	22169.4	1573.51	7.73741e+06	1295.62	7.47526e+06	1310.43	6.95098e+06	838036	1.17707e+06	8.15342e+06	3.07106e+07
59857	20230106.21	0.824187	17277.1	43619.5	5669.72	8.32349e+07	15450.5	1.53396e+07	49466.7	15764.6	8.2617e+06	13908.8	12589.5	4670.24	3547.54	3478.18	5734.8	4.59168e+06	5829.53	1.45532e+07	22136.6	1556.45	7.73741e+06	1288.43	7.47526e+06	1286.07	6.95098e+06	834573	1.18175e+06	8.14777e+06	3.08434e+07
59868	20230106.24	0.790629	17180.8	45688.2	6320.95	8.19242e+07	15386.8	1.50774e+07	47530.3	15929.1	8.2617e+06	13969.6	12598.1	4494.25	3549.42	3319.78	5683.6	4.32954e+06	5817.54	1.50774e+07	24101.1	1574.04	7.73741e+06	1284.61	7.21312e+06	1300.25	6.95098e+06	838278	1.18118e+06	8.13463e+06	3.16206e+07
59884	20230106.28	0.795828	17323.4	43547.6	6286.33	8.24484e+07	15226.6	1.50774e+07	49261.4	15824.1	8.2617e+06	13946.8	12335	4496.59	3644.43	3497.16	5691.5	4.59168e+06	5567.66	1.48153e+07	27214.7	1549.25	7.73741e+06	1274.97	7.47526e+06	1294.91	6.95098e+06	833582	1.17879e+06	8.15352e+06	3.14975e+07
59913	20230109.3	0.801341	17030.4	43745.9	6306.03	8.19242e+07	15416.5	1.50774e+07	49415.6	15958.3	8.2617e+06	13103.9	12604.9	4577.92	3575.21	3419.69	5774.8	4.59168e+06	5792.56	1.48153e+07	22710.6	1576.66	7.73741e+06	1284.27	7.21312e+06	1294.81	6.95098e+06	831861	1.17817e+06	8.13638e+06	3.10176e+07
59965	20230109.15	0.780423	17207.7	45587.6	6324.3	8.21863e+07	15694.6	1.50774e+07	48721.1	15977.1	8.2617e+06	13982.3	14053	4691.8	3651.59	3447.26	5780.6	4.59168e+06	5818.76	1.50774e+07	27856.7	1569.53	7.73741e+06	1288.17	7.21312e+06	1309.81	6.95098e+06	835622	1.17871e+06	8.15426e+06	3.07291e+07
60058	20230110.2	0.7761	17177	45887.3	6398.3	8.24484e+07	15764.4	1.53396e+07	47185	15979.8	8.2617e+06	14027.2	12793.6	4427.6	3553.85	3323.97	5811.5	4.59168e+06	5893.84	1.4291e+07	28237.9	1572.98	7.73741e+06	1285.54	7.21312e+06	1307.16	6.95098e+06	840375	1.17776e+06	8.17301e+06	3.09067e+07
60117	20230110.20	0.793709	17246.8	45684.9	6323.27	8.21863e+07	15710.9	1.53396e+07	46985.6	15956.6	8.2617e+06	13606.6	12624.1	4479.73	3536.62	3340.99	5785.1	4.59168e+06	5869.63	1.48153e+07	24286.8	1567.43	7.73741e+06	1286.53	7.73741e+06	1296.55	6.95098e+06	835130	1.17371e+06	8.15566e+06	3.13385e+07
60151	20230111.5	0.791139	17171.3	43901.4	6343.82	8.19242e+07	15433.3	1.53396e+07	46470.5	15951.2	7.99955e+06	13117.5	12389.4	4473.32	3536.04	3461.34	5778.4	4.59168e+06	5848.36	1.48153e+07	21576.2	1578.22	7.73741e+06	1284.88	7.21312e+06	1309.21	6.95098e+06	828282	1.18042e+06	8.1099e+06	3.07678e+07
60167	20230111.8	0.758201	17139.6	45992.8	6315.59	8.19242e+07	15534.9	1.50774e+07	47619.5	15970.3	8.2617e+06	12093.6	12479.7	4485.43	3573.63	3409.95	5782	4.59168e+06	5884.84	1.48153e+07	24799.6	1570.16	7.73741e+06	1297.01	7.21312e+06	1307.84	6.95098e+06	818104	1.18208e+06	8.1379e+06	3.08169e+07

run-snp-ctest

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_virtual_cft^	pi_ls_virtual_cft^	ls_jwt_virtual_cft^	pi_ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	ls_full_js_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_jwt_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	hist_sgx_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
60108	20230110.17	0.790914	17236.3	6285.89	8.19242e+07	45754.1	46683.4	12425.9	12910	4508.61	15451.9	1.53396e+07	15996.1	8.2617e+06	3585.52	5815.3	1.50774e+07	3354.08	5779.2	4.59168e+06	1567.63	7.73741e+06	23155.6	1292.57	7.47526e+06	1291.67	6.95098e+06	831710	1.17965e+06	8.14962e+06	3.09155e+07
60114	20230110.19	0.781169	17296.3	6218.7	8.24484e+07	45809.2	43162.8	12630.5	13182.5	4495.39	15474.1	1.50774e+07	15940.4	8.2617e+06	3625.4	5829.63	1.48153e+07	3319.42	5777.9	4.59168e+06	1568.42	7.73741e+06	21021.1	1285.54	7.73741e+06	1299.62	6.95098e+06	825917	1.17161e+06	8.14641e+06	3.11114e+07
60128	20230110.22	0.794375	17139.8	6264.16	8.19242e+07	46030.6	47574.6	14092.6	12016.6	4613.01	15337.9	1.50774e+07	15861	8.2617e+06	3525.99	5831.45	1.4291e+07	3457.71	5632.5	4.59168e+06	1556.4	7.73741e+06	22862	1270.91	7.21312e+06	1291.67	6.95098e+06	833286	1.18285e+06	8.15296e+06	3.08012e+07
60157	20230111.7	0.790922	17251.1	6352.13	8.29727e+07	43497.4	48910.7	12422.6	13587.2	4399.39	15488.8	1.50774e+07	15988.4	8.2617e+06	3557.28	5882.05	1.48153e+07	3325.16	5775.3	4.59168e+06	1572.31	7.73741e+06	25237.2	1297.37	7.21312e+06	1308.56	6.95098e+06	841967	1.1792e+06	8.15326e+06	3.08675e+07
60174	20230111.9	0.777148	17172.3	6328.58	8.24484e+07	43707.4	48573.4	12398.6	13867.1	4417.49	15502.9	1.50774e+07	15904.1	8.2617e+06	3566.34	5916.85	1.4291e+07	3310.67	5772.3	4.59168e+06	1565.1	7.73741e+06	27072.7	1293.72	7.47526e+06	1296.39	6.95098e+06	806360	1.1613e+06	8.15449e+06	3.0649e+07

[DomAyre]

Looks stable after 5 manual runs alongside PR run