Mark multi-threaded indexing tests as may_fail

[eddyashton]

The unit tests for multi-threaded indexing added in #4075 are occasionally failing, as the watchdog timer identifies that the test has been running for more than 10s. This may be a deadlock, or an unusually unlucky/slow run.

Decorating these tests with may_fail(true), so they are still run but won't cause a test failure when they fail. Will investigate the failure in more detail at a later date.

https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=48310&view=logs&j=e982d093-d12a-5a2d-1a1d-0c6d38b8823f&t=8be5454d-0d8f-51e4-7372-58df022e9ebc&l=441

18: Test command: /__w/1/s/build/indexing_test
18: Test timeout computed to be: 10000000
18: terminating with uncaught exception of type doctest::detail::TestFailureException
18: [doctest] doctest version is "2.4.8"
18: [doctest] run with "--help" for options
18: ===============================================================================
18: ../src/indexing/test/indexing.cpp:710:
18: TEST SUITE: indexing
18: TEST CASE:  multi-threaded indexing - bucketed
18: 
18: ../src/indexing/test/indexing.cpp:933: FATAL ERROR: REQUIRE( now - start_time < max_multithread_run_time ) is NOT correct!
18:   values: REQUIRE( {?} <  {?} )
18: 
18: ../src/indexing/test/indexing.cpp:710: FATAL ERROR: test case CRASHED: SIGABRT - Abort (abnormal termination) signal
18: 
18: ===============================================================================
18: [doctest] test cases:     5 |     4 passed | 1 failed | 3 skipped
18: [doctest] assertions: 12845 | 12844 passed | 1 failed |
18: [doctest] Status: FAILURE!
18/58 Test #18: indexing_test ....................Child aborted***Exception:  10.49 sec

[jumaffre]

Could this be included in #4086 too?

[eddyashton]

Could this be included in #4086 too?

Sure, I'll do that in a single PR rather than backporting both. Had forgotten that was still open.

[ghost]

indexing_thread_allow_failures@48322 aka 20220804.7 vs main ewma over 20 builds from 47950 to 48304

Click to see table

main

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
47950	20220722.11	6346.72	8.42835e+07	20498	1.66503e+07	5696.14	1.48153e+07	2547.54	1.00967e+07	1593.79	1.66597e+08	2087.82	9.31027e+06	1417.11	9.76528e+07	1929.17	8.78598e+06	20113.6	900215	1.37311e+06	9.21273e+06	3.56794e+07
47964	20220725.3	6445.58	8.45456e+07	20264.1	1.6126e+07	5715.87	1.53396e+07	2543.74	1.00967e+07	1612.58	1.66597e+08	2108.59	9.31027e+06	1435.99	9.76528e+07	1937.49	1.0621e+07	18993.5	902517	1.33865e+06	9.39876e+06	3.48893e+07
47991	20220726.3	6424.54	8.29727e+07	20484.1	1.58639e+07	5660.34	1.53396e+07	2552.74	1.00967e+07	1590.25	1.66597e+08	2122.4	9.04813e+06	1429.67	9.81771e+07	1926.78	8.78598e+06	20464.4	890933	1.38154e+06	9.48144e+06	3.62478e+07
47999	20220726.6	6463.07	8.3497e+07	20411.6	1.58639e+07	5679.05	1.56017e+07	2544.17	1.00967e+07	1621.25	1.66859e+08	2164.94	9.31027e+06	1426.71	9.84392e+07	2024.37	9.04813e+06	17785.5	908399	1.36251e+06	9.39872e+06	3.65714e+07
48019	20220726.14	6543.72	8.42835e+07	20091.2	1.56017e+07	5601.14	1.50774e+07	2499.24	9.83456e+06	1600.03	1.645e+08	2122.92	9.31027e+06	1420.35	9.79149e+07	2014.48	9.04813e+06	19942.2	903392	1.39861e+06	9.02994e+06	3.59298e+07
48039	20220726.21	6604.8	8.45456e+07	20061.9	1.63882e+07	5657.52	1.53396e+07	2462.46	1.29803e+07	1599.33	1.67121e+08	2133.23	9.04813e+06	1444.67	9.79149e+07	1932.82	9.04813e+06	20529.4	898398	1.37154e+06	9.20445e+06	3.59292e+07
48056	20220727.3	7376.83	8.48077e+07	20455.4	1.58639e+07	5645.94	1.53396e+07	2553.28	1.00967e+07	1612.2	1.66597e+08	2117.56	9.31027e+06	1424.76	9.81771e+07	1932.75	8.78598e+06	18180.7	905868	1.38004e+06	9.22514e+06	3.58669e+07
48076	20220728.1	6377.5	8.45456e+07	20423.6	1.6126e+07	5695.92	1.53396e+07	2536.06	1.03588e+07	1631.52	1.66859e+08	2129.7	9.31027e+06	1436.56	9.81771e+07	1937.91	9.04813e+06	19899.1	861326	1.3754e+06	9.20855e+06	3.55549e+07
48083	20220728.4	6537.16	8.32349e+07	20590.5	1.6126e+07	5720.58	1.56017e+07	2631.03	9.83456e+06	1599.78	1.64762e+08	2126.15	9.31027e+06	1454.7	9.81771e+07	1946.79	9.04813e+06	22317.3	890155	1.36259e+06	9.20855e+06	3.58669e+07
48101	20220728.10	6449.49	8.42835e+07	20328.8	1.56017e+07	5590.59	1.50774e+07	2545.39	1.00967e+07	1608.68	1.66597e+08	2124.69	9.31027e+06	1415.34	9.87014e+07	1928.96	9.04813e+06	19989.8	872335	1.34153e+06	9.21684e+06	3.58663e+07
48116	20220729.2	6419.03	8.40213e+07	20316.1	1.6126e+07	5998.81	1.53396e+07	2559.25	1.00967e+07	1611.63	1.66335e+08	2119.38	9.04813e+06	1399.69	9.63421e+07	1926.59	9.04813e+06	20842.4	905830	1.37219e+06	9.22934e+06	3.61199e+07
48182	20220729.22	7435.73	8.48077e+07	20197.3	1.69124e+07	5628.12	1.56017e+07	2549.84	1.00967e+07	1619.5	1.67121e+08	2151.72	9.57242e+06	1420.17	9.81771e+07	1929.97	9.04813e+06	20385.6	906430	1.3676e+06	9.27108e+06	3.56168e+07
48201	20220729.31	6490.45	8.37592e+07	20065.7	1.6126e+07	5634.34	1.53396e+07	2558.72	1.00967e+07	1588.91	1.66859e+08	2120.95	9.31027e+06	1413.06	9.79149e+07	1935.36	9.04813e+06	21085.2	906511	1.3756e+06	9.31752e+06	3.65708e+07
48214	20220731.3	5763.08	8.40213e+07	20400.8	1.71746e+07	5648.54	1.48153e+07	2559.51	1.00967e+07	1588.95	1.66597e+08	2125.83	9.31027e+06	1413.25	9.81771e+07	1933.22	9.04813e+06	19675.1	904759	1.39453e+06	9.40753e+06	3.59298e+07
48218	20220801.1	6390.34	8.29727e+07	20328.2	1.63882e+07	5687.58	1.53396e+07	2561.95	1.03588e+07	1597.78	1.66597e+08	2119.59	9.04813e+06	1432.89	9.79149e+07	2025.19	9.04813e+06	22238.5	909044	1.37311e+06	9.326e+06	3.57411e+07
48239	20220802.2	6341.07	8.32349e+07	20109.1	1.66503e+07	5683.14	1.56017e+07	2583.4	1.00967e+07	1609.45	1.66597e+08	2124.37	9.31027e+06	1433.98	9.79149e+07	1933.24	9.04813e+06	20349.6	891366	1.37368e+06	9.39881e+06	3.58048e+07
48248	20220802.6	6346.87	8.32349e+07	20263.5	1.6126e+07	5553.34	1.53396e+07	2552.54	1.00967e+07	1590.59	1.66597e+08	2115.99	9.31027e+06	1419.19	9.79149e+07	1937.45	8.78598e+06	18446.5	909334	1.39206e+06	9.28798e+06	3.5189e+07
48267	20220803.3	6373.45	8.45456e+07	20471.7	1.58639e+07	5651.34	1.53396e+07	2611.14	9.83456e+06	1583.33	1.66597e+08	2114.15	9.31027e+06	1395.8	9.81771e+07	1929.23	9.04813e+06	20297.9	902400	1.3769e+06	9.23354e+06	3.56174e+07
48290	20220803.14	6381.82	8.3497e+07	20458.1	1.58639e+07	5640.15	1.53396e+07	2540.33	1.00967e+07	1592.15	1.66597e+08	2120.64	9.31027e+06	1422.34	9.87014e+07	1924.29	9.04813e+06	18289	876257	1.36151e+06	9.20445e+06	3.58669e+07
48304	20220804.1	6283.98	8.37592e+07	20390.7	1.58639e+07	5621.96	1.56017e+07	2545.3	1.00967e+07	1617.06	1.64762e+08	2117.04	9.31027e+06	1438.5	9.84392e+07	2016.09	8.78598e+06	20426.3	908481	1.35574e+06	9.19206e+06	3.55556e+07

indexing_thread_allow_failures

build_id	build_number	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_sgx_cft^	ls_sgx_cft_mem	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	ls_v8_sgx_cft^	ls_v8_sgx_cft_mem	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_full_v8_sgx_cft^	ls_full_v8_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	hist_sgx_cft^	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
48322	20220804.7	6408.38	8.42835e+07	20413.5	1.58639e+07	5630.21	1.53396e+07	2554.48	1.00967e+07	1596.13	1.66597e+08	2124.24	9.31027e+06	1411.02	9.84392e+07	1936.62	9.04813e+06	21944.7	904708	1.36206e+06	9.37725e+06	3.56794e+07