Skip to content

Fix failures in multi-threaded indexing unit tests#4117

Merged
MahatiC merged 6 commits intomicrosoft:mainfrom
eddyashton:indexing_unit_test_deadlock
Aug 12, 2022
Merged

Fix failures in multi-threaded indexing unit tests#4117
MahatiC merged 6 commits intomicrosoft:mainfrom
eddyashton:indexing_unit_test_deadlock

Conversation

@eddyashton
Copy link
Member

@eddyashton eddyashton commented Aug 11, 2022

Resolves #4100.

Temporary Draft while I throw the CI at it

I believe I've fixed the issues in both tests that were leading to failures (aggressive backoff sometimes backing off too far, and unlucky timing ending a necessary tick before the polls had even started executing).

@ghost
Copy link

ghost commented Aug 11, 2022

indexing_unit_test_deadlock@48512 aka 20220811.12 vs main ewma over 20 builds from 48248 to 48499

Click to see table

main

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
48248 20220802.6 6346.87 8.32349e+07 20263.5 1.6126e+07 5553.34 1.53396e+07 2552.54 1.00967e+07 1590.59 1.66597e+08 2115.99 9.31027e+06 1419.19 9.79149e+07 1937.45 8.78598e+06 18446.5 909334 1.39206e+06 9.28798e+06 3.5189e+07
48267 20220803.3 6373.45 8.45456e+07 20471.7 1.58639e+07 5651.34 1.53396e+07 2611.14 9.83456e+06 1583.33 1.66597e+08 2114.15 9.31027e+06 1395.8 9.81771e+07 1929.23 9.04813e+06 20297.9 902400 1.3769e+06 9.23354e+06 3.56174e+07
48290 20220803.14 6381.82 8.3497e+07 20458.1 1.58639e+07 5640.15 1.53396e+07 2540.33 1.00967e+07 1592.15 1.66597e+08 2120.64 9.31027e+06 1422.34 9.87014e+07 1924.29 9.04813e+06 18289 876257 1.36151e+06 9.20445e+06 3.58669e+07
48304 20220804.1 6283.98 8.37592e+07 20390.7 1.58639e+07 5621.96 1.56017e+07 2545.3 1.00967e+07 1617.06 1.64762e+08 2117.04 9.31027e+06 1438.5 9.84392e+07 2016.09 8.78598e+06 20426.3 908481 1.35574e+06 9.19206e+06 3.55556e+07
48323 20220804.8 6308.84 8.32349e+07 20224.2 1.69124e+07 5631.75 1.53396e+07 2557.44 9.83456e+06 1617.92 1.66859e+08 2117.06 9.31027e+06 1364.2 9.76528e+07 2009.73 8.78598e+06 21082.9 893076 1.36898e+06 9.30064e+06 3.57417e+07
48348 20220804.16 6470.12 8.40213e+07 20162.3 1.63882e+07 5611.59 1.56017e+07 2610.84 1.00967e+07 1603.88 1.66859e+08 2133.3 9.31027e+06 1408.7 9.87014e+07 1936.8 9.04813e+06 20767.6 874353 1.31602e+06 8.95888e+06 3.50685e+07
48375 20220805.3 6581.93 8.40213e+07 20244.8 1.58639e+07 5588.85 1.53396e+07 2598.55 1.00967e+07 1590.43 1.66335e+08 2110.71 9.31027e+06 1437.79 9.79149e+07 1965.96 9.04813e+06 20769.3 901323 1.3709e+06 9.34298e+06 3.58669e+07
48376 20220805.4 6575.88 8.40213e+07 20602.8 1.58639e+07 5680.24 1.53396e+07 2480.28 1.00967e+07 1649.69 1.66859e+08 2122.21 9.31027e+06 1442.64 9.81771e+07 1929.88 9.04813e+06 20495.5 911473 1.37026e+06 9.41172e+06 3.63121e+07
48391 20220805.11 6446.79 8.32349e+07 20014.6 1.6126e+07 5628.3 1.58639e+07 2594.33 1.00967e+07 1603.05 1.66597e+08 2073.94 9.31027e+06 1415.05 9.63421e+07 1920.2 9.04813e+06 18389.9 909972 1.37044e+06 9.18377e+06 3.59298e+07
48399 20220808.2 6499.33 8.42835e+07 20415.4 1.66503e+07 5670.06 1.53396e+07 2554.96 1.00967e+07 1630.43 1.645e+08 2118.44 9.31027e+06 1412.02 9.81771e+07 1932.81 9.04813e+06 20513.5 871966 1.36106e+06 9.20859e+06 3.56168e+07
48407 20220808.5 6438.4 8.40213e+07 20249.2 1.6126e+07 5638.84 1.53396e+07 2548 1.00967e+07 1595.51 1.67121e+08 2108.24 9.31027e+06 1430.87 9.81771e+07 1927.33 8.78598e+06 19704.5 888306 1.36669e+06 9.19206e+06 3.58669e+07
48422 20220808.12 6398.02 8.3497e+07 20137.7 1.63882e+07 5695.03 1.53396e+07 2625.18 9.83456e+06 1582.61 1.66859e+08 2120.63 8.78598e+06 1404 9.73907e+07 1935.04 9.04813e+06 19485.4 902483 1.39112e+06 9.26278e+06 3.59298e+07
48427 20220808.14 6230.03 8.40213e+07 19679.8 1.58639e+07 5547.25 1.53396e+07 2417.7 1.03588e+07 1550.66 1.645e+08 2072.83 9.04813e+06 1369.48 9.76528e+07 1959.08 9.04813e+06 17632.4 920524 1.40023e+06 9.3901e+06 3.63114e+07
48435 20220808.17 6396.62 8.37592e+07 20212.4 1.58639e+07 5676.31 1.56017e+07 2492.05 1.00967e+07 1582.1 1.66859e+08 2084.38 9.31027e+06 1401.8 9.79149e+07 1936.03 9.04813e+06 22698 907599 1.38341e+06 9.30486e+06 3.55556e+07
48452 20220809.2 6397.96 8.37592e+07 20177.9 1.63882e+07 5695.08 1.56017e+07 2498.61 1.00967e+07 1602.55 1.66597e+08 2093.11 9.31027e+06 1399.63 9.81771e+07 1931.37 8.78598e+06 20235.1 903311 1.37495e+06 9.21688e+06 3.54939e+07
48461 20220809.7 6585.9 8.3497e+07 20158 1.63882e+07 5669.79 1.53396e+07 2557.5 1.00967e+07 1626.24 1.645e+08 2121.12 9.31027e+06 1443.42 9.84392e+07 1986.42 9.04813e+06 18271.3 886722 1.41396e+06 9.41163e+06 3.59292e+07
48466 20220809.9 6444.69 8.37592e+07 20005.2 1.56017e+07 5605.49 1.56017e+07 2541.33 1.00967e+07 1618.96 1.66859e+08 2115.35 9.31027e+06 1450.97 9.81771e+07 2014.27 9.04813e+06 20835.2 901011 1.37201e+06 9.22938e+06 3.58669e+07
48469 20220810.1 6186.62 8.42835e+07 20033.2 1.58639e+07 5604.55 1.53396e+07 2556.71 9.83456e+06 1587.94 1.64762e+08 2082.75 9.04813e+06 1421.15 9.84392e+07 1944.56 9.04813e+06 22527.5 906189 1.38172e+06 9.21684e+06 3.58669e+07
48486 20220811.1 6270.83 8.40213e+07 19935.8 1.58639e+07 5612.96 1.53396e+07 2499.35 1.2456e+07 1583.15 1.66859e+08 2112.67 9.31027e+06 1405.59 9.81771e+07 1923.87 9.04813e+06 23657.6 917679 1.38602e+06 9.28794e+06 3.56788e+07
48499 20220811.7 6318.24 8.45456e+07 20081.4 1.66503e+07 5615.16 1.53396e+07 2595.81 1.00967e+07 1589.3 1.66072e+08 2084.45 9.31027e+06 1416.86 9.84392e+07 1942.12 1.11453e+07 17960.4 888152 1.37708e+06 9.23767e+06 3.56788e+07

indexing_unit_test_deadlock

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
48503 20220811.8 6305.47 8.45456e+07 20255.3 1.58639e+07 5561.84 1.53396e+07 2539.84 1.00967e+07 1624.6 1.66859e+08 2119.02 9.31027e+06 1425.1 9.81771e+07 1925.88 1.0621e+07 23964.7 903428 1.36998e+06 9.23763e+06 3.58663e+07
48512 20220811.12 6213.85 8.50699e+07 20152.7 1.6126e+07 5423.05 1.53396e+07 2541.69 9.83456e+06 1615.85 1.66859e+08 2144.28 9.04813e+06 1433.89 9.81771e+07 1928.21 8.78598e+06 17932.3 913220 1.38191e+06 9.45077e+06 3.58669e+07

images

@eddyashton eddyashton marked this pull request as ready for review August 11, 2022 17:07
@eddyashton eddyashton requested a review from a team August 11, 2022 17:07
Copy link
Member

@MahatiC MahatiC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI seems happy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate failures in multi-threaded indexing unit tests

3 participants