500 Server Error related to stale tests for ci-kubernetes-unit #1099
Description
We discovered there may be some kind of a problem with the stale tests for this job, the problem can most easily be seen with:
❯ curl https://testgrid.k8s.io/sig-release-master-blocking/table\?show-stale-tests\=\&tab\=ci-kubernetes-unit
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>And the UI also shows a problem loading: https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit&show-stale-tests=
Which reports "Internal Error: Refresh, and file an issue if problem persists. (Your table may be too big; consider lower days_of_results or enable_test_methods = false)" I am unable to find a way to set these recommended values, tried query params with both _ and -.
The normal view of the job loads ok (although slowly): https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit
According to it there are 14,000 stale tests.