Automatic tests randomly fail

[Didayolo]

The CircleCI tests sometimes fail. They can be passed simply by re-running them, so it seems to be quite random.

Example (failure):

https://app.circleci.com/pipelines/github/codalab/codabench/887/workflows/8fcc2681-8b01-4f19-84eb-33a2d305da35/jobs/2554

After the re-run (success):

https://app.circleci.com/pipelines/github/codalab/codabench/887/workflows/7c377d7b-ce42-42d8-b8a0-82e02387d389/jobs/2556

The error is a time out exception on the following E2E selenium test:

FAILED src/tests/functional/test_submissions.py::TestSubmissions::test_v15_submission_end_to_end

/usr/local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py:80: TimeoutException

[bbearce]

I believe this is due to selenium requesting elements from the HTML DOM. It has to wait for them and sometimes it takes too long to get the elements. Sometimes not. We can continue to add sleep commands to fix this.

More details...like javascript selenium requests objects and acts on them. Sometimes an element is a pop-up window. On that pop-up window could be a button selenium wants to click. If that pop-up hasn't loaded into selenium before it asks to click the button, selenium waits to get the button, but it never will as there was no button to click. Eventually waiting to click the button may time out.

[Didayolo]

Interesting. Ideally we'd like to avoid these random failures because it is quite misleading. If adding some wait is sufficient to solve this then let's do it. I'm not sure if it is normal to need this or if it is bad practice.

[bbearce]

It smells like bad practice and could need tweaking over time continuously. This may become an issue as codabench grows and or more tests are added.

The fundamental issue is that javascript can have its code written so that functions are forced to wait on each other to finish before carrying on. Maybe we can do the same with selenium and implement a "wait till the previous command finishes" versus a random tweaked sleep which may or may not be enough time.

[bbearce]

I see this for instance:
https://www.selenium.dev/documentation/webdriver/waits/

Example:

driver.navigate("file:///race_condition.html")
WebDriverWait(driver, timeout=10).until(document_initialised)
el = driver.find_element(By.TAG_NAME, "p")
assert el.text == "Hello from JavaScript!"

The .until() is waiting for something. It still has a timeout but we can adjust that.

What we have is this:
https://github.com/codalab/codabench/blob/develop/src/tests/functional/test_submissions.py
line

# This clicks the page before it loads fully, delay it a bit...
self.wait(1)
self.find('.item[data-tab="participate-tab"]').click()

This code works only if the page indeed loaded within the 1 second specified by self.wait(1) and the "participate-tab" exists. It would be better to attach a .until() somehow.

[Didayolo]

Should be fixed by:

• Increase selenium waits to fix automatic tests #788

[Didayolo]

=================================== FAILURES ===================================
________________ TestSubmissions.test_v15_submission_end_to_end ________________

self = <tests.functional.test_submissions.TestSubmissions testMethod=test_v15_submission_end_to_end>

    def test_v15_submission_end_to_end(self):
>       self._run_submission_and_add_to_leaderboard('competition_15.zip', 'submission_15.zip', '*** prediction_score', has_solutions=False, timeout=2000)

src/tests/functional/test_submissions.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/tests/functional/test_submissions.py:55: in _run_submission_and_add_to_leaderboard
    assert self.find_text_in_class('.submission_output', expected_submission_output, timeout=timeout)
src/tests/utils.py:130: in find_text_in_class
    return wait.until(EC.text_to_be_present_in_element((By.CSS_SELECTOR, klass), text))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="80c02518-a446-44b6-b4a5-5e1b23981026")>
method = <selenium.webdriver.support.expected_conditions.text_to_be_present_in_element object at 0x7fc6d5da2580>
message = ''

    def until(self, method, message=''):
        """Calls the method provided with the driver as an argument until the \
        return value is not False."""
        screen = None
        stacktrace = None
    
        end_time = time.time() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, 'screen', None)
                stacktrace = getattr(exc, 'stacktrace', None)
            time.sleep(self._poll)
            if time.time() > end_time:
                break
>       raise TimeoutException(message, screen, stacktrace)
E       selenium.common.exceptions.TimeoutException: Message:

/usr/local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py:80: TimeoutException

[Didayolo]

The scoring logs are not fully written inside the "submission_output" container (the black box) for this test competition: