Skip to content

Conversation

@austin-denoble
Copy link
Contributor

@austin-denoble austin-denoble commented Mar 12, 2024

Problem

Currently, CollectionTest, and ConfigureIndexTest are very flakey. Collections tests take a long while to run due to the need to spin up a collection, wait until it's ready, and then creating an index and waiting until that's ready.

ConfigureIndexTest deletes the index as a part of the final @Test in the file, but it doesn't seem like the order of execution is guaranteed, so deleting an index like this may cause issues: https://www.baeldung.com/junit-5-test-order

Solution

  • Update CollectionTest to create a single collection from the initial index that's shared across both tests. Previously we were creating a collection for each test, which is not ultimately necessary and is very time-consuming. Also, use assertWithRetry() to retry upserting and describing index stats. These calls seemingly have a tendency to flake when made against a newly created index. Retrying a few times if we have an error on the initial pass seems to help here.
  • ConfigureIndexTest has a finally clause in the last @Test in the file deletes the index. Given we set up with @BeforeAll and the execution order is not guaranteed unless we specify, it feels like it makes more sense to clean things up in @AfterAll, which seems to resolve the issue with the test itself.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • None of the above: (integration tests)

Test Plan

Validate integration runs in CI. Runs should overall be a few minutes faster, and less prone to failing on these specific tests.

…ssertions, update configure index test to clean up @afterall
@austin-denoble austin-denoble marked this pull request as ready for review March 12, 2024 16:48
Copy link
Contributor

@rohanshah18 rohanshah18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this, I have left one comment but otherwise it looks great!

@austin-denoble austin-denoble merged commit 55b7db3 into main Mar 12, 2024
@austin-denoble austin-denoble deleted the adenoble/fix-collection-and-configure-index-integration-tests branch March 12, 2024 18:36
@rohanshah18 rohanshah18 mentioned this pull request Jan 10, 2025
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants