Skip to content

Configure test suite for edge-endpoint support #139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tyler-romero merged 4 commits into from
Dec 8, 2023
Merged

Configure test suite for edge-endpoint support #139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d08f2b3
Configure tests for edge-endpoint support
Dec 6, 2023
f952f60
Fixes
Dec 7, 2023
2c0ae81
Apply suggestions from code review
Dec 7, 2023
b6ced50
Address PR comments
Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,26 @@ PYTEST=poetry run pytest -v
# `make test TEST_ARGS="-k some_filter"`
TEST_ARGS=

CLOUD_FILTERS = -m "not run_only_for_edge_endpoint"
EDGE_FILTERS = -m "not skip_for_edge_endpoint"

test: install ## Run tests against the prod API (needs GROUNDLIGHT_API_TOKEN)
${PYTEST} ${TEST_ARGS} test
${PYTEST} ${TEST_ARGS} ${CLOUD_FILTERS} test

test-4edge: install ## Run tests against the prod API via the edge-endpoint (needs GROUNDLIGHT_API_TOKEN)
${PYTEST} ${TEST_ARGS} ${EDGE_FILTERS} test

test-local: install ## Run tests against a localhost API (needs GROUNDLIGHT_API_TOKEN and a local API server)
GROUNDLIGHT_ENDPOINT="http://localhost:8000/" ${PYTEST} ${TEST_ARGS} test
GROUNDLIGHT_ENDPOINT="http://localhost:8000/" ${PYTEST} ${TEST_ARGS} ${CLOUD_FILTERS} test

test-integ: install ## Run tests against the integ API server (needs GROUNDLIGHT_API_TOKEN)
GROUNDLIGHT_ENDPOINT="https://api.integ.groundlight.ai/" ${PYTEST} ${TEST_ARGS} test
GROUNDLIGHT_ENDPOINT="https://api.integ.groundlight.ai/" ${PYTEST} ${TEST_ARGS} ${CLOUD_FILTERS} test

test-docs: install ## Run the example code and tests in our docs against the prod API (needs GROUNDLIGHT_API_TOKEN)
poetry run pytest -v --markdown-docs ${TEST_ARGS} docs README.md
${PYTEST} --markdown-docs ${TEST_ARGS} docs README.md

test-docs-integ: install ## Run the example code and tests in our docs against the integ API (needs GROUNDLIGHT_API_TOKEN)
GROUNDLIGHT_ENDPOINT="https://api.integ.groundlight.ai/" poetry run pytest -v --markdown-docs ${TEST_ARGS} docs README.md
GROUNDLIGHT_ENDPOINT="https://api.integ.groundlight.ai/" ${PYTEST} --markdown-docs ${TEST_ARGS} docs README.md

# Adjust which paths we lint
LINT_PATHS="src test bin samples"
Expand Down
12 changes: 12 additions & 0 deletions test/integration/test_groundlight.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ def test_submit_image_query_with_human_review_param(gl: Groundlight, detector: D
assert is_valid_display_result(_image_query.result)


@pytest.mark.skip_for_edge_endpoint(reason="The edge-endpoint does not support passing metadata.")
Copy link
Contributor

@blaise-muhirwa blaise-muhirwa Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we were mentioned yesterday I think, we can also easily support this once we have persistent storage for image queries on the edge.

Copy link
Contributor Author

@tyler-romero tyler-romero Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but for now we want to unblock other PRs on the edge. And in general we may not be fully feature complete on the edge relative to the cloud.

@pytest.mark.parametrize("metadata", [None, {}, {"a": 1}, '{"a": 1}'])
def test_submit_image_query_with_metadata(
gl: Groundlight, detector: Detector, image: str, metadata: Union[Dict, str, None]
Expand All @@ -267,6 +268,7 @@ def test_submit_image_query_with_metadata(
assert retrieved_iq.metadata == expected_metadata


@pytest.mark.skip_for_edge_endpoint(reason="The edge-endpoint does not support passing metadata.")
def test_ask_methods_with_metadata(gl: Groundlight, detector: Detector, image: str):
metadata = {"a": 1}
iq = gl.ask_ml(detector=detector.id, image=image, metadata=metadata)
Expand All @@ -280,12 +282,14 @@ def test_ask_methods_with_metadata(gl: Groundlight, detector: Detector, image: s
# assert iq.metadata == metadata


@pytest.mark.skip_for_edge_endpoint(reason="The edge-endpoint does not support passing metadata.")
@pytest.mark.parametrize("metadata", ["", "a", b'{"a": 1}'])
def test_submit_image_query_with_invalid_metadata(gl: Groundlight, detector: Detector, image: str, metadata: Any):
with pytest.raises(TypeError):
gl.submit_image_query(detector=detector.id, image=image, human_review="NEVER", metadata=metadata)


@pytest.mark.skip_for_edge_endpoint(reason="The edge-endpoint does not support passing metadata.")
def test_submit_image_query_with_metadata_too_large(gl: Groundlight, detector: Detector, image: str):
with pytest.raises(ValueError):
gl.submit_image_query(
Expand All @@ -296,6 +300,14 @@ def test_submit_image_query_with_metadata_too_large(gl: Groundlight, detector: D
)


@pytest.mark.run_only_for_edge_endpoint
def test_submit_image_query_with_metadata_returns_user_error(gl: Groundlight, detector: Detector, image: str):
"""On the edge-endpoint, we raise an exception if the user passes metadata."""
with pytest.raises(openapi_client.exceptions.ApiException) as exc_info:
gl.submit_image_query(detector=detector.id, image=image, human_review="NEVER", metadata={"a": 1})
assert is_user_error(exc_info.value.status)


def test_submit_image_query_jpeg_bytes(gl: Groundlight, detector: Detector):
jpeg = open("test/assets/dog.jpeg", "rb").read()
_image_query = gl.submit_image_query(detector=detector.id, image=jpeg, human_review="NEVER")
Expand Down