Skip to content

fix(tests): simplify testing #1343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leshy merged 1 commit into from
Feb 24, 2026
Merged

fix(tests): simplify testing #1343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 2 additions & 98 deletions .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,70 +212,9 @@ jobs:
uses: ./.github/workflows/tests.yml
secrets: inherit
with:
cmd: "pytest && pytest -m ros" # run tests that depend on ros as well
cmd: "pytest --durations=0 -m 'not (tool or mujoco)'"
dev-image: ros-dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true' || needs.check-changes.outputs.ros == 'true') && needs.ros-dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

run-tests:
needs: [check-changes, dev]
if: ${{
always() &&
needs.check-changes.result == 'success' &&
(needs.check-changes.outputs.tests == 'true' ||
needs.check-changes.outputs.python == 'true' ||
needs.check-changes.outputs.dev == 'true')
}}
uses: ./.github/workflows/tests.yml
secrets: inherit
with:
cmd: "pytest"
dev-image: dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true') && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

# we run in parallel with normal tests for speed
run-heavy-tests:
needs: [check-changes, dev]
if: ${{
always() &&
needs.check-changes.result == 'success' &&
(needs.check-changes.outputs.tests == 'true' ||
needs.check-changes.outputs.python == 'true' ||
needs.check-changes.outputs.dev == 'true')
}}
uses: ./.github/workflows/tests.yml
secrets: inherit
with:
cmd: "pytest -m heavy"
dev-image: dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true') && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

run-lcm-tests:
needs: [check-changes, dev]
if: ${{
always() &&
needs.check-changes.result == 'success' &&
(needs.check-changes.outputs.tests == 'true' ||
needs.check-changes.outputs.python == 'true' ||
needs.check-changes.outputs.dev == 'true')
}}
uses: ./.github/workflows/tests.yml
secrets: inherit
with:
cmd: "pytest -m lcm"
dev-image: dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true') && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

run-integration-tests:
needs: [check-changes, dev]
if: ${{
always() &&
needs.check-changes.result == 'success' &&
(needs.check-changes.outputs.tests == 'true' ||
needs.check-changes.outputs.python == 'true' ||
needs.check-changes.outputs.dev == 'true')
}}
uses: ./.github/workflows/tests.yml
secrets: inherit
with:
cmd: "pytest -m integration"
dev-image: dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true') && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

run-mypy:
needs: [check-changes, ros-dev]
if: ${{
Expand All @@ -292,43 +231,8 @@ jobs:
cmd: "MYPYPATH=/opt/ros/humble/lib/python3.10/site-packages mypy dimos"
dev-image: ros-dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true' || needs.check-changes.outputs.ros == 'true') && needs.ros-dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

# Run module tests directly to avoid pytest forking issues
# run-module-tests:
# needs: [check-changes, dev]
# if: ${{
# always() &&
# needs.check-changes.result == 'success' &&
# ((needs.dev.result == 'success') ||
# (needs.dev.result == 'skipped' &&
# needs.check-changes.outputs.tests == 'true'))
# }}
# runs-on: [self-hosted, x64, 16gb]
# container:
# image: ghcr.io/dimensionalos/dev:${{ needs.check-changes.outputs.dev == 'true' && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}
# steps:
# - name: Fix permissions
# run: |
# sudo chown -R $USER:$USER ${{ github.workspace }} || true
#
# - uses: actions/checkout@v4
# with:
# lfs: true
#
# - name: Configure Git LFS
# run: |
# git config --global --add safe.directory '*'
# git lfs install
# git lfs fetch
# git lfs checkout
#
# - name: Run module tests
# env:
# CI: "true"
# run: |
# /entrypoint.sh bash -c "pytest -m module"

ci-complete:
needs: [check-changes, ros, python, ros-python, dev, ros-dev, run-tests, run-heavy-tests, run-lcm-tests, run-integration-tests, run-ros-tests, run-mypy]
needs: [check-changes, ros, python, ros-python, dev, ros-dev, run-ros-tests, run-mypy]
Copy link
Contributor

@leshy leshy Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to know what the plan was with ros/non ros parallel tests - I planned to treat ros build as a separate OS, since it does heavy intervention into the OS itslef, and wanted to make sure that unguarded ros imports don't crash non-ros machines, so potentially we want to re-introduce

Copy link
Contributor Author

@paul-nechifor paul-nechifor Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense to test that, but maybe not on every PR? Maybe we can have a periodic runner which runs all the tests on a variety of environments like Ubuntu 22.04/24.04, with ROS/without ROS, MacOS, etc.

runs-on: [self-hosted, Linux]
if: always()
steps:
Expand Down
2 changes: 1 addition & 1 deletion bin/pytest-slow
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
set -euo pipefail

. .venv/bin/activate
exec pytest "$@" -m 'not (tool or module or neverending or mujoco)' dimos
exec pytest "$@" -m 'not (tool or mujoco)' dimos
10 changes: 5 additions & 5 deletions dimos/agents/mcp/test_mcp_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def add(self, x: int, y: int) -> str:
return str(x + y)


@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.parametrize("dask", [False, True])
def test_can_call_tool(dask, agent_setup):
history = agent_setup(
Expand Down Expand Up @@ -66,7 +66,7 @@ def register_user(self, name: str) -> str:
return "User name registered successfully."


@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.parametrize("dask", [False, True])
def test_can_call_again_on_error(dask, agent_setup):
history = agent_setup(
Expand Down Expand Up @@ -118,7 +118,7 @@ def go_to_location(self, description: str) -> str:
return f"Going to the {description}."


@pytest.mark.integration
@pytest.mark.slow
def test_multiple_tool_calls_with_multiple_messages(agent_setup):
history = agent_setup(
blueprints=[MultipleTools.blueprint(), NavigationSkill.blueprint()],
Expand Down Expand Up @@ -172,7 +172,7 @@ def test_multiple_tool_calls_with_multiple_messages(agent_setup):
assert len(go_to_location_calls) == 2


@pytest.mark.integration
@pytest.mark.slow
def test_prompt(agent_setup):
history = agent_setup(
blueprints=[],
Expand All @@ -190,7 +190,7 @@ def take_a_picture(self) -> Image:
return Image.from_file(get_data("cafe-smol.jpg")).to_rgb()


@pytest.mark.integration
@pytest.mark.slow
def test_image(agent_setup):
history = agent_setup(
blueprints=[Visualizer.blueprint()],
Expand Down
4 changes: 2 additions & 2 deletions dimos/agents/skills/test_google_maps_skill_container.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ def __init__(self):
self._max_valid_distance = 20000


@pytest.mark.integration
@pytest.mark.slow
def test_where_am_i(agent_setup) -> None:
history = agent_setup(
blueprints=[FakeGPS.blueprint(), MockedWhereAmISkill.blueprint()],
Expand All @@ -80,7 +80,7 @@ def test_where_am_i(agent_setup) -> None:
assert "bourbon" in history[-1].content.lower()


@pytest.mark.integration
@pytest.mark.slow
def test_get_gps_position_for_queries(agent_setup) -> None:
history = agent_setup(
blueprints=[FakeGPS.blueprint(), MockedPositionSkill.blueprint()],
Expand Down
4 changes: 2 additions & 2 deletions dimos/agents/skills/test_gps_nav_skills.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def __init__(self):
self._max_valid_distance = 50000


@pytest.mark.integration
@pytest.mark.slow
def test_set_gps_travel_points(agent_setup) -> None:
history = agent_setup(
blueprints=[FakeGPS.blueprint(), MockedGpsNavSkill.blueprint()],
Expand All @@ -50,7 +50,7 @@ def test_set_gps_travel_points(agent_setup) -> None:
assert "success" in history[-1].content.lower()


@pytest.mark.integration
@pytest.mark.slow
def test_set_gps_travel_points_multiple(agent_setup) -> None:
history = agent_setup(
blueprints=[FakeGPS.blueprint(), MockedGpsNavSkill.blueprint()],
Expand Down
6 changes: 3 additions & 3 deletions dimos/agents/skills/test_navigation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def _navigate_using_semantic_map(self, query):
return f"Successfuly arrived at '{query}'"


@pytest.mark.integration
@pytest.mark.slow
def test_stop_movement(agent_setup) -> None:
history = agent_setup(
blueprints=[
Expand All @@ -86,7 +86,7 @@ def test_stop_movement(agent_setup) -> None:
assert "stopped" in history[-1].content.lower()


@pytest.mark.integration
@pytest.mark.slow
def test_start_exploration(agent_setup) -> None:
history = agent_setup(
blueprints=[
Expand All @@ -102,7 +102,7 @@ def test_start_exploration(agent_setup) -> None:
assert "explor" in history[-1].content.lower()


@pytest.mark.integration
@pytest.mark.slow
def test_go_to_semantic_location(agent_setup) -> None:
history = agent_setup(
blueprints=[
Expand Down
2 changes: 1 addition & 1 deletion dimos/agents/skills/test_unitree_skill_container.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def __init__(self):
self._bound_rpc_calls["GO2Connection.publish_request"] = lambda *args, **kwargs: None


@pytest.mark.integration
@pytest.mark.slow
def test_pounce(agent_setup) -> None:
history = agent_setup(
blueprints=[MockedUnitreeSkill.blueprint()],
Expand Down
10 changes: 5 additions & 5 deletions dimos/agents/test_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def add(self, x: int, y: int) -> str:
return str(x + y)


@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.parametrize("dask", [False, True])
def test_can_call_tool(dask, agent_setup):
history = agent_setup(
Expand Down Expand Up @@ -68,7 +68,7 @@ def register_user(self, name: str) -> str:
return "User name registered successfully."


@pytest.mark.integration
@pytest.mark.slow
@pytest.mark.parametrize("dask", [False, True])
def test_can_call_again_on_error(dask, agent_setup):
history = agent_setup(
Expand Down Expand Up @@ -120,7 +120,7 @@ def go_to_location(self, description: str) -> str:
return f"Going to the {description}."


@pytest.mark.integration
@pytest.mark.slow
def test_multiple_tool_calls_with_multiple_messages(agent_setup):
history = agent_setup(
blueprints=[MultipleTools.blueprint(), NavigationSkill.blueprint()],
Expand Down Expand Up @@ -174,7 +174,7 @@ def test_multiple_tool_calls_with_multiple_messages(agent_setup):
assert len(go_to_location_calls) == 2


@pytest.mark.integration
@pytest.mark.slow
def test_prompt(agent_setup):
history = agent_setup(
blueprints=[],
Expand All @@ -192,7 +192,7 @@ def take_a_picture(self) -> Image:
return Image.from_file(get_data("cafe-smol.jpg")).to_rgb()


@pytest.mark.integration
@pytest.mark.slow
def test_image(agent_setup):
history = agent_setup(
blueprints=[Visualizer.blueprint()],
Expand Down
14 changes: 8 additions & 6 deletions dimos/agents_deprecated/memory/test_image_embedding.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,13 @@
import numpy as np
import pytest
from reactivex import operators as ops
from reactivex.scheduler import ThreadPoolScheduler

from dimos.agents_deprecated.memory.image_embedding import ImageEmbeddingProvider
from dimos.stream.video_provider import VideoProvider


@pytest.mark.heavy
@pytest.mark.slow
class TestImageEmbedding:
"""Test class for CLIP image embedding functionality."""

Expand All @@ -45,6 +46,7 @@ def test_clip_embedding_initialization(self) -> None:

def test_clip_embedding_process_video(self) -> None:
"""Test CLIP embedding provider can process video frames and return embeddings."""
test_scheduler = ThreadPoolScheduler(max_workers=4)
try:
from dimos.utils.data import get_data

Expand All @@ -53,7 +55,9 @@ def test_clip_embedding_process_video(self) -> None:
embedding_provider = ImageEmbeddingProvider(model_name="clip", dimensions=512)

assert os.path.exists(video_path), f"Test video not found: {video_path}"
video_provider = VideoProvider(dev_name="test_video", video_source=video_path)
video_provider = VideoProvider(
dev_name="test_video", video_source=video_path, pool_scheduler=test_scheduler
)

video_stream = video_provider.capture_video_as_observable(realtime=False, fps=15)

Expand Down Expand Up @@ -146,6 +150,8 @@ def on_completed() -> None:

except Exception as e:
pytest.fail(f"Test failed with error: {e}")
finally:
test_scheduler.executor.shutdown(wait=True)

def test_clip_embedding_similarity(self) -> None:
"""Test CLIP embedding similarity search and text-to-image queries."""
Expand Down Expand Up @@ -205,7 +211,3 @@ def test_clip_embedding_similarity(self) -> None:

except Exception as e:
pytest.fail(f"Similarity test failed with error: {e}")


if __name__ == "__main__":
pytest.main(["-v", "--disable-warnings", __file__])
Loading
Loading