models/ refactor

[leshy]

• adds ignore missing imports for ros/mujoco

• writes consistent type spec for models
  visual language models models/vl/base.py
  embedding models models/embedding/base.py
  local models models/base.py
  huggingface models models/base.py

classDiagram
    class LocalModel {
        +device
        +dtype
        +_model
    }

    class HuggingFaceModel {
        +model_name
        +from_pretrained()
    }

    class Captioner {
        <<ABC>>
        +caption(image)
    }

    class VlModel {
        +query(image, query)
        +query_json()
        +query_detections()
    }

    class EmbeddingModel~E~ {
        <<ABC>>
        +embed(*images)
        +embed_text(*texts)
    }

    class Embedding {
        +vector
        +to_numpy()
        +to_torch()
    }

    LocalModel <|-- HuggingFaceModel

    Captioner <|-- VlModel

    Captioner <|-- Florence2Model
    HuggingFaceModel <|-- Florence2Model

    EmbeddingModel <|-- CLIPModel
    HuggingFaceModel <|-- CLIPModel

    EmbeddingModel <|-- MobileCLIPModel
    LocalModel <|-- MobileCLIPModel

    VlModel <|-- Qwen : remote API
    VlModel <|-- Moondream : local GPU
    VlModel <|-- MoondreamHosted : remote API

    EmbeddingModel ..> Embedding : produces

Loading

• comprehensive parametric tests for all vl/ embedding/ and captioning/ models (you don't write your own tests for a new module, just add it to a list)

• ensures models are resources, so have start() and stop() methods that clean up cuda resources, adds optional argument for warmup when initializing

• adds methods for batch querying of models (for speed)

• adds optional auto-resize of input images to models (for speed)

• adds florence2 fast captioning model (implements captioning model type, all vlms are captioners, not all captioners are vlms)

• adds torch reid models to lfs

• adds mobileclip models to lfs

• cleans up dead code (models/segmentation, pointcloud, labels, qwen)

• adds a quick cli graphing system at dimos/utils/cli/plot.py for profiling

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[greptile-apps]

Greptile Overview

Greptile Summary

This PR delivers a comprehensive refactoring of the models architecture, establishing a clean type hierarchy for vision-language models, embedding models, and captioning models. The refactor introduces consistent Resource lifecycle management with start() and stop() methods for proper CUDA memory cleanup.

Key improvements:

• Unified base classes (LocalModel, HuggingFaceModel) with proper device/dtype configuration
• Clear separation of concerns: Captioner ABC for captioning, VlModel for vision-language queries, EmbeddingModel for embeddings
• Performance optimizations: batch query methods, auto-resize for faster inference, warmup support
• New Florence-2 fast captioning model and MobileCLIP embedding model
• Comprehensive parametric test suite covering all model types
• CLI plotting utility for profiling model performance
• Dead code cleanup (segmentation, pointcloud, labels modules)

Architecture:
The class diagram in the PR description accurately reflects the new hierarchy. VLMs inherit from both Captioner and Resource, while embedding models follow a generic pattern with typed Embedding subclasses.

Integration:
Changes to detection types (adding is_valid() method, fixing typing to float for pixel coordinates) ensure proper integration with VLM detection queries. The reid system continues to work correctly with the refactored embedding models.

Confidence Score: 5/5

• This PR is safe to merge with excellent code quality and comprehensive test coverage
• This is a well-executed refactor with clean architecture, comprehensive tests, proper resource management, and minimal breaking changes. The code follows best practices with ABCs, cached properties, proper typing, and thorough documentation. All model integrations have been verified through parametric tests.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
dimos/models/base.py	5/5	Clean base class refactor introducing LocalModel and HuggingFaceModel with proper Resource lifecycle management
dimos/models/vl/base.py	5/5	Introduces Captioner ABC and VlModel with auto-resize support and batch query methods
dimos/models/embedding/base.py	5/5	Well-designed EmbeddingModel ABC with Embedding class supporting efficient GPU/CPU tensor operations
dimos/models/vl/florence.py	5/5	New Florence-2 captioning model with efficient batch processing and multiple detail levels
dimos/models/embedding/clip.py	5/5	CLIP model refactored to use new base classes with proper resource cleanup
dimos/models/embedding/mobileclip.py	5/5	MobileCLIP implementation using open_clip with local model weights from LFS
dimos/models/vl/moondream.py	5/5	Moondream VLM with query_multi optimization and auto-resize default at 512x512

Sequence Diagram

sequenceDiagram
    participant App
    participant VlModel
    participant LocalModel
    participant Resource
    participant GPU

    Note over App,GPU: Model Initialization
    App->>VlModel: __init__(auto_resize=(512,512))
    VlModel->>LocalModel: super().__init__()
    LocalModel->>LocalModel: Check autostart/warmup
    
    Note over App,GPU: Resource Lifecycle
    App->>VlModel: start()
    VlModel->>LocalModel: start()
    LocalModel->>LocalModel: Access _model @cached_property
    LocalModel->>GPU: Load model to device
    GPU-->>LocalModel: Model loaded
    VlModel->>VlModel: Run warmup query
    
    Note over App,GPU: Query Processing
    App->>VlModel: query(image, "What is this?")
    VlModel->>VlModel: _prepare_image() - resize if needed
    VlModel->>GPU: Forward pass
    GPU-->>VlModel: Response
    VlModel-->>App: String response
    
    Note over App,GPU: Batch Query Optimization
    App->>VlModel: query_batch([img1, img2, img3], "Describe")
    VlModel->>VlModel: Prepare all images
    VlModel->>GPU: Batched forward pass
    GPU-->>VlModel: Batch responses
    VlModel-->>App: List of responses
    
    Note over App,GPU: Cleanup
    App->>VlModel: stop()
    VlModel->>LocalModel: stop()
    LocalModel->>LocalModel: Delete _model from __dict__
    LocalModel->>GPU: torch.cuda.empty_cache()
    GPU-->>LocalModel: Memory freed

Loading

[greptile-apps]

_{37 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[leshy]

@greptileai rereview please, also don't include this sequence diagram please

[greptile-apps]

Additional Comments (1)

1. dimos/models/vl/moondream.py, line 154-156 (link)

   logic: query_detections() converts normalized coords to pixels using original image dimensions, but _to_pil() may have resized the image via auto_resize. When auto_resize is active, moondream returns coords normalized to the resized image, so scaling them back to the original dimensions will be incorrect.

   Use the resized dimensions from the PIL image instead:

_{37 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[leshy]

Additional Comments (1)

1. `dimos/models/vl/moondream.py`, line 154-156 ([link](/dimensionalos/dimos/blob/5381ff33e3aa1fb1a993fcbb6d72ab5f4e269966/dimos/models/vl/moondream.py#L154-L156))
   **logic:** `query_detections()` converts normalized coords to pixels using original image dimensions, but `_to_pil()` may have resized the image via `auto_resize`. When auto_resize is active, moondream returns coords normalized to the **resized** image, so scaling them back to the original dimensions will be incorrect.
   Use the resized dimensions from the PIL image instead:

37 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

@greptileai this is not correct, moondream returns normalized (0.0-1.0) coordinates so we don't need to take scaling factor into account when generating detections

[greptile-apps]

You're absolutely right! I made an error in my analysis.

Moondream's detect() method returns normalized coordinates (0.0-1.0) regardless of the input image size, so the coordinates should indeed be scaled directly to the original image dimensions without considering any resizing factor.

The current implementation in query_detections() is correct:

pil_image = self._to_pil(image)  # May resize image
result = self._model.detect(pil_image, query, settings=settings)
# result contains normalized coords (0.0-1.0)

# Scale normalized coords to original image dimensions - this is correct
detection = Detection2DBBox(
    bbox=(
        result["x1"] * image.width,   # Scale to original width
        result["y1"] * image.height,  # Scale to original height  
        result["x2"] * image.width,
        result["y2"] * image.height,
    ),
    # ...
)

Since Moondream normalizes its output coordinates, they represent the same relative positions regardless of input image size, so scaling them to the original image dimensions is the right approach. Thank you for the correction!

[leshy]

You're absolutely right! I made an error in my analysis.

@greptileai so why does your review score my moondream implementation 3/5 still, can u update

[greptile-apps]

_{37 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[paul-nechifor]

Should this be uncommented?