Visual prompts inference

[wudiabin]

1.Regarding the issue of cross-image detection with visual prompts. If I want to use visual prompts from img1 to detect img2 (with the refer_image and inference_images separated), is this possible, or will it cause problems?
2.If I want to provide both positive and negative visual prompts to complete reasoning together and improve accuracy, is that supported?

[jerpelhan]

Hi!

1. Yes, this is possible — I tested it myself and it works. The required change is fairly simple: you just need to store the prototypes. I already answered this in cross-image prompt or batch inferring #2.
   In short, you can run a forward pass on an image containing the exemplar as usual, and in GECO2/models/counter_infer.py store both hq_prototype_embeddings and prototype_embeddings inside the class (e.g., as self.hq_prototype_embeddings, etc.).
   After that, you can reuse them as “external” exemplars in all subsequent processing. I’ve used this approach in practice and it works well.
   You can also check issue cross-image prompt or batch inferring #2 — someone posted code there, although I haven’t reviewed it myself, it seemed to work.

2. Unfortunately, negative visual prompts are not supported at the moment. As a possible workaround, you could try running two inferences and then filtering highly overlapping bounding boxes, keeping the one with the higher predicted score.

[wudiabin]

Thank you very much for your reply.

1. I noticed a very strange phenomenon during the experiment: targets that clearly didn't resemble my visual prompts were still being incorrectly detected. For example, the circle in the image below.
2. I want to know how visual prompts are used in networks, and why irrelevant targets can still be detected.
3. I'm not sure if 'threshold' is the confidence score that I understand. Regarding confidence scores, I found that it doesn’t seem to work the way I thought, where the more similar it is to the visual prompt, the higher the confidence score. Sometimes, there are cases where it’s very similar to the visual prompt, but I need to lower the confidence score to a relatively low level to detect it. On the contrary, sometimes items that are very different from the visual prompt are more easily falsely detected.
4. I noticed that our network predicts both the BBox and the mask at the same time. If I only need the BBox, can I detect only the BBox without the mask? In that case, would it improve speed, or would it cause issues and affect detection accuracy?
5. I found two pretrained models, CNTQG_multitrain_ca44.pth and GECO2FSCD.pth. What is the difference between these two models? When I want to directly use the model you provided for prediction inference with visual prompts (without training), which model might be more practical?

[jerpelhan]

Hi! Sorry for the late reply, I’ve been quite busy.

1. Yes, I’m aware of this behavior -- the method is not perfect yet, and we’re still improving it. In cases like the circle you mentioned, simple postprocessing can help remove such outliers. For example, filtering by bounding box aspect ratio or area could already eliminate detections that clearly don’t match your expected target shape.

2. I recommend reading the GeCo and GECO2 papers for a deeper understanding of how visual prompts are used in the network:
   GeCo (NeurIPS 2024): https://neurips.cc/virtual/2024/poster/93724
   GECO2: https://arxiv.org/pdf/2511.08048

3. The threshold you are adjusting is not a pure confidence score. It is a combination of the SAM score and the GECO2 score. If you want a more confidence-like behavior, you can remove the SAM score and use only the GECO2 score.

4. Yes, you can use only the bounding box predictions without the mask. This will improve speed. Detection presence (i.e., which objects are detected) should remain the same, but localization accuracy may decrease slightly since the mask branch helps refine the boxes.

5. I would recommend CNTQG_multitrain_ca44.pth, since it was trained on a larger and more diverse dataset. For direct inference with visual prompts (without additional training), it is generally the more practical choice.

By the way, which model did you use to obtain the result you attached?

[wudiabin]

Thank you very much for your reply.

1. Regarding the performance of the two models CNTQG_multitrain_ca44.pth and GECO2FSCD.pth, I found that neither model is necessarily better overall. In some scenarios, CNTQG_multitrain_ca44.pth may perform better, while in other scenarios, GECO2FSCD.pth might perform better.（For dense object detection, GECO2FSCD.pth may perform better.）
2. Regarding the differences between the two models CNTQG_multitrain_ca44.pth and GECO2FSCD.pth, are their model architectures the same, with the only difference being the training dataset? Or are there differences in the model structures as well? I noticed that the emphasis was only on the differences in the training dataset for CNTQG_multitrain_ca44.pth.
3. Does the model support multiple reference images, meaning that different visual prompts come from different images, rather than multiple visual prompts from a single image?
4. I am still a bit unclear about the deviation and false detection in the test results. Why might objects that look very different from my visual prompts have relatively high responses or confidence (box_v)? What causes this, why does the model predict this behavior, and what exactly is this response? Clearly dissimilar objects have high confidence (I am not using SAM right now). Could you help clarify my doubts?