Get location and bounding box of objects in environment

[MudeLin]

How can I get the 2D location and 3D location of objects in the scene? For example, when the drone is flying on the road, can I get the 3D box of the cars or pedestrains on the road?

[sytelus]

Yes, you can but you will need to dig in to Unreal Engine programming. Basically environment has all these meshes so you can iterate through meshes that interests you and then there are direct methods to get its position and size in world coordinates. If you have never done any Unreal engine programming, its non-trivial, however. I would suggest looking at Unreal Engine programming guide or ask this question in their forums.

[MudeLin]

OK, thanks.

[cangokalp]

@Geekking Have you figured this out? I wish there was an easy way to get the coordinates and bounding box of the objects in the environment too. I need the same thing

[alessandro-montanari]

@Geekking looking for the same thing here.

[evroon]

Probably way too late, but for future reference: You can get bounding boxes without modifying the source code using the segmentation outputs.

First, set the object IDs of everything to zero, except the drone (if you want to get bounding boxes around the drone):

client.simSetSegmentationObjectID("[\w]*", 0, True)
client.simSetSegmentationObjectID("Drone[\w]*", 255, True)

Save the segmentation images as in the hello-drone example.

Now you can calculate the bounding box simply by checking which rows and columns contain non-zero pixels: (there is probably a more efficient way, but it works)

import numpy as np
import cv2

img = cv2.imread('path_to_image.png')

height, width = img.shape[:2]
img_size = np.array([width, height])
start_x, start_y = -1, -1
end_x, end_y = -1, -1

for y in range(height):
    if np.sum(img[y, :, :]) > 0.0:
        end_y = y

        if start_y == -1:
            start_y = y

for x in range(width):
    if np.sum(img[:, x, :]) > 0.0:
        end_x = x

        if start_x == -1:
            start_x = x

img = cv2.rectangle(img, (start_x, start_y), (end_x, end_y), (255, 0, 0))
cv2.imwrite('frame.png', img)

The (very thin, sorry) blue box shows the bounding box in the acquired segmentation image:

[rancheng]

Probably way too late, but for future reference: You can get bounding boxes without modifying the source code using the segmentation outputs.

First, set the object IDs of everything to zero, except the drone (if you want to get bounding boxes around the drone):

client.simSetSegmentationObjectID("[\w]*", 0, True)
client.simSetSegmentationObjectID("Drone[\w]*", 255, True)

Save the segmentation images as in the hello-drone example.

Now you can calculate the bounding box simply by checking which rows and columns contain non-zero pixels: (there is probably a more efficient way, but it works)

import numpy as np
import cv2

img = cv2.imread('path_to_image.png')

height, width = img.shape[:2]
img_size = np.array([width, height])
start_x, start_y = -1, -1
end_x, end_y = -1, -1

for y in range(height):
    if np.sum(img[y, :, :]) > 0.0:
        end_y = y

        if start_y == -1:
            start_y = y

for x in range(width):
    if np.sum(img[:, x, :]) > 0.0:
        end_x = x

        if start_x == -1:
            start_x = x

img = cv2.rectangle(img, (start_x, start_y), (end_x, end_y), (255, 0, 0))
cv2.imwrite('frame.png', img)

The (very thin, sorry) blue box shows the bounding box in the acquired segmentation image:

This will only work when there is no overlapping of the object, if anything obscured this object, 2D bounding box will be off.

[evroon]

@rancheng Yes that is true, that is exactly what someone noted here as well. So if you want bounding box detections where occlusion is taking place, you can use PR #3472.

[evroon]

But I think that in a lot of cases, the segmentation method suffices, and it is easier to work with, but indeed it has its limitations.

[zohaibjan]

@evroon that is exactly what I am trying to do but sort of struggling with the PR #3472. Is there a way to determine if objects are overlapped or occluded ?.

Thanks.