Backport #26, #28, #30 from VORC to VRX

[mabelzhang]

• Backport Play back trial from within Docker container instead of host machine #26 from VORC to VRX
• Backport Adding compressed image topics #28 and Enable NVIDIA runtime for competitor container #30 from VORC to VRX (very small fixes)
• Backport robot_locked:=true in launch command from VORC to VRX. Needed to eliminate errors like:

  [Err] [scoring_plugin.cc:235] Unable to release [wamv_external_pivot_joint]
  [Err] [scoring_plugin.cc:235] Unable to release [wamv_external_riser]
  

  Note for future: This is likely a patch, as only the stationkeeping task should really lock the vehicle. scoring_plugin.cc probably should be changed so that it does not attempt to release the locking joints, which are nonexistent for other tasks.

To test:

# Remove if exists
docker network rm vorc-network

# Rebuild image
./vrx_server/build_image.bash -n
# Prepare team
./prepare_team_wamv.bash example_team

# Change duration of stationkeeping0.world to 60 seconds to shorten test time.
vim generated/task_generated/stationkeeping/worlds/stationkeeping0.world 

# Run some task
./run_trial.bash -n example_team stationkeeping 0

# Test log playback
./replay_trial.bash -n example_team stationkeeping 0
# Test log recording
./generate_trial_video.bash -n example_team stationkeeping 0

[M1chaelM]

@mabelzhang I'm getting stuck on

./run_trial.bash -n example_team stationkeeping 0

For me this launches the competitor image and then hangs forever. Could you confirm it's working on your end?

[mabelzhang]

It'll take me a while to get the environment set up again. Quick check - does it work on master for you?

[M1chaelM]

It'll take me a while to get the environment set up again. Quick check - does it work on master for you?

It does! Hm, so I guess the problem is not with the competitor image.

[mabelzhang]

Maybe unrelated, I just realized why it says vorc-network. I think it is not a typo. It was needed because if we were running vorc before that, that network would exist and be hogging the port, possibly the 127.0.0.1. Then the vrx-network will not be able to be established on that port. Something like that.

[M1chaelM]

Seems like something has gone wrong in the new Docker server image. If I build the image on the master branch, switch to this branch and run a trial without rebuilding, it works. Looking at the Dockerfile, the only non-trivial modifications that could affect running the trial are the new / updated scripts:

• run_vrx_trial.sh
• gz_utils.sh

[M1chaelM]

OK, I think I found it. The basic problem was just that sandisland.launch has been moved out of vrx_gazebo to vrx_2019 so the launch command in run_vrx_trial.sh was failing. This was confusing because it worked using the master branch, which should not be the case since the change is in the vrx repo.

The reason this was happening has to do with Docker's caching behavior and the way we are building the image. Since we had to modify an early layer in the image when we added psmisc, Docker produced a whole new version of the image for the branch. This included building the vrx repo again, which caused us to get the latest version in the new image. But, when I switched back the master branch, removing the psmisc line caused Docker to use a cached version for everything. That cached version had the old version of vrx in it.

The crux of the problem is that Docker can't tell when a remote repository like vrx needs to be downloaded and rebuilt again, and this can lead to confusing behavior. We might want to change the way we are doing this at some point.

[M1chaelM]

The trial now runs and playback and recording work, but the wamv is spawning way off the end of the world. This may be a separate issue.

[M1chaelM]

This may be a separate issue.

As suspected, this is a separate problem. We have updated to use vrx.launch in this branch, but this branch doesn't include practice worlds in the Sydney Regatta, so the WAMV is set in the wrong place. This should be OK and I don't think it needs to be fixed before merging.

[M1chaelM]

See my comments above; I think this is now working as expected and can be merged.

[mabelzhang]

Thank you for reviewing and testing!!

Nice find re the Docker layer! Yeah we encounter that with Docker and building repos from source.
One safe way is to specify --no-cache after the docker build command, and let the developer remove it. If they don't remember to remove it, then it'll just always take forever to build.
I'm lazy, so I don't specify it in the repo, and instead add the --no-cache manually when something is fishy.