Play back trial from within Docker container instead of host machine

[mabelzhang]

Addresses Infrastructure bullet 1 in #20.

I completely rewrote the script for trial playback and video recording, so that the trial recording state.log is now played back from within the server Docker container, instead of on the host machine.

This was necessary, because during evaluation rehearsal, we found that when the log was being played back on the host machine, we had to do a hack of replicating the Docker container's /vorc_ws/install/cora_description directory structure on the host machine, in order for the robot to show up in Gazebo. Other environment configurations were probably also different between host and the server container depending on the evaluator.

Both the playback and video recording work quite well for me now. There were some kinks two days ago, but I've fixed all of them. Everything is autopilot.

I'm really happy about this fix. Hope this works for other people too.

To test:

Preliminaries:

# Rebuild Docker image
./vorc_server/build_image.bash -n
# Prepare teams and tasks
./multi_scripts/prepare_all_teams.bash 
./multi_scripts/prepare_all_task_trials.bash
# Run some task of your choice
./run_trial.bash -n ghostship wayfinding 0

Test playing back log file:

./replay_trial.bash -n ghostship wayfinding 0

Test playing back log file and generate screen recording (this calls replay_trial.bash internally):

./generate_trial_video.bash -n ghostship wayfinding 0

[mabelzhang]

At some point in the future, this will probably want to be backported for VRX (master branch). I can create a followup issue for bookkeeping. It won't be needed until the next VRX, which is a long way away. I could backport it this month if the context helps, otherwise I'll leave it.

[caguero]

What is the condition that triggers the playback termination? I'm pausing the playback to check a few things but the playback shutdowns eventually.

[caguero]

I made one minor comment, but overall this works great and it's way better than what we had. Good job!

[caguero]

At some point in the future, this will probably want to be backported for VRX (master branch). I can create a followup issue for bookkeeping. It won't be needed until the next VRX, which is a long way away. I could backport it this month if the context helps, otherwise I'll leave it.

That would be great now that you have everything fresh.

[mabelzhang]

What is the condition that triggers the playback termination? I'm pausing the playback to check a few things but the playback shutdowns eventually.

Haha. The way that the scripts detects log playback finished is by checking Gazebo is paused again - initially the world starts paused, then the script unpauses to start the playback, and when playback is done, Gazebo becomes paused again. But yeah if the evaluator wants to pause to check something, it will trigger the script into thinking "playback is done" and it'll go shut down everything.

Is there another way of knowing if log playback is finished?

[caguero]

What is the condition that triggers the playback termination? I'm pausing the playback to check a few things but the playback shutdowns eventually.

Haha. The way that the scripts detects log playback finished is by checking Gazebo is paused again - initially the world starts paused, then the script unpauses to start the playback, and when playback is done, Gazebo becomes paused again. But yeah if the evaluator wants to pause to check something, it will trigger the script into thinking "playback is done" and it'll go shut down everything.

Is there another way of knowing if log playback is finished?

That's fine, I just forgot which was the termination condition. We can always check the video and pause if needed :)

[mabelzhang]

There is an alternative currently, if the evaluator wants to take a close look. It just requires manual shut down of Gazebo through an injection into the container, instead of autopilot. I can add it to the README before merging this PR (if I don't instantly forget after posting this comment). The default behavior is autopilot because the shut downs can get annoying if they aren't done cleanly.

You can pass in --keep-gz from either of the top-level scripts, generate_trial_video.bash or replay_trial.bash:

vrx-docker/generate_trial_video.bash

Line 23 in 2328c80

source ${DIR}/replay_trial.bash --keep-docker --manual-play "$@"

It will be picked up by the script play_vorc_log.sh running inside the container:

vrx-docker/vorc_server/vorc-server/play_vorc_log.sh

Lines 23 to 24 in 2328c80

	echo "--keep-gz: Keep Gazebo server and client running after playback ends."
	echo " By default, they are shut down automatically."

When this is issued, Gazebo won't shut down automatically when playback is paused.
Then you just have to shut down Gazebo manually by issuing this injection when you're done:

docker exec -it vorc-server-system killall -w gzclient gzserver

This is the same way the script shuts it down, so it should be clean.

The internal script will still wait around and clean up ROS nodes, roslaunch, etc., properly after it detects the manual shut down in wait_until_gzserver_is_down:

vrx-docker/vorc_server/vorc-server/play_vorc_log.sh

Lines 141 to 147 in 2328c80

	if [ $kill_gz -eq 1 ]; then
	wait_until_playback_ends
	killall -w gzserver gzclient
	else
	echo "Waiting for Docker injection to terminate Gazebo..."
	wait_until_gzserver_is_down
	fi

The script should still finish cleanly and you shouldn't have to kill the container yourself. Let me know if you try and something doesn't shut down cleanly.

[mabelzhang]

README updated with --keep-gz option to pause for inspection.

That would be great now that you have everything fresh.

Okay. Will backport to VRX in master branch (updated #20 to track this).