In picking.py,
When the self.object_randomization flag is set to True, _reset_internal() also activate the self.hard_reset flag on.
The flag is used in environments/base.py:reset().
When self.hard_reset is on, the following code is called starting on L311:
if self.hard_reset and not self.deterministic_reset: #TODO: investigate increasing memory consumption when calling this every rollout
self._destroy_viewer()
self._load_model() # Create a manipulation task objec (arena/robot/object/placement of objects/goal objects)
self._postprocess_model()
self._initialize_sim()
When this is called at the end of every rollout, our memory usage increases significantly over the lifetime of the experiment, always resulting in an out of RAM segfault.
If we only randomize every 200 epochs we see this stabilizes, but we have not figured out the underlying cause.
In picking.py,
When the self.object_randomization flag is set to True,
_reset_internal()also activate theself.hard_resetflag on.The flag is used in environments/base.py:reset().
When
self.hard_resetis on, the following code is called starting on L311:When this is called at the end of every rollout, our memory usage increases significantly over the lifetime of the experiment, always resulting in an out of RAM segfault.
If we only randomize every 200 epochs we see this stabilizes, but we have not figured out the underlying cause.