Nanny reporting errors - may be working anyway?

[keflavich]

I'm seeing a slew of messages like this in a log file:

Traceback (most recent call last):
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/tornado/ioloop.py", line 905, in _run
    return self.callback()
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/distributed/nanny.py", line 417, in memory_monitor
    process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 8>>

This is on a supercomputer with slurm running job control. These messages only appear if I run the job in batch mode, not if I run them interactively.

The code I'm using is here: https://github.com/ALMA-IMF/reduction/blob/master/analysis/cube_stats_grid.py

A dask LocalCluster is started with:

                cluster = LocalCluster(n_workers=1,
                                       threads_per_worker=int(nthreads),
                                       memory_target_fraction=0.95,
                                       memory_limit=memlimit)
                client = Client(cluster)

after which the actual code being run is handled by spectral-cube

versions:

dask==2021.1.1
dask-image==0.6.0
dask-jobqueue==0.7.2

Is this a bug that I can fix? Is there a workaround to silence these error messages? I can't tell if the code is running successfully or not both because there are too many errors to search through and because the code is getting killed for different reasons.

[jrbourbeau]

Thanks for reporting @keflavich! It looks like #4733 probably fixed this issue. Could you try upgrading dask and distributed to the latest released versions (2021.09.1) to see if the issue still persists?

[keflavich]

Huh, I thought I already had upgraded distributed, but I did pip freeze | grep dask so I didn't see it. Indeed, it appears I'm still on 2020.12.0. pip install --upgrade boosted me to distributed-2021.3.0, which looks older than the latest released version? I suspect that's because I'm using an older python:

$ pip install git+https://github.com/dask/distributed.git#egg=distributed
Collecting distributed
  Cloning https://github.com/dask/distributed.git to /scratch/local/9307298/pip-install-hx_ucvf9/distributed_98a40bcd418f479c8e170f36f5fa7567
Requirement already satisfied: click>=6.6 in /blue/adamginsburg/adamginsburg/miniconda3/envs/casa61_py36/lib/python3.6/site-packages (from distributed) (7.1.2)
Requirement already satisfied: cloudpickle>=1.5.0 in /blue/adamginsburg/adamginsburg/miniconda3/envs/casa61_py36/lib/python3.6/site-packages (from distributed) (1.6.0)
ERROR: Could not find a version that satisfies the requirement dask==2021.09.1 (from distributed)
ERROR: No matching distribution found for dask==2021.09.1

I'm trying again with a py3.9 environment, which installed this version: 2021.09.1+26.gcf018a19

So I think we can call this closed - I just needed a more recent version of dask, which meant I needed to use a more recent (>3.6) version of python.