subprocess.Popen in manager.py looking for directories not specified by user

[wbuchanan]

$ python plotman.py plot 
...starting plot loop
Traceback (most recent call last):
  File "plotman.py", line 92, in <module>
    wait_reason = manager.maybe_start_new_plot(dir_cfg, sched_cfg, plotting_cfg)
  File "/home/billy/Desktop/plotman/manager.py", line 120, in maybe_start_new_plot
    p = subprocess.Popen(plot_args,
  File "/home/billy/anaconda3/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/billy/anaconda3/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'chia'

Here is what my config file looks like:

# Where to plot and log.
directories:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        log: /home/billy/.chia/mainnet/plotman

        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /media/billy/Data/chiatmp
                - /media/billy/Windows/chiatmp

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a

        # One or more directories; the scheduler will use all of them.
        # These again are presumed to be on independent physical devices,
        # so writes (plot jobs) and reads (archivals) can be scheduled
        # to minimize IO contention.
        dst:
                - /media/billy/chia-1-16TB/chiaplots
                - /media/billy/chia-2-16TB/chiaplots
                - /media/billy/chia-3-16TB/chiaplots
                - /media/billy/chia-4-16TB/chiaplots
                - /media/billy/chia-5-16TB/chiaplots

        # Archival configuration.  Optional; if you do not wish to run the
        # archiving operation, comment this section out.
        #
        # Currently archival depends on an rsync daemon running on the remote
        # host, and that the module is configured to match the local path.
        # See code for details.


# Plotting scheduling parameters
scheduling:
        # Don't run a job on a particular temp dir until all existing jobs
        # have progresed at least this far.  Phase major corresponds to the
        # plot phase, phase minor corresponds to the table or table pair
        # in sequence.
        tmpdir_stagger_phase_major: 3
        tmpdir_stagger_phase_minor: 4

        # Don't run more than this many jobs at a time on a single temp dir.
        tmpdir_max_jobs: 4

        # Don't run any jobs (across all temp dirs) more often than this.
        global_stagger_m: 15

        # How often the daemon wakes to consider starting a new plot job
        polling_time_s: 60


# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        k: 32
        e: True              # Use -e plotting option
        n_threads: 4         # Threads per job
        n_buckets: 128       # Number of buckets to split data into
        job_buffer: 6750     # Per job memory

I have no idea where or why this is looking for a directory named 'chia' when nothing I've specified has a directory name of 'chia'.

[altendky]

It is trying to run the chia command to start a plot. I think the general expectation is that you have activated the chia virtual environment such that you could manually run chia.

plotman/manager.py

Lines 104 to 123 in 349d13a

	plot_args = ['chia', 'plots', 'create',
	'-k', str(plotting_cfg['k']),
	'-r', str(plotting_cfg['n_threads']),
	'-u', str(plotting_cfg['n_buckets']),
	'-b', str(plotting_cfg['job_buffer']),
	'-t', tmpdir,
	'-d', dstdir ]
	if 'e' in plotting_cfg and plotting_cfg['e']:
	plot_args.append('-e')
	if 'tmp2' in dir_cfg:
	plot_args.append('-2')
	plot_args.append(dir_cfg['tmp2'])
	logmsg = ('Starting plot job: %s ; logging to %s' % (' '.join(plot_args), logfile))
	# start_new_sessions to make the job independent of this controlling tty.
	p = subprocess.Popen(plot_args,
	stdout=open(logfile, 'w'),
	stderr=subprocess.STDOUT,
	start_new_session=True)

[altendky]

Perhaps it would be worth catching this particular exception and reporting an explanatory and instructional message.

[wbuchanan]

I thought I was calling it from a shell where I did have the virtual environment activated, but can double check later (I kicked off a job manually to start plotting). I definitely think more informative exception messages could be useful.

[wbuchanan]

I've run the commands from within the virtual environment and get the same errors.

[wbuchanan]

Calling from the virtual environment I get a new error:

...starting plot loop
Warning: unrecognized args: -k32 -n1
Warning: unrecognized args: -t/media/billy/Data/chiatmp -2/media/billy/Data/chiatmp
Warning: unrecognized args: -d/media/billy/chia-1-16TB/chiaplots -b13500
Warning: unrecognized args: -u64 -r4
Warning: unrecognized args: -a2792509996 -e
Traceback (most recent call last):
  File "plotman.py", line 92, in <module>
    wait_reason = manager.maybe_start_new_plot(dir_cfg, sched_cfg, plotting_cfg)
  File "/home/billy/Desktop/chia-blockchain/manager.py", line 69, in maybe_start_new_plot
    jobs = job.Job.get_running_jobs(dir_cfg['log'])
  File "/home/billy/Desktop/chia-blockchain/job.py", line 74, in get_running_jobs
    jobs.append(Job(proc, logroot))
  File "/home/billy/Desktop/chia-blockchain/job.py", line 127, in __init__
    self.init_from_logfile()
  File "/home/billy/Desktop/chia-blockchain/job.py", line 133, in init_from_logfile
    assert self.logfile
AssertionError

I had plotman cloned to ~/Desktop and navigated there and tried things, and got an error. Then I copied the plotman directory into the directory where I have the chia stuff located ~/Desktop/chia-blockchain/plotman, tried running the command there and got an error. Then I tried moving the individual python scripts into ~/Desktop/chia-blockchain and tried to execute the script there and the trackback above is from that last attempt.

I think in addition to having more informative exception messages, it might be useful to have a bit more information available about the installations/set-up assumptions as well.

[ericaltendorf]

Completely agreed. Volunteers are actively working on improving the installation/setup process and assumptions.

The other error you're seing (unrecognized args) is plotman trying to make sense of a plotting job that was started not-by-plotman with different format command line. This is something it can't quite do yet, although there are some fixes coming that should make it more robust to that.

[altendky]

(eric beat me to submitting this, but since i had already typed it up... and i totally agree with you as well...)

The present tendencies around setting up Chia and plotman do seem messy and confusing to me. Chia is already available on PyPI and we're working towards plotman being available as well. At that point users won't need to involve git in the setup at all. Hopefully then we can provide tidier setup instructions.

Probably best not to be moving files around. Their absolute location isn't super relevant and definitely has nothing to do with being 'in the venv'.

As to those warnings, you seem to be hitting the issue being worked on over in #60. plotman has assumed that arguments would have spaces such as -k 32 rather than -k32 as it is finding in situation. I think the fix is good but we're trying to coordinate a few changes. I'll try to chat with the other devs about that. If you want to try that PR you can work with the jkbecker-fix41b branch at https://github.com/jkbecker/plotman.

Sorry for all the troubles.

[altendky]

FYI, I merged the argument-without-space fix into development.

[wbuchanan]

@altendky
I figured that and tried modifying the code to add a space in the definition but it was still removed.

[wbuchanan]

@ericaltendorf
I'm not sure that was the cause of the error since I didn't have any other plotting happening at that time.

[wbuchanan]

@altendky
It seems like whatever changes you made in development worked. Do you know if the chia GUI is able to detect plotting that is started using plotman? At the moment it doesn't seem like that is happening so I can't tell how many actual plots are being created at the moment, but it seems like it is only a single plot? If that is correct, is there a way to force multiple plots to be created in parallel?

[altendky]

I have no idea what the GUI does, sorry. If you want to do a manual check of what is running to compare with the GUI and plotman, there's ps aux | grep 'chia plots create'. plotman can be configured in various ways with per temp drive limits, global limits, phase staggered starts, time staggered starts, and i think more. Take a look through the example configuration. Maybe start a new ticket or check in on keybase if you need more help with that.

Is the original issue reported here still outstanding? Or has it been resolved.

[wbuchanan]

@altendky
Sorry I should have closed the ticket. It's running now.

[sweetcake2017]

@altendky
Sorry I should have closed the ticket. It's running now.

hi ,i try to working on WSL2. The same problem heppen to me , can you tell me how to fix this proble? thx.

Traceback (most recent call last):
  File "/home/cake/.local/bin/plotman", line 8, in <module>
    sys.exit(main())
  File "/home/cake/.local/lib/python3.8/site-packages/plotman/plotman.py", line 145, in main
    wait_reason = manager.maybe_start_new_plot(cfg.directories, cfg.scheduling, cfg.plotting)
  File "/home/cake/.local/lib/python3.8/site-packages/plotman/manager.py", line 130, in maybe_start_new_plot
    p = subprocess.Popen(plot_args,
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'chia'