bug: location of mf6 executable using relative path

[mbakker7]

I am stumbling with flopy to locate the mf6 executable on my Mac.

I define the location with a relative path, as per documentation (Relative path to MODFLOW 6 executable from the simulation working folder).

exe_name='../bin/mf6'

But when I run the model I get:

FileNotFoundError: [Errno 2] No such file or directory: '../bin/mf6'

When in the next code cell I do:

ls ../bin/

I get

mf6* mp7*

So mf6 clearly exist in the right location and is executable. I can even run it

!../bin/mf6

Gives

ERROR REPORT: 1. mf6: mfsim.nam is not present in working directory.

When I move this mf6 to the same directory as the notebook and change exe_name to

path = os.getcwd()
exe_name=path + '/mf6'

But then it is an absolute path.

My colleague claims that

exe_name='../bin/mf6.exe'

works on his Windows machine.

Is this a Mac problem?

[mbakker7]

Suggested solution:

In flopy.mf6.MFSimulation, replace self.exe_name = exe_name by os.path.abspath(exe_name).
(and import os)

[spaulins-usgs]

@mbakker7, I like your solution, though it should be implemented in mbase.py run_model, and only if the attempt to find the executable using shutil.which fails.

[spaulins-usgs]

@mbakker7, I added code that tries converting the executable path to an absolute path (os.path.abspath) when all else fails. Also, updated the comments for the documentation. The changes are checked in to the develop branch. Please let me know if this fixes the problem.

[mbakker7]

Sorry, @spaulins-usgs, still doesn't work. My relative path is ../bin/mf6. When I specify that, I get the error message FileNotFoundError: [Errno 2] No such file or directory: '../bin/mf6'. The simulation runs fine when I specify exe_name=os.path.abspath('../bin/mf6').

Is there a way to ask flopy what executable (full path) it will use, so I can check where it goes wrong?

[spaulins-usgs]

@mbakker7, to output the executable path being used try setting the silent flag to False:

sim.run_simulation(silent=False)

I am not 100% sure what is wrong and I do not have a Mac to test this, though I do have some ideas. If you are willing to test these, there are some potential fixes on my fork of flopy here:

flopy

The only modified file in this fork is flopy/mbase.py

Please let me know if these updates fix the issue.

[mbakker7]

@spaulins-usgs, sorry for the late reply. I was on break for a while, but I am back and determined to help fix this issue.

The message I get when running modflow is: FloPy is using the following executable to run the model: ../bin/mf6. Now that is indeed the path I provide. When I run ls ../bin/mf6, I nicely get back ../bin/mf6*, so the file exist. And the previously discussed use of os.path.abspath works fine, so the error must be somewhere in mfbase, where the executable path is stored. Surely other developers (@jdhughes , @langevin-usgs) must run into similar problems as I thought they use Macs.

For your information, the full trace back is:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [5], in <cell line: 90>()
     88 # Write input files and solve
     89 sim.write_simulation(silent=True)
---> 90 success, _ = sim.run_simulation(silent=False) 
     91 if success == 1:
     92     print('Model solved successfully')

File ~/git/flopy/flopy/mf6/modflow/mfsimulation.py:1467, in MFSimulation.run_simulation(self, silent, pause, report, normal_msg, use_async, cargs)
   1465     else:
   1466         silent = True
-> 1467 return run_model(
   1468     self.exe_name,
   1469     None,
   1470     self.simulation_data.mfpath.get_sim_path(),
   1471     silent=silent,
   1472     pause=pause,
   1473     report=report,
   1474     normal_msg=normal_msg,
   1475     use_async=use_async,
   1476     cargs=cargs,
   1477 )

File ~/git/flopy/flopy/mbase.py:1743, in run_model(exe_name, namefile, model_ws, silent, pause, report, normal_msg, use_async, cargs)
   1740         argv.append(t)
   1742 # run the model with Popen
-> 1743 proc = Popen(argv, stdout=PIPE, stderr=STDOUT, cwd=model_ws)
   1745 if not use_async:
   1746     while True:

File ~/opt/anaconda3/lib/python3.9/subprocess.py:951, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask)
    947         if self.text_mode:
    948             self.stderr = io.TextIOWrapper(self.stderr,
    949                     encoding=encoding, errors=errors)
--> 951     self._execute_child(args, executable, preexec_fn, close_fds,
    952                         pass_fds, cwd, env,
    953                         startupinfo, creationflags, shell,
    954                         p2cread, p2cwrite,
    955                         c2pread, c2pwrite,
    956                         errread, errwrite,
    957                         restore_signals,
    958                         gid, gids, uid, umask,
    959                         start_new_session)
    960 except:
    961     # Cleanup if the child failed starting.
    962     for f in filter(None, (self.stdin, self.stdout, self.stderr)):

File ~/opt/anaconda3/lib/python3.9/subprocess.py:1821, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session)
   1819     if errno_num != 0:
   1820         err_msg = os.strerror(errno_num)
-> 1821     raise child_exception_type(errno_num, err_msg, err_filename)
   1822 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: '../bin/mf6'

[mbakker7]

I think I found the problem.

In mfbase, you try the following:

    exe = which(exe_name)
    if exe is None:
        if exe_name.lower().endswith(".exe"):
            # try removing .exe suffix
            exe = which(exe_name[:-4])
    if exe is None:
        # try abspath
        exe = which(os.path.abspath(exe_name))

When a relative path is passed to the which function, it happily returns the same relative path. Which is not what you want (on a Mac), as it needs to be an absolute path.

from shutil import which
which('../bin/mf6')

returns

'../bin/mf6'

Hence, exe is never None, and the absolute path is never set. Now this may work different on Windows. Or simply running an executable with a relative path may work on Windows, I don't know. But on a Mac running with a relative path doesn't work. Hope this helps.

[jlarsen-usgs]

@mbakker7

I just reproduced this same error on windows. When I supply a relative path without the extension (some_rel_path\mp7) flopy crashes ( with modpath7), however if I supply the extension (some_rel_path\mp7.exe on windows) it runs. Can you test this quirk on mac? It seems like we need to identify the platform and provide the extension prior to calling which()

[mbakker7]

I can confirm the same error on Windows: If you specify a relative path you must also specify '.exe'. This is odd, as flopy just removed the requirement to add .exe. But with a relative path, it is still required.

An executable program on a Mac doesn't have an extension but a special mode flag that the OS recognizes.

May I (re)suggest to always use os.path.abspath in flopy on the provided path? What is the downside to always use os.path.abspath?

[spaulins-usgs]

@mbakker7, I think it would be fine to use os.path.abspath as long as the exe_name supplied is more than just an executable name (i.e. it is either an absolute path or contains at least one folder name in it). However, if just an executable name is supplied, FloPy should resolve the full path to the executable by looking for the executable in the Windows/Mac defined paths (Windows/Mac "PATH" variable). The which command will not do this with an absolute path. For example, on Windows which(os.path.abspath("MF6.exe")) will not search the "PATH" variable paths for "MF6.exe", but which("MF6.exe") will. Therefore, I think the behavior should be:

  if <exe_name is just a file name>:
        exe = which(exe_name)
  else:
        exe = which(os.path.abspath(exe_name))

[spaulins-usgs]

@mbakker7, I implemented the behavior described in my post above here:

https://github.com/scottrp/flopy/blob/develop/flopy/mbase.py

Let me know if that works for you.

[mbakker7]

Still won't work, because os.path.dirname('../bin/mf6') gives '../bin' and not an empty string.

I understand you want to use which to search the path. What I suggest is to remove

   if os.path.dirname(exe_name) == "":
         exe = which(exe_name)
         if exe is None:
             if exe_name.lower().endswith(".exe"):
                 # try removing .exe suffix
                 exe = which(exe_name[:-4])
                 if exe is not None:
                     exe_name = exe_name[:-4]
     else:
         exe = os.path.abspath(exe_name)

and replace it with the simple

os.path.abspath(shutil.which(exe_name))

Not sure what to do with the exe extension. You can remove that on Windows if you want (I cannot test that). Will this work for you? It works for me!