Problems setting up shared memory with a Python process

[perrygreenfield]

I have tried unsuccessfully to share memory between a Python process and a Julia process on MacOS. I have been successful at doing so between two Python processes as well as Python and a different language. But the result of doing so with Julia results in a "ERROR: stat returned zero type for a valid path". From a search, it is suggested the problem arises from the fact that fstat does not work for shared memory file descriptors. The julia code tries to obtain the size of the shared memory from this file descriptor and it fails as a result. It was also suggested that the necessary information should be obtained by using the structure obtained from the shmat() system call.

When I say "suggested" I only see this explanation on an AI-generated response so I'm not sure of the validity. Here is the text that was generated:

Why does fstat not work with shared memory file descriptors on macOS?

On macOS, the fstat system call does not work with shared memory file descriptors. 
This is because shared memory file descriptors are not actual files, but rather a way
to access a shared memory segment. The fstat system call is designed to provide
information about a file, and shared memory file descriptors do not fit this model.

Instead, you can use the shmat system call to attach to a shared memory segment and then use 
the shmctl system call to get information about the shared memory segment. For example:

int shmid = shmget(key, size, IPC_CREAT | IPC_EXCL);
if (shmid == -1) {
    perror("shmget");
    exit(1);
}

int *ptr = shmat(shmid, NULL, 0);
if (ptr == (void *)-1) {
    perror("shmat");
    exit(1);
}

struct shmid_ds shmid_ds;
shmctl(shmid, IPC_STAT, &shmid_ds);
printf("Shared memory segment ID: %d\n", shmid_ds.shmid);
printf("Shared memory segment size: %d bytes\n", shmid_dsshm_segsz);
This code creates a new shared memory segment, attaches to it, and then uses the shmctl system call
to get information about the shared memory segment.

Note that there appears to be a typo in the last printf statement!

I create the shared memory in Python as follows:

from multiprocessing import shared_memory as shmem
sh = shmem.SharedMemory('shm', True, 1024) # The python package prepends a slash to the name internally
buffer = shc.buf # This byte array can be modified to demonstrate changes appear in the other process

I do this in julia:

using InterProcessCommunication
sh = ShareMemory("/shm")

Which produces this:

ERROR: stat returned zero type for a valid path
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] stat(fd::RawFD)
   @ Base.Filesystem ./stat.jl:156
 [3] stat
   @ ./stat.jl:168 [inlined]
 [4] filesize
   @ ./stat.jl:461 [inlined]
 [5] SharedMemory(name::String, flags::Int32, mode::UInt16, len::Int64, volatile::Bool)
   @ InterProcessCommunication ~/.julia/packages/InterProcessCommunication/rpDdn/src/shm.jl:153
 [6] #SharedMemory#24
   @ ~/.julia/packages/InterProcessCommunication/rpDdn/src/shm.jl:120 [inlined]
 [7] SharedMemory(name::String)
   @ InterProcessCommunication ~/.julia/packages/InterProcessCommunication/rpDdn/src/shm.jl:116
 [8] top-level scope
   @ REPL[2]:1

[emmt]

I have tried your example (beware you have some typos) on Linux: it works fine. I do not have a Mac but I will see how to use shmctl to retrieve the size of the shared memory segment.

[perrygreenfield]

Thanks. I was competing with the AI for typo supremacy.

[emmt]

I have inspected the code in the package, calling shmctl as you suggested is for System V shared memory, not for POSIX shared memory. In fact, for System V shared memory, shmcl is already called to get the size. The MacOS seems to not support completely POSIX shared memory (which is a shame). If you can figure out the size by other means, a workaround can be to build the SharedMemory structure specifically (but this is bad). If you can make two Python processes communicate via POSIX shared memory, this means that they have found a way to fix this issue and we may use their solution in the Julia package. Unfortunately I do not have a Mac to test that.

You may use System V shared memory which, to my knowledge, used to work more correctly (except for the automatic deletion on last detach). However note that System V and POSIX shared memory segments can coexist in a system but that they are totally independent and have no interoperability.

Another possibility is to use memory mapped which is built-in Julia and certainly also in Python.