Interfaces for writing to files using mmap#237
Interfaces for writing to files using mmap#237pranav1344 wants to merge 4 commits intoMPLLang:mainfrom
Conversation
|
I've added writing to files using records as well as an example for writing in parallel to Running a primitive test on this write functionality using this code by generating 1GB arrays containing just a single letter gives this result:
|
…mmap and virtual memory changes to support the interfaces.
|
I'm not too sure if this part is too convoluted, but I wanted it to have parity with the read functionality, which supports reading from a certain offset. |
basis-library/mpl/file.sig
Outdated
| val readChars: t -> int -> char ArraySlice.slice -> unit | ||
| val readWord8s: t -> int -> Word8.word ArraySlice.slice -> unit | ||
| val writeChar : {file: t, file_offset: int, array_slice_offset: int} -> char -> unit | ||
| val writeWord8s : {file: t, file_offset: int, array_slice_offset: int} -> Word8.word ArraySlice.slice -> unit |
There was a problem hiding this comment.
Here I find the two offsets confusing; I think it would be easier to just pass a single offset.
My understanding is that openFileWriteable s n gives us a file of n bytes. So, then, writeChar should be able to pass a single offset, somewhere in the range [0,n).
There was a problem hiding this comment.
The two offsets are to handle the index to which the buffer should start writing and the second index is the index from which the array slice is supposed to write. It was a case of leaking abstraction and was probably user hostile, so I've changed the interface to take in just an offset where the file starts writing file_offset and the array slice.
The new interfaces are
val writeChar : {file: t, file_offset: int} -> char -> unit
val writeWord8s : {file: t, file_offset: int} -> Word8.word ArraySlice.slice -> unit
| fun openFileWriteable path final_size = | ||
| let | ||
| open Posix.FileSys | ||
| val file = createf (path, O_RDWR, O.append, S.flags [S.irusr, S.iwusr, S.irgrp, S.iroth]) |
There was a problem hiding this comment.
I'm curious, what happens here if the file already exists? Does createf behave like openf in that case?
There was a problem hiding this comment.
Yes. In case the file already exists, it works as openf and opens the file in append mode. The user can then start writing from where the file originally ended.
basis-library/mpl/file.sml
Outdated
| open Posix.FileSys | ||
| val file = createf (path, O_RDWR, O.append, S.flags [S.irusr, S.iwusr, S.irgrp, S.iroth]) | ||
| val fileSize = Position.toInt (ST.size (fstat file)) | ||
| val size = final_size + fileSize |
There was a problem hiding this comment.
Shouldn't this just be final_size ?
There was a problem hiding this comment.
The idea was to take the size of the expected input to the file from the user in file_size and then append it to the file, the code then returns the increased size of the file mapped to the memory as well as the "offset" from there user can start writing to the file. I've renamed this variable to buffer_size for better clarity on what the code is trying to do.
…file permissions for readwrite mode
Add functionality to file.sig and file.sml to allow writes through mmap and mmunmap to replace PosixWriteFile.
(WIP and open to review)