Fix DiskLruCache to work on Windows#5941
Merged
swankjesse merged 1 commit intomasterfrom Apr 12, 2020
Merged
Conversation
As originally designed DiskLruCache assumes an inode-like file system, where it's fine to delete files that are currently being read or written. On Windows the file system forbids this, so we must be more careful when deleting and renaming files. These operations come up a lot internally in the cache: - deleting to evict an entry - renaming to commit a dirty file to a clean file The workaround is simple if unsatisfying: we don't permit concurrent reads and writes on Windows. We can have multiple concurrent reders, or a single writer. One challenge in this implementation is detecting whether we're running on Windows or a good operating system. We deliberately don't look at System properties here because the OS and file system may disagree, such as when a Windows machine has an ext4 partition, or when a Linux machine has an NTFS partition. Instead of detecting we just attempt an edit and see what happens. Another challenge in this implementation is what to do when a file needs to be deleted but cannot be because it is currently open. In such cases we now mark the cache entry as a 'zombie'. When the files are later closed they now check for zombie status and delete the files if necessary. Note that it is not possible to store a new cache entry while it is a zombie. Closes: #5761
swankjesse
force-pushed
the
jwilson.0411.fix_cache_on_windows
branch
from
aef5b12 to
64d3b07
Compare
yschimke
reviewed
Apr 12, 2020
| } | ||
| throw IllegalStateException(builder.toString()) | ||
| check(openResources.isEmpty()) { | ||
| "Resources acquired but not closed:\n * ${openResources.joinToString(separator = "\n * ")}" |
yschimke
approved these changes
Apr 12, 2020
Collaborator
yschimke
left a comment
yschimke
left a comment
There was a problem hiding this comment.
Makes sense, but minimal review by me.
yschimke
reviewed
Apr 12, 2020
| } | ||
| } | ||
|
|
||
| civilizedFileSystem = fileSystem.isCivilized(journalFileBackup) |
Collaborator
There was a problem hiding this comment.
This is probably worthy of logging when false.
Collaborator
Author
There was a problem hiding this comment.
ooooh I think it’d be pretty boring. It’ll pretty much always be true on Windows, and false elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As originally designed DiskLruCache assumes an inode-like
file system, where it's fine to delete files that are
currently being read or written.
On Windows the file system forbids this, so we must be
more careful when deleting and renaming files. These
operations come up a lot internally in the cache:
The workaround is simple if unsatisfying: we don't
permit concurrent reads and writes on Windows. We
can have multiple concurrent reders, or a single
writer.
One challenge in this implementation is detecting
whether we're running on Windows or a good operating
system. We deliberately don't look at System properties
here because the OS and file system may disagree, such
as when a Windows machine has an ext4 partition, or when
a Linux machine has an NTFS partition. Instead of detecting
we just attempt an edit and see what happens.
Another challenge in this implementation is what to
do when a file needs to be deleted but cannot be because
it is currently open. In such cases we now mark the
cache entry as a 'zombie'. When the files are later
closed they now check for zombie status and delete the
files if necessary. Note that it is not possible to
store a new cache entry while it is a zombie.
Closes: #5761