Error writing to datastore on Windows 10

[hsn10]

Version information:

0.5 rc1

Description:

New flatfs datastore version introduced in 0.5 has problems with writing blocks on Windows 10.

2020-04-07T16:46:32.917+0200    �[31mERROR�[0m  bitswap go-bitswap@v0.2.8/bitswap.go:342
Error writing 3 blocks to datastore: 
  rename C:\tmp\ipfs2\blocks\.temp\temp-043813542
 C:\tmp\ipfs2\blocks\AJ\CIQCCVMFRWGRR2S7OUU7F3JWGSYDBLXX7SJKXT343KSIH2XRLPM3AJI.data: Access is denied.
github.com/ipfs/go-bitswap.(*Bitswap).receiveBlocksFrom
        pkg/mod/github.com/ipfs/go-bitswap@v0.2.8/bitswap.go:342
github.com/ipfs/go-bitswap.(*Bitswap).ReceiveMessage
        pkg/mod/github.com/ipfs/go-bitswap@v0.2.8/bitswap.go:419
github.com/ipfs/go-bitswap/network.(*impl).handleNewStream
        pkg/mod/github.com/ipfs/go-bitswap@v0.2.8/network/ipfs_impl.go:271
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1
        pkg/mod/github.com/libp2p/go-libp2p@v0.7.4/p2p/host/basic/basic_host.go:417

There is a very large number of temp-* files in blocks\.temp directory which means that this error happens quite often. It does not fail writing for all blocks, some blocks are written correctly. My guess is that file has to be closed first before it can be renamed on windows platform.

[hsanjuan]

We do close the files before renaming.

This suggests that some other process (like an antivirus) might be opening them though.

It would be helpful to try this golang/go#36568 (comment) and see if it's indeed the antivirus.

The change we made was to create the temp files in specific temp-folder that can be cleaned up later, rather than creating in their final folder and rename them there. I guess the fact that it has to move cross-folders causes issues.

Workarounds:

• See if an antivirus is touching those files and prevent it from doing it
• Switch to badger datastore instead of flatfs

@Stebalien I don't see a good fix here, should we revert the temp file thing for the moment while we rethink it?

[Stebalien]

@hsn10 does this reproduce on go-ipfs 0.4.23

@hsanjuan I don't see how that could be affecting this. Maybe we need to give these files a .data extension to get the AV to ignore it? edit: nevermind, that shouldn't be related

[hsn10]

Older IPFS works correctly. I checked antivirus and there are no exceptions defined for old IPFS data directory.

[hsanjuan]

I checked antivirus and there are no exceptions defined for old IPFS data directory.

But does it still happen if you add such exceptions?

[hsn10]

I added exception to AVG antivirus and there is still number of files left in blocks\.temp and occasional exceptions - No change from previous behaviour, I didn't get Error writing to datastore stack trace every time.

I am not sure if defined exception in AVG actually works because while browsing http://docs.ipfs.io.ipns.localhost:8080/guides/concepts/dht/ I got error:

internalWebError: open C:\tmp\IPFS2\blocks\52\AFYBEIADW54ULQFTVNGZ2A4RHZCY77MD5VFGOKUYKSKG6BOEZTW4W7O52A.data: The process cannot access the file because it is being used by another process.

To be sure I disabled AVG completely and these errors are still there. No change - still temp files stuck and stack traces.

[Stebalien]

Can you build and use a custom branch: https://github.com/ipfs/go-ipfs/tree/test/flatfs?

This branch reverts to a version of flatfs before the change, to make sure we're on the right track.

[Stebalien]

Note: the easiest way to build on windows is to run:

GOPROXY=direct go get github.com/ipfs/go-ipfs/cmd/ipfs@test/flatfs

Alternatively, you can use a pre-built binary.

[hsn10]

I did the same testing at your provided binary, IPFS directory added to antivirus exceptions.

This error is still there:

internalWebError: open C:\tmp\IPFS2\blocks\GG\AFKREIGC7KSDCN3EL747P3XF4JNA5VQ3ENROYJPDYZITFED7FDEL3SSGGQ.data: The process cannot access the file because it is being used by another process.

There is lot of put-DDDDD files in repository. I checked ipfs repo from stable branch and no put-DDDD files are there. After inspecting blocks and checking file sizes of put-DDDDD and an .data block file it looks like everything what is in put-DDDDD file left behind is also in .data file, so it looks like cleanup problem.

Sample of such directory, look carefully at file dates:

$ ls --full-time
total 20
-rw-r--r-- 1 hsn 197609  816 2020-04-09 10:55:15.785266500 +0200 AFYBEIA465WMGALGEBK5FC53RZO3H7PCAR4TBPWTYZLHIFFZGMIB6YQ2AU.data
-rw-r--r-- 1 hsn 197609 5796 2020-04-09 10:45:45.701833100 +0200 CIQMEQ4UZMWVETA2EP6YGXPVBJFBNCT2X7Y5FA2IJHNAANDLCEYX2AA.data
-rw-r--r-- 1 hsn 197609 5796 2020-04-09 10:45:45.703783400 +0200 put-764934526

and more interesting case:

$ ls --full-time
total 16
-rw-r--r-- 1 hsn 197609 3202 2020-04-09 10:57:00.550484900 +0200 AFYBEIC6CKGQFF3ALSQPXGOCSIZYJACBR4HOTEGA7D7JA7E3LFGQYHBA2A.data
-rw-r--r-- 1 hsn 197609 3202 2020-04-09 10:57:00.553414300 +0200 put-310544969
-rw-r--r-- 1 hsn 197609 3202 2020-04-09 10:57:00.557319000 +0200 put-343986019
-rw-r--r-- 1 hsn 197609 3202 2020-04-09 10:57:00.552437300 +0200 put-348454676

It looks like put file left behind is written after block already finished downloading from other source, rename to an existing file fails but there is no cleanup procedure for deleting stale put-DDDDDD.

[Stebalien]

Yeah, ok, so it's likely due to some change in timing in bitswap. It looks like we should be opening all files in SHARE_DELETE mode.