putMany uses as many FDs as there are entries

[magik6k]

Current implementation of putMany seems to open as many FDs as there are entries to put, with batches few K in size this will cause too many files open errors.

https://github.com/ipfs/go-ds-flatfs/blob/master/flatfs.go#L422

[Kubuxu]

1. we should write files concurrently 2. we should limit number of files open at once
   It might be useful to have a separate, per datastore pool of goroutines that task would be operating on files thus solving FD the problem completely.

[MichaelMure]

A possible strategy to address that issue could be to rework closer() (

go-ds-flatfs/flatfs.go

Lines 560 to 574 in ff6502b

	closer := func() error {
	for closed < len(files) {
	fi := files[closed].file
	if fs.sync {
	if err := syncFile(fi); err != nil {
	return err
	}
	}
	if err := fi.Close(); err != nil {
	return err
	}
	closed++
	}
	return nil
	}

) into a closer(maxOpen int) that would be called after every writes to introduce an upper bound of open files within a single large putMany. This function would close the oldest opened files (that should already be flushed on disk by now) but also keep the youngest open to leave the kernel time to flush when it makes sense.

go-ds-flatfs/flatfs.go

Line 613 in ff6502b

err := closer()

would then become closer(0), sharing the same logic.

[lidel]

@gammazero do you think this was fixed by

• PR refactor: rewrite batch mode to use temp directory #142

?

iiuc batch mode rewritten to use a temp staging directory means the blocks are now written to flat temp dir and atomically renamed on commit, instead of opening all destination FDs at once

does this mean we can also close:

• We should only sync dirs once on putMany #77
• Handle puts *and* deletes in Batch #49