KAFKA-13098: Fix NoSuchFileException during snapshot recovery

[jsancio]

Java's FileTreeIterator throws an NoSuchFileException when visting
@metadata-0/partition.metadata.tmp. This is most like do to the
fact that the Log type asynchronously creates and delete that file.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[jsancio]

This commit needs to get cherry picked to 2.8 and 3.0.

[cmccabe]

LGTM

[cmccabe]

I cherry-picked this to 3.0, but the commit does not apply to 2.8. Since we never created snapshot files in 2.8, are you sure it needs to be present there?

[cmccabe]

Possibly related: #11056

[jsancio]

I cherry-picked this to 3.0, but the commit does not apply to 2.8. Since we never created snapshot files in 2.8, are you sure it needs to be present there?

@cmccabe My understanding is that @jolshan and @junrao are planning to cherry pick #11056 to 2.8 because it is a performance regression in 2.8.

Even though there are no snapshots in the metadata directory in 2.8, this code exists and gets executed in 2.8:
https://github.com/apache/kafka/blob/2.8.0/core/src/main/scala/kafka/raft/KafkaMetadataLog.scala#L336-L360

We can either proactively cherry pick this change now or wait until #11056 get cherry picked to 2.8.

[jolshan]

Since I've confirmed that we don't use partition.metadata files for the metadata topic in 2.8, I've removed this version from the ticket. Thanks @jsancio for this patch.