[C#] CommitFailureException

[jmasek]

Hi,
we are using FasterLog as a message queue between producer and consumer. In addition to producer and consumer, there is a routine that periodically commits the FasterLog (every 1000ms to prevent losing data in case there are too few data to trigger automated commit).
Our software runs on circa 20k computers around the world on diverse hardware. We are observing the following exception (with various address ranges).

Exception : FASTER.core.CommitFailureException: Commit of address range [576352-576584] failed with error code 1
   at FASTER.core.Utility.SlowWithCancellationAsync[T](Task`1 task, CancellationToken token, Boolean useSynchronizationContext, Boolean continueOnCapturedContext)
   at FASTER.core.FasterLog.CommitAsync(CancellationToken token)

The commit is retried and retries succeed. We are getting about 100 of these exceptions from various computers every day so it is not a big problem.
In addition to the stack trace, we have added the logging that might help. Please note that the debug information is gathered after catching the exception.

Commiter Error. Begin: 576352, Flushed Until: 576584, Tail Address: 576584: Error: FASTER.core.CommitFailureException: Commit of address range [576352-576584] failed with error code 1

where begin is FasterLog.BeginAddress flushedUntil FasterLog.FlushedUntilAddress, tail address FasterLog.TailAddress.

Could you suggest to us what could cause these exceptions - error 1 is not telling me much and I was not able to determine what could be the root cause?

[badrishc]

It is possible you have too many Commit ranges being flushed in parallel. I have a couple of questions:

1. Do you call Commit after every Enqueue on the Producer code path?
2. How many producer threads are there?
3. What is the configured page size?

[badrishc]

We are also working on a new version of FasterLog with option for fast commits, to be part of FASTER v2. There are some breaking changes, but improvements in commit latencies. Would appreciate your input on this. Prototype code and details are in this PR: #587

[badrishc]

Update: in v2, these errors have been completely eliminated. happy to cherry pick the change into current version if it's need quickly.

[jmasek]

Hi Badrish,
thanks for the reply. Regarding your questions

1. No. We are adding the messages to the log independently on the routine that periodically commits (typically every 1s). So there are two threads one Enqueueing - it does not commit, one that is only periodically committing.
2. Typically, one producer thread. However, our application is a web server that is accepting client requests so there can be occasional multiple producers but at top lower units of producer threads because of mapping clients to the faster devices.
3. page size is configured to 24 (16 MB)

Great news about the v2! I will definitely check it out.

Thanks for your help.