Expiration fails for large ServiceControl instances

[gbiellem]

From Google Groups... (Desk Case 7173)

I created the following ServiceControl.exe.config and placed it in ServiceControl install directory:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<configuration>
  <appSettings>
        <add key="ServiceControl/DbPath" value="e:\Particular\ServiceControl\localhost-33333" />
    <add key="ServiceControl/ExpirationProcessTimerInSeconds" value="120" />
    <add key="ServiceControl/HoursToKeepMessagesBeforeExpiring" value="240" />
    <add key="ServiceControl/ExposeRavenDB" value="true" />
  </appSettings>
</configuration>

By doing this I thought I was telling ServiceControl to only store 10 days of recent history instead of the default 30 days, thus cutting down on the amount of storage that the internal RavenDB would need. However, I am a couple weeks in and the data file continues to grow from 20 to 50 to 80 and now 125GB.

Logs show this:

2014-10-03 07:11:25.0096|915|Error|DocumentsExpiryLogger|Error when trying to find expired documents
System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at Raven.Database.DocumentDatabase.<>c__DisplayClass8f.<Query>b__84(IStorageActionsAccessor actions)
   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext)
   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken externalCancellationToken, Action`1 headerInfo, Action`1 onResult)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken token)
   at ServiceControl.Infrastructure.RavenDB.Expiration.ExpiredDocumentsCleaner.TimerCallback(Object state) in c:\BuildAgent\work\b7f882efc2cd360b\src\ServiceControl\Infrastructure\RavenDB\Expiration\ExpiredDocumentsCleaner.cs:line 69

[gbiellem]

@andreasohlund - I think this should be a patched

[dannycohen]

@gbiellem - does the number of messages stored keep growing or is it just the size ? (i.e. is there a DB file size compacting issue, or is it an issue with the expiration ?)

// CC @synhershko

[gbiellem]

I think it only happens when the db gets "big"..there is a 5 minute time out on the query in the expired document cleaner code. I think if this fails no docs are removed

[dannycohen]

This indeed sounds like a patch / Hotfix + notification to customers.

// @andreasohlund, @udidahan

[andreasohlund]

Yes, sounds like a patch

On Tue, Oct 7, 2014 at 8:04 AM, Danny Cohen notifications@github.com
wrote:

This indeed sounds like a patch / Hotfix + notification to customers.

// @andreasohlund https://github.com/andreasohlund, @udidahan
https://github.com/udidahan

—
Reply to this email directly or view it on GitHub
#387 (comment)
.

[synhershko]

The only reason for this to happen is if the first cleanup operation was run against a huge database with high cardinality of dates (that is, many unique timestamps). That still shouldn't take 5 minutes though. Can you check with the OP what was the size of the DB when he enabled the expiry (or triggered it for the first time)

[toddlangdon]

I believe the size when I changed the number of days from default of 30 to 10 was in the 20-30GB range but don't know the exact value...

[synhershko]

Ok then I think I know what the issue is, will try to come up with a decent fix. Still surprised the query took THAT long.

[synhershko]

@toddlangdon just for the sake of completeness -

1. Can you confirm no cleanup was ever run successfully?
2. Are there any more helpful log entries?

I'm assuming the query was taking 5mins because of Lucene range queries having way too many terms, but since it seems Esent was already involved I want to verify this and rule out everything else.

[toddlangdon]

can you tell me how I would verify if a cleanup was ever run successfully? Sorry, I'm just not familiar enough with RavenDB to know the answer...

[synhershko]

You should see this in the logs: "Deleting {0} expired documents: ..."

https://github.com/Particular/ServiceControl/blob/develop/src/ServiceControl/Infrastructure/RavenDB/Expiration/ExpiredDocumentsCleaner.cs#L96

Alternatively, you can login to the management studio and see what is the oldest entry.