Skip to content

DuckDB checkpoint optimization and timing fix #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erikdarlingdata merged 1 commit into from
Feb 19, 2026
Merged

DuckDB checkpoint optimization and timing fix #159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion Lite/Database/DuckDbInitializer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,10 @@

/// <summary>
/// Gets the connection string for the DuckDB database.
/// Disables automatic WAL checkpoints to prevent 2-3s stop-the-world stalls
/// during collector writes. Manual CHECKPOINT runs between collection cycles instead.
/// </summary>
public string ConnectionString => $"Data Source={_databasePath}";
public string ConnectionString => $"Data Source={_databasePath};checkpoint_threshold=1GB";

/// <summary>
/// Ensures the database exists and all tables are created.
Expand All @@ -38,20 +40,20 @@
/// </summary>
public async Task InitializeAsync()
{
_logger?.LogInformation("Initializing DuckDB database at {Path}", _databasePath);

Check warning on line 43 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 43 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

var directory = Path.GetDirectoryName(_databasePath);
if (!string.IsNullOrEmpty(directory) && !Directory.Exists(directory))
{
Directory.CreateDirectory(directory);
_logger?.LogInformation("Created database directory: {Directory}", directory);

Check warning on line 49 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 49 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}

var archivePath = Path.Combine(directory ?? ".", "archive");
if (!Directory.Exists(archivePath))
{
Directory.CreateDirectory(archivePath);
_logger?.LogInformation("Created archive directory: {ArchivePath}", archivePath);

Check warning on line 56 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 56 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}

/* Try to open the database. If the DuckDB storage version has changed,
Expand Down Expand Up @@ -80,7 +82,7 @@

if (existingVersion < CurrentSchemaVersion)
{
_logger?.LogInformation("Schema upgrade needed: v{Old} -> v{New}", existingVersion, CurrentSchemaVersion);

Check warning on line 85 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 85 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
await RunMigrationsAsync(connection, existingVersion);
await SetSchemaVersionAsync(connection, CurrentSchemaVersion);
}
Expand All @@ -95,7 +97,7 @@
await ExecuteNonQueryAsync(connection, indexStatement);
}

_logger?.LogInformation("Database initialization complete. Schema version: {Version}", CurrentSchemaVersion);

Check warning on line 100 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 100 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}
}

Expand Down Expand Up @@ -146,7 +148,7 @@
cmd.CommandText = $"EXPORT DATABASE '{exportDir.Replace("'", "''")}' (FORMAT PARQUET)";
await cmd.ExecuteNonQueryAsync();
exported = true;
_logger?.LogInformation("Exported old database to {ExportDir}", exportDir);

Check warning on line 151 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 151 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}
}
catch (Exception ex)
Expand All @@ -160,7 +162,7 @@
{
/* DuckDB may have .wal files too */
File.Move(_databasePath, backupPath);
_logger?.LogInformation("Backed up old database to {BackupPath}", backupPath);

Check warning on line 165 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 165 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

var walPath = _databasePath + ".wal";
if (File.Exists(walPath))
Expand Down Expand Up @@ -390,12 +392,12 @@
}

if (serverNames.Count > 0)
_logger?.LogInformation("Fixed server_id in {Table} for {Count} server(s)", table, serverNames.Count);

Check warning on line 395 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 395 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}
catch (Exception ex)
{
/* Table might not exist yet — that's fine, it will be created with correct IDs */
_logger?.LogDebug(ex, "Skipped server_id fix for {Table} (may not exist yet)", table);

Check warning on line 400 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)

Check warning on line 400 in Lite/Database/DuckDbInitializer.cs

View workflow job for this annotation

GitHub Actions / build 

Evaluation of this argument may be expensive and unnecessary if logging is disabled (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1873)
}
}
}
Expand All @@ -408,6 +410,26 @@
return new DuckDBConnection(ConnectionString);
}

/// <summary>
/// Runs a manual WAL checkpoint. Call this between collection cycles
/// to flush the WAL during idle time instead of during collector writes.
/// </summary>
public async Task CheckpointAsync()
{
try
{
using var connection = CreateConnection();
await connection.OpenAsync();
using var command = connection.CreateCommand();
command.CommandText = "CHECKPOINT";
await command.ExecuteNonQueryAsync();
}
catch (Exception ex)
{
_logger?.LogDebug(ex, "Manual checkpoint failed (non-critical)");
}
}

/// <summary>
/// Executes a non-query SQL statement.
/// </summary>
Expand Down
4 changes: 4 additions & 0 deletions Lite/Services/CollectionBackgroundService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
IsCollecting = true;
await _collectorService.RunDueCollectorsAsync(stoppingToken);
LastCollectionTime = DateTime.UtcNow;

/* Flush WAL during idle time instead of letting auto-checkpoint
stall collectors mid-write with 2-3s stop-the-world pauses */
await _collectorService.CheckpointAsync();
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
Expand Down
123 changes: 64 additions & 59 deletions Lite/Services/RemoteCollectorService.BlockedProcessReport.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,71 +390,76 @@ as it lingers in the ring buffer across collection cycles. */
_lastSqlMs = sqlSw.ElapsedMilliseconds;

var duckSw = Stopwatch.StartNew();
using var duckConnection = _duckDb.CreateConnection();
await duckConnection.OpenAsync(cancellationToken);

using var appender = duckConnection.CreateAppender("blocked_process_reports");

while (await reader.ReadAsync(cancellationToken))
using (var duckConnection = _duckDb.CreateConnection())
{
var eventTime = reader.IsDBNull(0) ? (DateTime?)null : reader.GetDateTime(0);
var reportXml = reader.IsDBNull(1) ? null : reader.GetString(1);

if (string.IsNullOrEmpty(reportXml))
{
continue;
}
await duckConnection.OpenAsync(cancellationToken);

/* Parse the blocked process report XML in C# */
var parsed = ParseBlockedProcessReportXml(reportXml, eventTime);
if (parsed == null)
using (var appender = duckConnection.CreateAppender("blocked_process_reports"))
{
continue;
while (await reader.ReadAsync(cancellationToken))
{
var eventTime = reader.IsDBNull(0) ? (DateTime?)null : reader.GetDateTime(0);
var reportXml = reader.IsDBNull(1) ? null : reader.GetString(1);

if (string.IsNullOrEmpty(reportXml))
{
continue;
}

/* Parse the blocked process report XML in C# */
var parsed = ParseBlockedProcessReportXml(reportXml, eventTime);
if (parsed == null)
{
continue;
}

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(parsed.EventTime)
.AppendValue(parsed.DatabaseName)
.AppendValue(parsed.BlockedSpid)
.AppendValue(parsed.BlockedEcid)
.AppendValue(parsed.BlockingSpid)
.AppendValue(parsed.BlockingEcid)
.AppendValue(parsed.WaitTimeMs)
.AppendValue(parsed.WaitResource)
.AppendValue(parsed.LockMode)
.AppendValue(parsed.BlockedStatus)
.AppendValue(parsed.BlockedIsolationLevel)
.AppendValue(parsed.BlockedLogUsed)
.AppendValue(parsed.BlockedTransactionCount)
.AppendValue(parsed.BlockedClientApp)
.AppendValue(parsed.BlockedHostName)
.AppendValue(parsed.BlockedLoginName)
.AppendValue(parsed.BlockedSqlText)
.AppendValue(parsed.BlockingStatus)
.AppendValue(parsed.BlockingIsolationLevel)
.AppendValue(parsed.BlockingClientApp)
.AppendValue(parsed.BlockingHostName)
.AppendValue(parsed.BlockingLoginName)
.AppendValue(parsed.BlockingSqlText)
.AppendValue(parsed.BlockedTransactionName)
.AppendValue(parsed.BlockingTransactionName)
.AppendValue(parsed.BlockedLastTranStarted)
.AppendValue(parsed.BlockingLastTranStarted)
.AppendValue(parsed.BlockedLastBatchStarted)
.AppendValue(parsed.BlockingLastBatchStarted)
.AppendValue(parsed.BlockedLastBatchCompleted)
.AppendValue(parsed.BlockingLastBatchCompleted)
.AppendValue(parsed.BlockedPriority)
.AppendValue(parsed.BlockingPriority)
.AppendValue(reportXml)
.EndRow();

rowsCollected++;
}
}

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(parsed.EventTime)
.AppendValue(parsed.DatabaseName)
.AppendValue(parsed.BlockedSpid)
.AppendValue(parsed.BlockedEcid)
.AppendValue(parsed.BlockingSpid)
.AppendValue(parsed.BlockingEcid)
.AppendValue(parsed.WaitTimeMs)
.AppendValue(parsed.WaitResource)
.AppendValue(parsed.LockMode)
.AppendValue(parsed.BlockedStatus)
.AppendValue(parsed.BlockedIsolationLevel)
.AppendValue(parsed.BlockedLogUsed)
.AppendValue(parsed.BlockedTransactionCount)
.AppendValue(parsed.BlockedClientApp)
.AppendValue(parsed.BlockedHostName)
.AppendValue(parsed.BlockedLoginName)
.AppendValue(parsed.BlockedSqlText)
.AppendValue(parsed.BlockingStatus)
.AppendValue(parsed.BlockingIsolationLevel)
.AppendValue(parsed.BlockingClientApp)
.AppendValue(parsed.BlockingHostName)
.AppendValue(parsed.BlockingLoginName)
.AppendValue(parsed.BlockingSqlText)
.AppendValue(parsed.BlockedTransactionName)
.AppendValue(parsed.BlockingTransactionName)
.AppendValue(parsed.BlockedLastTranStarted)
.AppendValue(parsed.BlockingLastTranStarted)
.AppendValue(parsed.BlockedLastBatchStarted)
.AppendValue(parsed.BlockingLastBatchStarted)
.AppendValue(parsed.BlockedLastBatchCompleted)
.AppendValue(parsed.BlockingLastBatchCompleted)
.AppendValue(parsed.BlockedPriority)
.AppendValue(parsed.BlockingPriority)
.AppendValue(reportXml)
.EndRow();

rowsCollected++;
}

duckSw.Stop();
_lastDuckDbMs = duckSw.ElapsedMilliseconds;
}
Expand Down
48 changes: 26 additions & 22 deletions Lite/Services/RemoteCollectorService.Cpu.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,30 +128,34 @@ drs.end_time DESC

/* Insert into DuckDB using Appender for bulk performance */
var duckSw = Stopwatch.StartNew();
using var duckConnection = _duckDb.CreateConnection();
await duckConnection.OpenAsync(cancellationToken);

using var appender = duckConnection.CreateAppender("cpu_utilization_stats");

while (await reader.ReadAsync(cancellationToken))
using (var duckConnection = _duckDb.CreateConnection())
{
var sampleTime = reader.GetDateTime(0);

/* Client-side dedup for ring buffer (computed sample_time can't be filtered in SQL) */
if (!isAzureSqlDb && lastSampleTime.HasValue && sampleTime <= lastSampleTime.Value)
continue;

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(sampleTime)
.AppendValue(reader.IsDBNull(1) ? 0 : reader.GetInt32(1))
.AppendValue(reader.IsDBNull(2) ? 0 : reader.GetInt32(2))
.EndRow();

rowsCollected++;
await duckConnection.OpenAsync(cancellationToken);

using (var appender = duckConnection.CreateAppender("cpu_utilization_stats"))
{
while (await reader.ReadAsync(cancellationToken))
{
var sampleTime = reader.GetDateTime(0);

/* Client-side dedup for ring buffer (computed sample_time can't be filtered in SQL) */
if (!isAzureSqlDb && lastSampleTime.HasValue && sampleTime <= lastSampleTime.Value)
continue;

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(sampleTime)
.AppendValue(reader.IsDBNull(1) ? 0 : reader.GetInt32(1))
.AppendValue(reader.IsDBNull(2) ? 0 : reader.GetInt32(2))
.EndRow();

rowsCollected++;
}
}
}

duckSw.Stop();
Expand Down
39 changes: 22 additions & 17 deletions Lite/Services/RemoteCollectorService.Deadlocks.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -388,26 +388,31 @@ and was previously misattributed as DuckDB time. */
}

var duckSw = Stopwatch.StartNew();
using var duckConnection = _duckDb.CreateConnection();
await duckConnection.OpenAsync(cancellationToken);

using var appender = duckConnection.CreateAppender("deadlocks");

foreach (var (deadlockTime, victimProcessId, victimSqlText, graphXml) in deadlockRows)
using (var duckConnection = _duckDb.CreateConnection())
{
var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(deadlockTime)
.AppendValue(victimProcessId)
.AppendValue(victimSqlText)
.AppendValue(graphXml)
.EndRow();

rowsCollected++;
await duckConnection.OpenAsync(cancellationToken);

using (var appender = duckConnection.CreateAppender("deadlocks"))
{
foreach (var (deadlockTime, victimProcessId, victimSqlText, graphXml) in deadlockRows)
{
var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(deadlockTime)
.AppendValue(victimProcessId)
.AppendValue(victimSqlText)
.AppendValue(graphXml)
.EndRow();

rowsCollected++;
}
}
}

duckSw.Stop();
_lastDuckDbMs = duckSw.ElapsedMilliseconds;
}
Expand Down
80 changes: 42 additions & 38 deletions Lite/Services/RemoteCollectorService.FileIo.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,46 +120,50 @@ AND vfs.database_id < 32761

/* Insert into DuckDB with delta calculations */
var duckSw = Stopwatch.StartNew();
using var duckConnection = _duckDb.CreateConnection();
await duckConnection.OpenAsync(cancellationToken);

using var appender = duckConnection.CreateAppender("file_io_stats");

foreach (var stat in fileStats)
using (var duckConnection = _duckDb.CreateConnection())
{
var deltaKey = $"{stat.DatabaseId}_{stat.FileId}";
var deltaReads = _deltaCalculator.CalculateDelta(serverId, "file_io_reads", deltaKey, stat.NumOfReads);
var deltaWrites = _deltaCalculator.CalculateDelta(serverId, "file_io_writes", deltaKey, stat.NumOfWrites);
var deltaReadBytes = _deltaCalculator.CalculateDelta(serverId, "file_io_read_bytes", deltaKey, stat.ReadBytes);
var deltaWriteBytes = _deltaCalculator.CalculateDelta(serverId, "file_io_write_bytes", deltaKey, stat.WriteBytes);
var deltaStallReadMs = _deltaCalculator.CalculateDelta(serverId, "file_io_stall_read", deltaKey, stat.IoStallReadMs);
var deltaStallWriteMs = _deltaCalculator.CalculateDelta(serverId, "file_io_stall_write", deltaKey, stat.IoStallWriteMs);

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(stat.DatabaseName)
.AppendValue(stat.FileName)
.AppendValue(stat.FileType)
.AppendValue(stat.PhysicalName)
.AppendValue(stat.SizeMb)
.AppendValue(stat.NumOfReads)
.AppendValue(stat.NumOfWrites)
.AppendValue(stat.ReadBytes)
.AppendValue(stat.WriteBytes)
.AppendValue(stat.IoStallReadMs)
.AppendValue(stat.IoStallWriteMs)
.AppendValue(deltaReads)
.AppendValue(deltaWrites)
.AppendValue(deltaReadBytes)
.AppendValue(deltaWriteBytes)
.AppendValue(deltaStallReadMs)
.AppendValue(deltaStallWriteMs)
.EndRow();

rowsCollected++;
await duckConnection.OpenAsync(cancellationToken);

using (var appender = duckConnection.CreateAppender("file_io_stats"))
{
foreach (var stat in fileStats)
{
var deltaKey = $"{stat.DatabaseId}_{stat.FileId}";
var deltaReads = _deltaCalculator.CalculateDelta(serverId, "file_io_reads", deltaKey, stat.NumOfReads);
var deltaWrites = _deltaCalculator.CalculateDelta(serverId, "file_io_writes", deltaKey, stat.NumOfWrites);
var deltaReadBytes = _deltaCalculator.CalculateDelta(serverId, "file_io_read_bytes", deltaKey, stat.ReadBytes);
var deltaWriteBytes = _deltaCalculator.CalculateDelta(serverId, "file_io_write_bytes", deltaKey, stat.WriteBytes);
var deltaStallReadMs = _deltaCalculator.CalculateDelta(serverId, "file_io_stall_read", deltaKey, stat.IoStallReadMs);
var deltaStallWriteMs = _deltaCalculator.CalculateDelta(serverId, "file_io_stall_write", deltaKey, stat.IoStallWriteMs);

var row = appender.CreateRow();
row.AppendValue(GenerateCollectionId())
.AppendValue(collectionTime)
.AppendValue(serverId)
.AppendValue(server.ServerName)
.AppendValue(stat.DatabaseName)
.AppendValue(stat.FileName)
.AppendValue(stat.FileType)
.AppendValue(stat.PhysicalName)
.AppendValue(stat.SizeMb)
.AppendValue(stat.NumOfReads)
.AppendValue(stat.NumOfWrites)
.AppendValue(stat.ReadBytes)
.AppendValue(stat.WriteBytes)
.AppendValue(stat.IoStallReadMs)
.AppendValue(stat.IoStallWriteMs)
.AppendValue(deltaReads)
.AppendValue(deltaWrites)
.AppendValue(deltaReadBytes)
.AppendValue(deltaWriteBytes)
.AppendValue(deltaStallReadMs)
.AppendValue(deltaStallWriteMs)
.EndRow();

rowsCollected++;
}
}
}

duckSw.Stop();
Expand Down
Loading
Loading