Skip to content

Sort and apply attributes once per table partition #694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pmurphy979 merged 1 commit into from
Feb 12, 2025
Merged

Sort and apply attributes once per table partition #694

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions code/processes/tickerlogreplay.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,13 +165,9 @@ checkcount:{[h;p;counter;td]
.replay.currentcount:0]}

// function used to finish off the replay
// generally this will be to re-sort the table, and set an attribute
finishreplay:{[h;p;td]
// save down any tables which haven't been saved
savetab[td;h;p] each tabsincountorder[.replay.tablestoreplay];
// sort data and apply the attributes
if[sortafterreplay;applysortandattr[.replay.pathlist]];

// invoke any user defined post replay function
.save.postreplay[h;p];
}
Expand All @@ -190,7 +186,7 @@ replaylog:{[logfile]
[.lg.o[`replay;"skipping first ",(string firstmessage)," messages"];
@[`.;`upd;:;.replay.initialupd]];
@[`.;`upd;:;.replay.realupd]];
.replay.tablecounts:.replay.errorcounts:.replay.pathlist:()!();
.replay.tablecounts:.replay.errorcounts:()!();

// If not running in segmented mode, reset replay date and clean HDB directory on each loop
.replay.zipped:$[logfile like "*.gz";1b;0b];
Expand Down Expand Up @@ -415,7 +411,9 @@ initandrun:{

// Replay all logs and exit
.lg.o[`initandrun;"Replaying the following log(s): ",csv sv 1_'string .replay.logstoreplay];
.replay.pathlist:()!();
.replay.replaylog each .replay.logstoreplay;
if[sortafterreplay;applysortandattr[.replay.pathlist]];
if[partandmerge;postreplaymerge[tempdir;.replay.replaydate;hdbdir]];
.lg.o[`replay;"replay complete"];
if[.replay.exitwhencomplete;exit 0];
Expand Down