#672 make sort process call initmissingtables and notifyidbs instead … #681
Conversation
code/processes/wdb.q
Outdated
| .lg.o[`idb;"starting idb reload"]; | ||
| if[writedownmode in `partbyenum`default; | ||
| .lg.o[`eod;"initialising wdbhdb for partition: ",string[currentpartition]]; | ||
| $[.proc.proctype~`sort;{ws:exec w from .servers.getservers[`proctype;`wdb;()!();1b;0b];[first ws](`.wdb.initmissingtables;[1b])}[];initmissingtables[0b]]; |
There was a problem hiding this comment.
what about sortworker processes? do they need to be handled?
why is the arg 1b if running it from sort process but 0b if running directly on wdb?
There was a problem hiding this comment.
I don't believe the sortworker procs will ever be involved here but I could be wrong
When the initmissingtables function was called from the wdb with the new set up, it was called before currentpartition is updated to the next day
The new argument for the function allows the function to be called for either the current value of currentpartition or the next one.
Honestly not too sure on this, so I am open to suggestions
There was a problem hiding this comment.
When the initmissingtables function was called from the wdb with the new set up, it was called before currentpartition is updated to the next day
Sorry I missed this comment earlier, I'm not really getting it. Why is it called before currentpartition is updated if called from wdb but after if called from sort? is there any chance of a timing issue here where if sort is very fast it could call it before it gets updated etc.?
this feels quite messy to me
There was a problem hiding this comment.
It seems to me that if the sorting is done by the sort proc, then the wdb has already moved on and updated currentpartiton, however when the wdb does the sort currentpartition hasn't been updated as the endofday method is blocked
There was a problem hiding this comment.
so should the initilisation of next partition perhaps be tied to wdb moving to next partition, rather than being tied to completion of EOD? to me it seems like if wdb is going to be already writing to new partition, we need to have initialised it?
code/processes/wdb.q
Outdated
| .lg.o[`idb;"starting idb reload"]; | ||
| if[writedownmode in `partbyenum`default; | ||
| .lg.o[`eod;"initialising wdbhdb for partition: ",string[currentpartition]]; | ||
| $[.proc.proctype~`sort;{ws:exec w from .servers.getservers[`proctype;`wdb;()!();1b;0b];[first ws](`.wdb.initmissingtables;[1b])}[];initmissingtables[0b]]; |
There was a problem hiding this comment.
When the initmissingtables function was called from the wdb with the new set up, it was called before currentpartition is updated to the next day
Sorry I missed this comment earlier, I'm not really getting it. Why is it called before currentpartition is updated if called from wdb but after if called from sort? is there any chance of a timing issue here where if sort is very fast it could call it before it gets updated etc.?
this feels quite messy to me
|
There might be a better way to to the initmissingtables to make sure it is right, going to look into this and try to come up with something better |
jonathonmcmurray
left a comment
jonathonmcmurray
left a comment
There was a problem hiding this comment.
Couple of little things, but looking good
code/processes/wdb.q
Outdated
| ]; | ||
| }; | ||
|
|
||
| /- funtion to initialise partition at EOD and notifyidbs to rollover, if .wdb.mode is not a sort mode, initmissingtables done on sort proc |
There was a problem hiding this comment.
this comment is a bit confusing, we don't seem to be checking .wdb.mode?
There was a problem hiding this comment.
https://github.com/DataIntellectTech/TorQ/pull/681/files#diff-6a3cbc5795f37aeb51ed652b0a6abb04020354619518f84a6c8423d0f888ca55R599
it's this line, if the proc type the function is running on is sort then the wdb mode is not a sort mode
There was a problem hiding this comment.
ok, but my point is this comment refers to .wdb.mode, and we're not checking a variable by that name?
code/processes/wdb.q
Outdated
| .save.postreplay:@[value;`.save.postreplay;{{[d;p] }}]; /-post EOD function, invoked after all the tables have been written down | ||
|
|
||
| / - end of default parameters | ||
| - end of default parameters |
There was a problem hiding this comment.
this shouldn't have been removed
jonathonmcmurray
deleted the
672-idb-reload-occurs-before-previous-partition-deleted-in-writedownmodedefault
branch
Fix bug where idb rollover fails to find new partition