Client reported problems with idbreload #683
Description
After testing the latest version, still has some issues on default mode
currentpartition in sort/sortworker is not updated, it is always the value when you start that process, the command
"currentpartition::pt+1;" only works in wdb process
while idb update notification is sending from sort process, so it causes the partition sending to idb is wrong, which can be found in idb log, the command "idbreload[currentpartition+1];"seems to me it is better to use "idbreload[pt+1];" in wdb.q
in wdb by default it is 10s saving data to disk, assuming no data feed-handler working before eod processing, if last saving to disk is 23:59:58, while notifying the idb rollover from sort starts at the time before 00:00:08, for example, eod processing finished at 00:00:02s, then the wdb is empty at that time, so partitionsize::count key idbdir is 0, that results in the loadidb[] can't get the new partition, and the idb query QError after eod processing ; this issue confused me a long time because I can't reproduce this error every time when doing eod processing. for example, if last saving to disk is 23:59:51, and next is 00:00:01, the idb loading would work at 00:00:02.
however, my workaround is updating this function in idb.q when doing intraday loading:
partitioncounthaschanged:{[]
if[writedownmode~`default;$[1j~partitionsize;0b;1b]];
$[partitionsize<>c:count key idbdir;[partitionsize::c; 1b];0b]
};
this can reload idb again, hopefully you would have better solution about this bug, thanks a lot