The server doesn't index

[scarlehoff]

I've just tried uploading a fit, waited for a few minutes and it was still not indexed. Tried to upload it again and it it worked. Tried again and this time it failed (so it was already there)*.

I think using said failure as the marker of whether the fit is upload it or not might be a quick and dirty but useful hack for the bot...

while [[ $? -eq 0 ]]
do
    vp-upload ${fitname}
    sleep 10s ; # don't want to chain uploads...
done

For a better solution, I guess we could have a vp-check (or vp-get --dry or whatever) that just tells you whether the file is indexed or not (and that forces reindex).

I've had a quick look at the script and maybe it makes sense to go for a simple systemd based rule? It's also based on inotify so it would only work if the problem is due to the external wrapper and not to the system forgetting to notify.
Tried to have a look at why it failed but can't open the logs. If you want to have a look the fit I was trying to upload was
210127-n3fit-001

*Come to think about it I've actually gotten a few mails lately asking me to upload this or that fit and I just assumed I had forgotten to do it (which is also a very likely explanation) but it might not have been my fault!

[siranipour]

Just wanted to add that this happens to me basically every time I upload a fit and attempting a reupload fixes the indexing issue every time.

[Zaharid]

If I login into the server and try touching some of the files. it updates instanteously:

(base) [nnpdf@nnpdf fits]$ touch patata
(base) [nnpdf@nnpdf fits]$ ls -t | head
fitdata.json
patata

So I am not sure what is happening.

[Zaharid]

That said I am all for improving that infrastructure.

[scarlehoff]

Are there no errors in the logs? Maybe in /var/logs/cron?

[Zaharid]

No, nothing interesting.

[scarlehoff]

Meh, I was hoping for a CurioError or whatever.

I guess it would be good to add a logger to print "file received: " "starting index" and so on to check (if it exists you should see 2 of those for the 210127 fit).

Otherwise a vp-check (even done automagically at the end of every upload) would do the trick I guess.

[wilsonmr]

would somebody sshing in and running the index script themself screw it up?

EDIT: asking for a friend

[Zaharid]

Unlikely. Theoretically it could cause the fit seeing less data to finish last so some data gets lost until another reindexing is triggered, but seems convoluted.