Skip to content

return dummy files on error and update in the background. #76

Closed
djay wants to merge 16 commits into
drinfernoo:masterfrom
djay:djay/handle_errors
Closed

return dummy files on error and update in the background. #76
djay wants to merge 16 commits into
drinfernoo:masterfrom
djay:djay/handle_errors

Conversation

@djay
Copy link
Copy Markdown
Collaborator

@djay djay commented Jan 27, 2021

and Notify when background updating

@djay djay marked this pull request as draft January 27, 2021 17:21
@drinfernoo
Copy link
Copy Markdown
Owner

Ooh, I like this. I'll test it this evening.

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Jan 28, 2021

@drinfernoo tested on the case where cache is empty and it seems to work well.
haven't tested when there is an error in the path yet. I guess I can disable the plugin maybe to test

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Jan 28, 2021

@drinfernoo btw I think I left a comment in the code. its not clear what should happen when we get an error in a path. At the moment if we have a cached version we just keep showing that.
We can show a notification but its not going to show up for very long as it will move to the next widget to update.
Replacing the path with an simple item that says error seems a bit harsh since it could just be a temp error.
One possibility I thought of is adding a single item to the cached path that allows the user to reload and says there was an error updating this particular path. Maybe added to teh start. If its merged group then some parts could have update items embedded in it and other parts not.
or just leave it as is and some temp issue with a plugin might eventually resolve itself on next startup or playback

@drinfernoo
Copy link
Copy Markdown
Owner

I still need to take a look, but unfortunately I ran out of time to test tonight, so I may not get to it for a few days 😬

Comment thread plugin.program.autowidget/resources/lib/common/utils.py Outdated
Comment thread plugin.program.autowidget/resources/lib/common/utils.py Outdated
@drinfernoo
Copy link
Copy Markdown
Owner

drinfernoo commented Jan 29, 2021

I'm also seeing that this breaks exploding/cloning, as the only item it sees is UPDATING_FILE 🤔

EDIT: This may have been broken since caching was introduced... I don't know that I actually tested it before.

Comment thread plugin.program.autowidget/resources/lib/refresh.py Outdated
@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Feb 1, 2021

I'm also seeing that this breaks exploding/cloning, as the only item it sees is UPDATING_FILE 🤔

Which part does it break? The initial adding of an exploded folder because it can't get the names of the sub paths?
Actually displaying an exploded group should be ok right?
Maybe when adding groups we do need to get the paths synchronously? any idea how to differentiate a request from a path and a request when adding a group?

EDIT: This may have been broken since caching was introduced... I don't know that I actually tested it before.

Possibly not because before it either returned the old cached version or blocked and returned the latest version, both of which had proper names and items.

@drinfernoo
Copy link
Copy Markdown
Owner

drinfernoo commented Feb 1, 2021

Not sure exactly what's happening, but it only adds 1 item to the exploded group: "Loading Items..." .

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Feb 1, 2021

@drinfernoo I think I fixed the exploding problem.
In someways exploding should perhaps be done in widget load rather then during add though? otherwise how does it get updated if more subpaths get added? or perhaps it could get updated as part of autowidget refreshes?

@drinfernoo
Copy link
Copy Markdown
Owner

I actually have an issue for that already 😉 #34

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Feb 1, 2021

Not sure whats going on here. Note it down to look at later

2021-02-01 15:13:58.053 T:123145509105664   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.IndexError'>
                                            Error Contents: list index out of range
                                            Traceback (most recent call last):
                                              File "/Users/dylanjay/Library/Application Support/Kodi/addons/plugin.program.autowidget/main.py", line 13, in <module>
                                                router.dispatch(_plugin, _handle, _params)
                                              File "/Users/dylanjay/Library/Application Support/Kodi/addons/plugin.program.autowidget/resources/lib/common/router.py", line 71
, in dispatch
                                                is_dir, category = menu.merged_path(group, widget_id)
                                              File "/Users/dylanjay/Library/Application Support/Kodi/addons/plugin.program.autowidget/resources/lib/menu.py", line 455, in mer
ged_path
                                                titles=titles, num=len(paths), merged=True)
                                              File "/Users/dylanjay/Library/Application Support/Kodi/addons/plugin.program.autowidget/resources/lib/menu.py", line 242, in sho
w_path
                                                color = widget_def['path'][idx].get('color', default_color)
                                            IndexError: list index out of range
                                            -->End of Python script error report<--

@drinfernoo
Copy link
Copy Markdown
Owner

@djay Do you have a merged widget? Or perhaps one showing a different page?

@djay djay mentioned this pull request Feb 10, 2021
@djay djay marked this pull request as ready for review February 11, 2021 02:44
@drinfernoo
Copy link
Copy Markdown
Owner

As a test of the "error" state, I exploded the TheMovieDB Helper -> Trakt -> Your Lists path (resulting in a group with 14 paths), and added a cycling widget to my home screen. I let a widget load, and then revoked my authorization via TMDbHelper's settings. I expected this to give me an error state, as there are no longer an items in that menu.

I'm now stuck in a loop of the same list trying to be loaded over and over again:

...
2021-02-12 07:35:34.746 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=halloween-episodes&user_slug=drinfernoo&widget=True
2021-02-12 07:35:34.747 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:13): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:35:35.090 T:2480  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144134.75 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:35:35.099 T:2480  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:14): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:35:49.750 T:8012  NOTICE: plugin.program.autowidget: Dequeued cache update: baf30
2021-02-12 07:35:50.711 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=halloween-episodes&user_slug=drinfernoo&widget=True
2021-02-12 07:35:50.713 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:15): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:35:51.118 T:6192  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144150.71 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:35:51.133 T:6192  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:16): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:36:05.723 T:8012  NOTICE: plugin.program.autowidget: Dequeued cache update: baf30
2021-02-12 07:36:07.098 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=halloween-episodes&user_slug=drinfernoo&widget=True
2021-02-12 07:36:07.099 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:15): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:36:07.405 T:5204  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144167.1 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:36:07.417 T:5204  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:16): baf30 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
...

Forcing a widget refresh didn't help, as it's now just trying to load a different list:

...
2021-02-12 07:39:53.557 T:5216  NOTICE: [plugin.video.themoviedb.helper]
                                            No JSON object could be decoded
2021-02-12 07:39:54.255 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=kids-christmas&user_slug=drinfernoo&widget=True
2021-02-12 07:39:54.256 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:15): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:39:54.800 T:4428  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144394.26 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:39:54.819 T:4428  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:16): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:40:09.299 T:8012  NOTICE: plugin.program.autowidget: Dequeued cache update: 6f9f8
2021-02-12 07:40:09.880 T:10320  NOTICE: [plugin.video.themoviedb.helper]
                                            No JSON object could be decoded
2021-02-12 07:40:10.576 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=kids-christmas&user_slug=drinfernoo&widget=True
2021-02-12 07:40:10.578 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:15): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:40:11.020 T:2184  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144410.58 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:40:11.034 T:2184  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:16): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:40:25.609 T:8012  NOTICE: plugin.program.autowidget: Dequeued cache update: 6f9f8
2021-02-12 07:40:26.279 T:12256  NOTICE: [plugin.video.themoviedb.helper]
                                            No JSON object could be decoded
2021-02-12 07:40:27.372 T:8012   ERROR: XFILE::CDirectory::GetDirectory - Error getting plugin://plugin.video.themoviedb.helper/?info=trakt_userlist&owner=true&list_slug=kids-christmas&user_slug=drinfernoo&widget=True
2021-02-12 07:40:27.375 T:8012  NOTICE: plugin.program.autowidget: Error cache 0B (exp:-1 day, 23:59:40, last:0:00:16): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
2021-02-12 07:40:27.906 T:8688  NOTICE: plugin.program.autowidget: [ action: cycling ][ group: your_lists-1613143449.16 ][ mode: path ][ refresh: 1613144427.38 ][ id: 5a786fe6-65d4-4303-b536-83cabc8b8f78 ]
2021-02-12 07:40:27.928 T:8688  NOTICE: plugin.program.autowidget: Empty cache 0B (exp:-1 day, 23:59:40, last:0:00:16): 6f9f8 [u'5a786fe6-65d4-4303-b536-83cabc8b8f78']
...

Reauthorizing Trakt technically helped, but I had to manually clear the cache files in order for it reload the widget that should've been showing... and I was completely unable to trigger an error state.

Something's still not quite right 😅

@drinfernoo
Copy link
Copy Markdown
Owner

It also appears that the code in manage.clean(...) isn't working properly anymore 🤔 I haven't tried to step through it yet, but that process was always somewhat minor anyways (to ensure we weren't wasting cycles refreshing widgets that are no longer in use). Ideally, it looks in settings.xml for any installed skin (and the Skin Shortcuts files, for systems that have that) to see if any of our defined widgets are not being used anymore, and then deletes the associated .widget file. I'll say that is an issue for another time 😉

In any case, I think the final snag for this PR is to get it to realize that when there are no items returned, we should be in an "error" state, and stop trying to read the cache, rather than trying over and over again.

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Feb 17, 2021

@drinfernoo if there is a cache and an error is returned then it doesn't return a error tile. It returns the cache. This makes sense for temporary problems but perhaps not for longer term issues. I highlighted this above. We could return an error but then its a pain for a temp issue (like throttling). We could also add an error tile to the start of the path to show there was a problem? What do you think?
It shouldn't be looping trying to update the broken widget. I can look into that.

@drinfernoo
Copy link
Copy Markdown
Owner

Still seeing looping with failed widgets... I added a handful of my Trakt paths from TMDbHelper, and then deauthorized my account. This is making TMDbHelper throw 401 errors, and unable to load these lists, but AutoWidget just keeps on trying 😅 I think if after... maybe three (?) retries with no path returned, let's just return the error state.

@drinfernoo
Copy link
Copy Markdown
Owner

drinfernoo commented Jul 6, 2021

Hey, I think it would be a good time to try to punch these through 😉

Copy link
Copy Markdown
Owner

@drinfernoo drinfernoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that I never actually submitted this review 😅

Comment thread plugin.program.autowidget/resources/lib/common/utils.py
Comment thread plugin.program.autowidget/resources/lib/common/utils.py Outdated
Comment thread plugin.program.autowidget/resources/lib/add.py
@drinfernoo
Copy link
Copy Markdown
Owner

I've added a rudimentary "error state" for widgets that end up with an error... but it is admittedly very basic and could use some help: af7d6fd

@drinfernoo drinfernoo requested review from drinfernoo and removed request for drinfernoo July 18, 2021 23:29
@drinfernoo
Copy link
Copy Markdown
Owner

I've gone ahead and done the rebase (as far as I can tell, still needs testing) in my handle_errors branch: https://github.com/drinfernoo/plugin.program.autowidget/tree/djay/handle_errors

I'm gonna have some people give this some testing, but I'd like to get this one merged soonish.

@drinfernoo
Copy link
Copy Markdown
Owner

I also went ahead and resolved my conversations, as I'm not sure they actually apply anymore... been a long time since I made that review.

@drinfernoo
Copy link
Copy Markdown
Owner

Definitely have some issue here... likely due to the way I changed .widget files for 3.3.0. I never get anything but a "Loading Content..." tile 😅

@drinfernoo
Copy link
Copy Markdown
Owner

I'm not sure how we can best collaborate on this now... I have a branch and you have a branch 😅 I'm trying to get the pre-caching working for paged widgets, but I think I'm missing something, or possibly goofed up the rebase. I'm never getting a .queue file created... and it seems the only time those actually get worked are whenever the service starts at boot, anyways?

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Jul 19, 2021

@drinfernoo not sure. hard to remember what I was doing. brief look at my code there is more stuff in there than handling errors such as switching to background progress.
I don;t think any of my code directly read your .widget files so changes to that shouldn't effect caching I believe. It acts by intercepting a request for a given path and caches that.

@drinfernoo
Copy link
Copy Markdown
Owner

drinfernoo commented Jul 19, 2021

@djay, What I'm seeing when I step through, is that:

utils.cache_expiry(utils.path2hash(file['file']), widget_id, background=True)

ends up calling through to:

But that touch method doesn't seem to actually be doing anything 🤔

No .queue file is ever created, which I find odd. I have seen them rarely pop up in my current builds, but not in this build...

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Jul 19, 2021

@drinfernoo could be a platform specific thing? I only ever tested on mac and android. Would certainly help to have automated tests for this stuff so easy to run on different platforms and see which bit is breaking.

@djay
Copy link
Copy Markdown
Collaborator Author

djay commented Jul 19, 2021

@drinfernoo https://stackoverflow.com/questions/42630281/utime-has-no-effect-in-windows

One thing that would be good to do is put in an assert statement in touch that checks to make sure it worked

@drinfernoo
Copy link
Copy Markdown
Owner

@drinfernoo https://stackoverflow.com/questions/42630281/utime-has-no-effect-in-windows

One thing that would be good to do is put in an assert statement in touch that checks to make sure it worked

After making some changes like they showed in that SO thread, something still wasn't working out... I tried doing this:

def push_cache_queue(hash, widget_id=None):
    queue_path = os.path.join(_addon_data, "{}.queue".format(hash))
    history = read_history(hash, create_if_missing=True)  # Ensure its created
    if widget_id is not None and widget_id not in history["widgets"]:
        history_path = os.path.join(_addon_data, "{}.history".format(hash))
        history["widgets"].append(widget_id)
        write_json(history_path, history)

    if os.path.exists(queue_path):
        pass  # Leave original modification date so item is higher priority
    else:
        write_file(queue_path, hash)  # write the file instead of only touching
        # touch(queue_path)

And now I can get a file created, but I'm having trouble figuring out how to get them to be used properly. This caching stuff is a bit beyond me, and I'm just trying to wade through what you set up 😉

I did find that a small change needed to be made in getting the path from the widget, since I've now changed the widget format from holding a full copy of the path, to only its ID: dd87fde

@drinfernoo
Copy link
Copy Markdown
Owner

Closing this in favor of #98, as it's much more up-to-date and shouldn't have as many conflicts 😅

@drinfernoo drinfernoo closed this Jul 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Request] Confusing when background updates are happening [Bug] Refreshing widgets

2 participants