Improve previews

[rullzer]

This PR improves the preview system:

• Previews are stored in the AppData
  • Allows sharing of previews for shared data
  • Allows sharing of previews for external storage
• Now always respects min/max preview size
• Previews are stored in power of 2 sizes (should be more than enough)

Todo:

• React on update/deletion of files (and fodlers for delete) (CC: @icewind1991)
• Move all endpoints over
• Move to controller
• Test file access control
• Tests!
• Cleanup commits
• Add cache
• Remove old previews
• Look into command to generate new previews

FYI @oparoz

[mention-bot]

@rullzer, thanks for your PR! By analyzing the history of the files in this pull request, we identified @LukasReschke, @nickvergessen and @icewind1991 to be potential reviewers.

[nickvergessen]

Please namespace (we have a Preview namespace) then you can also get rid of the 2

header missing

[rullzer]

Yeah well actually the old preview should go. But yeah lets move it.

[nickvergessen]

docs

[rullzer]

File_sharing (public previews) is moved over. Awesome side effect is that we now (thanks to the FileDisplayResponse) get preview client side caching for free!

[codecov-io]

Current coverage is 58.03% (diff: 66.92%)

Merging #1741 into master will increase coverage by 0.27%

@@             master      #1741   diff @@
==========================================
  Files          1088       1092     +4   
  Lines         62156      62424   +268   
  Methods        6938       6978    +40   
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          35901      36227   +326   
+ Misses        26255      26197    -58   
  Partials          0          0          

Diff Coverage	File Path
0%	apps/files_versions/lib/Storage.php
0%	apps/files_sharing/appinfo/routes.php
0%	apps/files_versions/appinfo/routes.php
0%	new lib/private/Preview/GeneratorHelper.php
0%	apps/files_trashbin/appinfo/routes.php
0%	core/routes.php
•• 20%	lib/private/PreviewManager.php
••••• 55%	apps/files/lib/Controller/ApiController.php
•••••• 60%	new .../files_versions/lib/Controller/PreviewController.php
•••••• 62%	new ...s_sharing/lib/Controller/PublicPreviewController.php

Review all 17 files changed

Powered by Codecov. Last update b129adf...f44cd7a

[rullzer]

Ok all is looking good from my POV.
I'm just not sure how to handle migration. Since there might be more (different) previews of the same image. And picking the right one can be tricky. Also moving everything could take a long time! (even on installations that don't have a dedicated cron job there could be a lot of previews).

Any ideas? @oparoz @MorrisJobke @nickvergessen ?

[icewind1991]

Maybe don't move old previews over but allow reading the old preview when there is none in the new location, then when a file is changed delete the old ones while generating the new ones

[rullzer]

@icewind1991 mmm could work. But that would still mean that currently wrong previews would still be displayed wrong (so for shared files etc)

[adsworth]

or just delete the previews in the old location when first generating a preview with the new logic. unlinking around 5-10 (I guess that's how many there might be) files shouldn't take to long.

Not migrating old preview would create extra load on big installs though, since all previews need to be regenerated.

[rullzer]

Ok after some more thinking I think just deleting all previews and starting over is the easiest and cleanest way forward.

• For a single file there might now be a few (different) previews lying around
• Even small (sqlite) instances might have a shitload of previews so migrating them when there is no dedicated cron is going to time out.
• Looking into two places is not very performant. Risks selecting the wrong preview. And at some point we have to get rid of the old preview folders anyways.

So I vote for just killing all preview folders. And have the previews be regenerated. In most cases this should be significantly less previews since the previews are now shared.

[oparoz]

This will require some good communication. Last time this happened, everybody complained that Pictures/Gallery was slow, because the initial load would take minutes. Now at least we show something in the first few seconds, but I'm just saying that it won't be great for people will large photo collections.

Maybe it's time to resurrect the occ thumbnail generator. It can take hours for the process to complete, but it's better than having to wait when using the GUI imo.

[rullzer]

@oparoz yes I agree. Good communication is key here. We have a nice list of improvements. But gettings this properly is key.

I would not mind the thumbnail generator resurrection. But I also have been thinking about a listner that injects new/edited files. So we can say generate those in the background. But that is a discussion for another time.

I'll see if the thumbnail generator is easy to resurrect. Because that seems like the fastest solution.

[rullzer]

See https://github.com/rullzer/PreviewGenerator for my ultra simple cli preview generator. It isn't smart. Just looks over all files and if it can generate previews.

It doesn't do anything fancy. because of the caching previews if a preview exists it just returns the file.

[rullzer]

Ok I will update the previewgenerator app with a command to also delete the old previews. That seems the most sensible solution IMO.

So review time.

[MorrisJobke]

Code looks good and also works 👍

[oparoz]

Cool about the changelog. We'll have to test a few apps I guess and communicate heavily if things break so that people can fix their app for launch.

rullzer fixed it

[rullzer]

So 1 more thumbs up plz :)

[LukasReschke]

LGTM