scan-photos process memory consumption increases over runtime

[myxor]

After some time the scan-photos process starts consuming a lot of memory and the amount increases even more over the time.

This is a screenshot from htop running on my Nextcloud host:

and some time later:

This leads to crashing of the process on my machine with 2GB of RAM after scanning around 32k pictures.

[1] 19349 killed sudo -u www-data php /var/www/nextcloud/occ maps:scan-photos marco

Is it possible to run the job with a set memory limit?

[tacruc]

I head this problem, too. But no time and idea how to fix it. I just increased the memory limit, which is not an option if you do not have more RAM. If you have big Tiff pictures, it might be worth to remove them temporally, as one library for reading the exif extracts the hole picture to the RAM. I head cases where a 150mb TIFF used multiple GB of Ram.

[myxor]

I am not using TIFF images, mostly JPEGs.
As the amount of RAM consumed grows constant it seems that there is some memory leak inside the script. I will have a look into it and see if i can find anything.

[bilibilistack]

I have the same issue, the process posts killed after scanned 30k photos.

[YoSiJo]

I have the same problem.
But see the following messages in the logs:

Sep 07 14:54:08 waimanu-nc Nextcloud[8516]: {PHP} A non-numeric value encountered at /var/www/nc.waimanu.io/main/apps/maps/vendor/lsolesen/pel/src/PelIfd.php#370
Sep 07 14:54:12 waimanu-nc Nextcloud[8516]: {PHP} A non well formed numeric value encountered at /var/www/nc.waimanu.io/main/apps/maps/vendor/lsolesen/pel/src/PelIfd.php#370
Sep 07 14:54:12 waimanu-nc Nextcloud[8516]: {PHP} A non well formed numeric value encountered at /var/www/nc.waimanu.io/main/apps/maps/vendor/lsolesen/pel/src/PelIfd.php#370     
Sep 07 14:54:17 waimanu-nc Nextcloud[31572]: {core} {"Exception":"OC\\AppFramework\\Middleware\\Security\\Exceptions\\NotLoggedInException","Message":"Current user is not logged in","Code":401,"Trace":[{"file":"\/var\/www\/nc.waiman
u.io\/main\/lib\/private\/AppFramework\/Middleware\/MiddlewareDispatcher.php","line":95,"function":"beforeController","class":"OC\\AppFramework\\Middleware\\Security\\SecurityMiddleware","type":"->","args":[{"__class__":"OCA\\Files\
\Controller\\ViewController"},"index"]},{"file":"\/var\/www\/nc.waimanu.io\/main\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":97,"function":"beforeController","class":"OC\\AppFramework\\Middleware\\MiddlewareDispatcher"
,"type":"->","args":[{"__class__":"OCA\\Files\\Controller\\ViewController"},"index"]},{"file":"\/var\/www\/nc.waimanu.io\/main\/lib\/private\/AppFramework\/App.php","line":126,"function":"dispatch","class":"OC\\AppFramework\\Http\\D
ispatcher","type":"->","args":[{"__class__":"OCA\\Files\\Controller\\ViewController"},"index"]},{"file":"\/var\/www\/nc.waimanu.io\/main\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php","line":47,"function":"main","clas
s":"OC\\AppFramework\\App","type":"::","args":["ViewController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"files.view.index"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\Ro
uteActionHandler","type":"->","args":[{"_route":"files.view.index"}]},{"file":"\/var\/www\/nc.waimanu.io\/main\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\
\RouteActionHandler"},{"_route":"files.view.index"}]},{"file":"\/var\/www\/nc.waimanu.io\/main\/lib\/base.php","line":975,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/apps\/files\/"]},{"file":"\/var\/www\/nc
.waimanu.io\/main\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/nc.waimanu.io\/main\/lib\/private\/AppFramework\/Middleware\/Security\/SecurityMiddleware.php","Line":143,"C
ustomMessage":"--"}

In the end the job took about 40Gb.
With -vvv I couldn't find more, because the output hasn't changed.

[tacruc]

@eneiluj Do you have an Idea what might leak the memory here?

[YoSiJo]

Ok I have found a defective image file.

# exiftool -exif:all= -tagsfromfile @ -all:all -unsafe 'example.jpg'                                                                                                  
Warning: [minor] Fixed incorrect URI for xmlns:MicrosoftPhoto - example.jpg                                                                                                                                  
Warning: [minor] Fixed incorrect URI for xmlns:MicrosoftPhoto - example.jpg                                                                                                                                  
    1 image files updated 

After the repair the image, the error massage have changed. I'm still checking whether the new error is still related to it.

[tacruc]

@YoSiJo I think your having two different problems, the first is the memory leak discussed in this issue and the second are files which can not be read by php_exif or lsolesen/pel. Lets stick here to the first issue. I think the second is probably better disscussed in php_exif or loslesen/pel.

[YoSiJo]

The error with the RAM is now gone. The new error is related to loslesen/pel and has probably nothing to do with this issue.
The solution for me was this:

find . -type f -regextype posix-egrep -iregex '.*jpg' -print0 | xargs -0 exiftool -warning
exiftool -exif:all= -tagsfromfile @ -all:all -unsafe ${FILE}

[tacruc]

I tried to find the leak today. I have the feeling that there are different parts leaking memory, but the biggest seams to be the two methods for exif decoding. Pel probably the most.

But no madder how I changed the code, the memory consumption did only increase.
@eneiluj what do you think of starting one process per Picture or small chunk as a work arround?

[myxor]

It seems that the Pel project https://github.com/pel/pel is pretty much dead :(