large negative increment values in line profiler

[nathantsoi]

i am using memory profile as follows:

an_instance = AClass()
precision=1
backend='psutil'
with open("mem-profile.txt", 'w') as stream:
  prof = mp.LineProfiler(backend=backend)
  res = prof(an_instance.window)(template)
  mp.show_results(prof, stream=stream, precision=precision)
return res

however the results seem strange, I see large negative increments, much larger than any calls with positive increments:

Line #    Mem usage    Increment
================================
    51    192.2 MiB    192.2 MiB
    52                          
    53                          
    54                          
    55                          
    56                          
    57                          
    58    192.2 MiB      0.0 MiB
    59    736.4 MiB     -8.1 MiB
    60    736.4 MiB     -6.8 MiB
    61    736.4 MiB     -7.4 MiB
    62    736.4 MiB     -8.1 MiB
    63                          
    64    736.4 MiB     -7.7 MiB
    65    736.4 MiB     -8.0 MiB
    66    736.4 MiB     -7.9 MiB
    67    736.4 MiB     -8.1 MiB
    68                          
    69    736.4 MiB     -7.9 MiB
    70    736.4 MiB     -8.0 MiB
    71    736.4 MiB     -8.1 MiB
    72    736.4 MiB     -8.1 MiB
    73                          
    74    736.4 MiB     -8.1 MiB
    75    736.4 MiB     -8.1 MiB
    76    736.4 MiB     -8.1 MiB
    77                          
    78                          
    79                          
    80                          
    81    736.4 MiB    -47.7 MiB
    82    736.4 MiB  -1028.5 MiB
    83    736.4 MiB  -5789.3 MiB
    84    736.4 MiB  -4799.5 MiB
    85                          
    86    736.4 MiB  -4796.0 MiB
    87                          
    88                          
    89    736.4 MiB  -4789.0 MiB
    90                          
    91                          
    92                          
    93                          
    94    736.4 MiB  -4790.4 MiB
    95    599.3 MiB      6.2 MiB
    96    599.3 MiB     85.1 MiB
    97    599.3 MiB     48.6 MiB
    98                          
    99    599.3 MiB      0.0 MiB
   100    599.3 MiB     42.1 MiB
   101    599.3 MiB      0.0 MiB
   102    599.3 MiB      2.3 MiB
   103                          
   104    736.4 MiB  -4782.2 MiB
   105    736.4 MiB  -4767.1 MiB
   106    736.4 MiB  -4795.4 MiB
   107    736.4 MiB  -4797.3 MiB
   108                          
   109    736.4 MiB  -4605.9 MiB
   110                          
   111                          
   112                          
   113    736.4 MiB      0.0 MiB
   114    753.1 MiB   -807.8 MiB
   115    753.1 MiB -22614.2 MiB
   116    753.1 MiB -21805.3 MiB
   117    753.1 MiB -21806.2 MiB
   118    753.1 MiB -21806.2 MiB
   119    753.1 MiB -12706.5 MiB
   120                          
   121    753.1 MiB -12691.2 MiB
   122    753.1 MiB -12707.0 MiB
   123                          
   124    753.1 MiB  -9099.8 MiB
   125    753.1 MiB   -807.8 MiB
   126    753.1 MiB   -807.8 MiB
   127                          
   128    753.1 MiB   -807.8 MiB
   129    753.1 MiB   -807.8 MiB
   130    753.1 MiB   -807.8 MiB
   131    753.1 MiB   -807.8 MiB
   132                          
   133    753.1 MiB   -807.8 MiB
   134                          
   135    753.1 MiB   -807.8 MiB
   136    753.1 MiB   -807.8 MiB
   137                          
   138                          
   139    753.1 MiB   -807.7 MiB
   140    753.1 MiB   -807.7 MiB
   141                          
   142    753.1 MiB   -807.8 MiB
   143                          
   144    752.9 MiB     -0.2 MiB
   145                          
   146                          
   147                          
   148                          
   149                          
   150                          
   151                          
   152    752.9 MiB      0.0 MiB

[alowayed]

Similar issue here. Totals don't add up and the increments are large and negative.

Line #    Mem usage    Increment   Line Contents
================================================
     5   23.891 MiB   23.891 MiB
     6
     7
     8   23.891 MiB    0.000 MiB
     9   23.891 MiB    0.000 MiB
    10
    11   24.137 MiB    0.246 MiB
    12
    13
    14  112.980 MiB   88.844 MiB
    15
    16  112.980 MiB    0.000 MiB
    17
    18  136.840 MiB -2654.883 MiB
    19
    20  136.840 MiB -2676.617 MiB

[fabianp]

Thanks @nathantsoi and @alowayed for the report. This bug doesn't seem to appear in the standard "test" suite (i.e., typing make test from the root directory). Could you paste a minimal reproducing example for this?

[karan10111]

Any progress on this?

[astrojuanlu]

@karan10111 we would need some help in having a short, self-contained, correct example to reproduce this. Do you have a short snippet that we can use for that?

[alowayed]

@fabianp I'll need some time to reduce the actual code. This appeared while an XML tree was being walked through Hadoop.

[karan10111]

So, most of the times you will see this in loops.
train, test = next(islice(cv.split(X, y), 2, 3))

cv is from sklearn.

[fabianp]

Thanks for the info. That makes sense. When a line is executed multiple times, only the maximum is displayed, which might lead to some confusing information being displayed. PRs or ideas on how to handle this better are welcome.

[astrojuanlu]

Unless we find other cases where this happens, I guess this is a duplicate of #9.

[astrojuanlu]

@fabianp proposed a fix for this in #202. @karan10111 @alowayed @nathantsoi could you please try pip install https://github.com/fabianp/memory_profiler/archive/fix_195.zip and rerun your examples?