Skip to content

implement some more memory optimizations for article collection #90

@frnsys

Description

@frnsys

using a generator for membrane.collector.collect helped a lot, but the ponder process is still killed for too much memory consumption (albeit after much longer than it used to be).

so there are still some optimizations which can happen to make this not such a ridiculous memory strain.

take a look at:
http://guillaume.segu.in/blog/code/487/optimizing-memory-usage-in-python-a-case-study/

for profiling memory usage, look at heapy:
https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended

potentially use namedtuples?
https://docs.python.org/2/library/collections.html#collections.namedtuple

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions