setting cpu affinity doesn't work on py3

[Valodim]

python[8411]: 2018-09-27 18:54:25,831 - root - 212 - WARNING - None- ***** STARTING SERVER *****
python[8411]: 2018-09-27 18:54:25,848 - root - 215 - WARNING - None- Server /home/valodim/cpy/lib/python3.5/site-packages/synapse/app/homeserver.py v
python[8411]: 2018-09-27 18:54:25,858 - twisted - 243 - WARNING - None- /home/valodim/cpy/lib/python3.5/site-packages/psycopg2/__init__.py:144: built
python[8411]: 2018-09-27 18:54:26,048 - twisted - 243 - ERROR - - ******************************************************
python[8411]: 2018-09-27 18:54:26,049 - twisted - 243 - ERROR - -  Missing package 'affinity' required for cpu_affinity
python[8411]: 2018-09-27 18:54:26,050 - twisted - 243 - ERROR - -  option
python[8411]: 2018-09-27 18:54:26,050 - twisted - 243 - ERROR - -
python[8411]: 2018-09-27 18:54:26,051 - twisted - 243 - ERROR - -  Install by running:
python[8411]: 2018-09-27 18:54:26,051 - twisted - 243 - ERROR - -
python[8411]: 2018-09-27 18:54:26,052 - twisted - 243 - ERROR - -     pip install affinity
python[8411]: 2018-09-27 18:54:26,052 - twisted - 243 - ERROR - -
python[8411]: 2018-09-27 18:54:26,052 - twisted - 243 - ERROR - -
python[8411]: 2018-09-27 18:54:26,053 - twisted - 243 - ERROR - - ******************************************************

the affinity package is installed, I ran pip in that virtualenv to make sure, and checked it's there in site-packages. not sure why synapse can't find the package.

perhaps the setting no longer makes sense with py3? anyways this prevents synapse from starting for me. commenting out the setting works fine, however.

This is on 0.33.5.1

[Half-Shot]

Probably the first two questions to ask are "did you install this on pip3 rather than pip, which is usually the py2 version" and "how did you install synapse?"

[Valodim]

> bin/pip --version
pip 18.0 from /home/valodim/cpy/lib/python3.5/site-packages/pip (python 3.5)
> bin/pip install affinity
Requirement already satisfied: affinity in ./lib/python3.5/site-packages (0.1.0)
> bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
[above error]

synapse is installed in the virtualenv with bin/pip install ./synapse

[hawkowl]

This is something that should work on Python 3, so I'll take a look. Thanks for the report, @Valodim .

[logic]

FWIW, it looks like the affinity module hasn't been ported to Python 3, and appears to be a dead project. PyPI says the last release was March 23, 2006, and just importing it in Python3 throws syntax errors around exception syntax:

>>> import affinity
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/srv/matrix.logic.net/venv3.6/lib/python3.6/site-packages/affinity/__init__.py", line 29
    except pywintypes.error, e:
                           ^
SyntaxError: invalid syntax
>>>

[hawkowl]

That's rather unfortunate.

However, looking at what it actually wraps, I believe a CFFI-using replacement which covers our purposes (Linux only) would be almost trivial to write, so I'll take a stab at that next week.

[ara4n]

Do we even need this on Py3? The reason it was added in the first place was due to https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/, which i thought might have been addressed in Python 3.5+?

[hawkowl]

No -- it's up to the operating system to decide where to schedule something, not the application. Linux schedulers sometimes try and avoid breaking caches and keep things on the same CPU, but that isn't always the case. Doing a bit of research, it appears psutil has CPU affinity setting, and since I believe we already depend on that library, we can use that instead. https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity

…

On Fri, 5 Oct 2018, 08:32 Matthew Hodgson, ***@***.***> wrote: Do we even need this on Py3? The reason it was added in the first place was due to https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/, which i thought might have been addressed in Python 3.5+? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#3973 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJ2XJbzxWxCSWe5JPaHhOApiGEKJVN6ks5uhoyLgaJpZM4W9dwR> .

[erikjohnston]

No -- it's up to the operating system to decide where to schedule
something, not the application. Linux schedulers sometimes try and avoid
breaking caches and keep things on the same CPU, but that isn't always the
case.

The problem is that it interacts badly with the implementation of the GIL, since the OS may schedule the threads onto multiple cores, but then the python threads fight/thrash over the GIL locks.

It looks like the GIL may have changed to make it thrash less in python 3.2 http://www.dabeaz.com/python/UnderstandingGIL.pdf ?

If psutil has an implementation, lets just use that. Once we get matrix.org on py3 we should definitely try removing the CPU affinity options to see if the thrashing has indeed been fixed or not, since pinning to a single core does have some performance penalties.

[hawkowl]

Fixed in #4027