Skip to content

Tidying of OpenSSL 1.0.2/Python 3.9 (and earlier) handling #5854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jakirkham merged 5 commits into from
Mar 12, 2022
Merged

Tidying of OpenSSL 1.0.2/Python 3.9 (and earlier) handling #5854

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
98dec9b
Rename `C_INT_MAX` for clarity
jakirkham Feb 24, 2022
e630c6a
Move comment next to global variable
jakirkham Feb 24, 2022
1fee0ba
Link Python bug report related to this
jakirkham Feb 24, 2022
2f7eb2f
Use larger OpenSSL chunk size limit w/Python 3.10+
jakirkham Feb 24, 2022
3b0622d
Specify Python 3.10+ connection
jakirkham Feb 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions distributed/comm/tcp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,14 @@
logger = logging.getLogger(__name__)


C_INT_MAX = 256 ** ctypes.sizeof(ctypes.c_int) // 2 - 1
# Workaround for OpenSSL 1.0.2.
# Can drop with OpenSSL 1.1.1 used by Python 3.10+.
# ref: https://bugs.python.org/issue42853
if sys.version_info < (3, 10):
OPENSSL_MAX_CHUNKSIZE = 256 ** ctypes.sizeof(ctypes.c_int) // 2 - 1
else:
OPENSSL_MAX_CHUNKSIZE = 256 ** ctypes.sizeof(ctypes.c_size_t) - 1
Comment on lines +46 to +52
Copy link
Copy Markdown
Member

@fjetter fjetter Feb 28, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the comment about dropping this with py3.10 still true or why do we need another constant? Or is the py3.10 constant simply max integer and therefore a noop or smth like that?

Copy link
Copy Markdown
Member Author

@jakirkham jakirkham Feb 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly in the Python 3.10 case OPENSSL_MAX_CHUNKSIZE is the maximum value size_t can contain. IOW the largest integer. So is effectively a no-op.


MAX_BUFFER_SIZE = MEMORY_LIMIT / 2


Expand Down Expand Up @@ -212,9 +219,9 @@ async def read(self, deserializers=None):
(frames_nbytes,) = struct.unpack(fmt, frames_nbytes)

frames = host_array(frames_nbytes)
# Workaround for OpenSSL 1.0.2 (can drop with OpenSSL 1.1.1)
for i, j in sliding_window(
2, range(0, frames_nbytes + C_INT_MAX, C_INT_MAX)
2,
range(0, frames_nbytes + OPENSSL_MAX_CHUNKSIZE, OPENSSL_MAX_CHUNKSIZE),
):
chunk = frames[i:j]
chunk_nbytes = len(chunk)
Expand Down Expand Up @@ -351,8 +358,7 @@ class TLS(TCP):
A TLS-specific version of TCP.
"""

# Workaround for OpenSSL 1.0.2 (can drop with OpenSSL 1.1.1)
max_shard_size = min(C_INT_MAX, TCP.max_shard_size)
max_shard_size = min(OPENSSL_MAX_CHUNKSIZE, TCP.max_shard_size)

def _read_extra(self):
TCP._read_extra(self)
Expand Down