[Python] mutex lock failed: Invalid argument when importing after tink (M1, macOS 14.2.1)

[reeseann]

Describe the bug, including details regarding any error messages, version, and platform.

• macOS 14.2.1, M1
• clang 15.0.0 (clang-1500.1.0.2.5, arm64-apple-darwin23.2.0)
• Python 3.8.18
• Pyarrow 15.0.0
• Tink 1.9.0 (install used prebuilt wheel tink-1.9.0-cp38-cp38-macosx_11_0_universal2.whl)
• OpenSSL 3.2.0 23 Nov 2023

import tink
import pyarrow

Results in:

libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

Reversing import order results in the process hanging with:

[mutex.cc : 453] RAW: Lock blocking 0x156747c38   @

Same error message as #15189. Just in case similar root cause, OpenSSL version Python is built with is OpenSSL 3.2.0 23 Nov 2023.

Component(s)

Python

[pitrou]

Can you perhaps obtain a backtrace using lldb?

[reeseann]

0.	Program arguments: /Library/Developer/CommandLineTools/usr/bin/lldb /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lldb                     0x000000010414f7dc llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  lldb                     0x000000010414ed38 llvm::sys::RunSignalHandlers() + 112
2  lldb                     0x000000010414fe14 SignalHandler(int) + 304
3  libsystem_platform.dylib 0x000000018180da24 _sigtramp + 56
4  libsystem_pthread.dylib  0x00000001817ddcc0 pthread_kill + 288
5  libsystem_c.dylib        0x00000001816e9a40 abort + 180
6  libc++abi.dylib          0x0000000181795070 __cxxabiv1::__aligned_malloc_with_fallback(unsigned long) + 0
7  libc++abi.dylib          0x0000000181785110 demangling_terminate_handler() + 320
8  libobjc.A.dylib          0x000000018142b99c _objc_terminate() + 160
9  libc++abi.dylib          0x0000000181794434 std::__terminate(void (*)()) + 16
10 libc++abi.dylib          0x0000000181797520 __cxa_get_exception_ptr + 0
11 libc++abi.dylib          0x0000000181797464 __cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*) + 0
12 libc++.1.dylib           0x00000001817136b8 std::__1::__throw_system_error(int, char const*) + 100
13 libc++.1.dylib           0x0000000181708318 std::__1::mutex::try_lock() + 0
14 libarrow.1500.dylib      0x0000000110d6a330 google::protobuf::internal::OnShutdownRun(void (*)(void const*), void const*) + 72
15 libarrow.1500.dylib      0x0000000110d61880 google::protobuf::internal::InitProtobufDefaultsSlow() + 84
16 libarrow.1500.dylib      0x0000000110dfaca8 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 128
17 libarrow.1500.dylib      0x0000000110dfacec google::protobuf::internal::AddDescriptorsRunner::AddDescriptorsRunner(google::protobuf::internal::DescriptorTable const*) + 24
18 dyld                     0x0000000181479a24 invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
19 dyld                     0x00000001814bf0f4 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 172
20 dyld                     0x00000001814b2668 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 496
21 dyld                     0x00000001814592fc dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 300
22 dyld                     0x00000001814b16a0 dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
23 dyld                     0x00000001814b4188 dyld3::MachOFile::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, bool&) block_pointer) const + 160
24 dyld                     0x00000001814bede8 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432
25 dyld                     0x0000000181475b38 dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 524
26 dyld                     0x000000018147bf70 dyld4::JustInTimeLoader::runInitializers(dyld4::RuntimeState&) const + 36
27 dyld                     0x0000000181475f24 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
28 dyld                     0x0000000181475f00 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
29 dyld                     0x0000000181475f00 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
30 dyld                     0x0000000181475f00 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
31 dyld                     0x0000000181475f00 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
32 dyld                     0x0000000181479ab0 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
33 dyld                     0x00000001814760f0 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 380
34 dyld                     0x00000001814947ac dyld4::APIs::dlopen_from(char const*, int, void*) + 1576
35 Python3                  0x0000000104ccedf4 _Py_GetLocaleconvNumeric + 756
36 Python3                  0x0000000104c9f66c PyImport_AppendInittab + 5100
37 Python3                  0x0000000104c9ee08 PyImport_AppendInittab + 2952
38 Python3                  0x0000000104beb188 PyCMethod_New + 764
39 Python3                  0x0000000104c77e50 _PyEval_EvalFrameDefault + 23428
40 Python3                  0x0000000104c7b7ec _PyEval_EvalFrameDefault + 38176
41 Python3                  0x0000000104ba8f10 _PyFunction_Vectorcall + 236
42 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
43 Python3                  0x0000000104c77b78 _PyEval_EvalFrameDefault + 22700
44 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
45 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
46 Python3                  0x0000000104c77b54 _PyEval_EvalFrameDefault + 22664
47 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
48 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
49 Python3                  0x0000000104c77bf4 _PyEval_EvalFrameDefault + 22824
50 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
51 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
52 Python3                  0x0000000104c77bf4 _PyEval_EvalFrameDefault + 22824
53 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
54 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
55 Python3                  0x0000000104c77bf4 _PyEval_EvalFrameDefault + 22824
56 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
57 Python3                  0x0000000104baa0cc PyObject_CallMethodObjArgs + 492
58 Python3                  0x0000000104baa258 _PyObject_CallMethodIdObjArgs + 112
59 Python3                  0x0000000104c9daf4 PyImport_ImportModuleLevelObject + 1284
60 Python3                  0x0000000104c76888 _PyEval_EvalFrameDefault + 17852
61 Python3                  0x0000000104c7b7ec _PyEval_EvalFrameDefault + 38176
62 Python3                  0x0000000104c72204 PyEval_EvalCode + 80
63 Python3                  0x0000000104c6eb5c _PyAST_ExprAsUnicode + 19156
64 Python3                  0x0000000104beb188 PyCMethod_New + 764
65 Python3                  0x0000000104c77e50 _PyEval_EvalFrameDefault + 23428
66 Python3                  0x0000000104c7b7ec _PyEval_EvalFrameDefault + 38176
67 Python3                  0x0000000104ba8f10 _PyFunction_Vectorcall + 236
68 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
69 Python3                  0x0000000104c77b78 _PyEval_EvalFrameDefault + 22700
70 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
71 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
72 Python3                  0x0000000104c77b54 _PyEval_EvalFrameDefault + 22664
73 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
74 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
75 Python3                  0x0000000104c77bf4 _PyEval_EvalFrameDefault + 22824
76 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
77 Python3                  0x0000000104c7aa38 _PyEval_EvalFrameDefault + 34668
78 Python3                  0x0000000104c77bf4 _PyEval_EvalFrameDefault + 22824
79 Python3                  0x0000000104ba8fc8 _PyFunction_Vectorcall + 420
80 Python3                  0x0000000104baa0cc PyObject_CallMethodObjArgs + 492
81 Python3                  0x0000000104baa258 _PyObject_CallMethodIdObjArgs + 112
82 Python3                  0x0000000104c9daf4 PyImport_ImportModuleLevelObject + 1284
83 Python3                  0x0000000104c76888 _PyEval_EvalFrameDefault + 17852
84 Python3                  0x0000000104c7b7ec _PyEval_EvalFrameDefault + 38176
85 Python3                  0x0000000104c72204 PyEval_EvalCode + 80
86 Python3                  0x0000000104cb62f0 PyParser_ASTFromStringObject + 500
87 Python3                  0x0000000104cb4eb4 PyRun_StringFlags + 152
88 LLDB                     0x0000000116b8f3f4 lldb_private::python::runStringMultiLine(llvm::Twine const&, lldb_private::python::PythonDictionary const&, lldb_private::python::PythonDictionary const&) + 148
89 LLDB                     0x0000000116b9503c lldb_private::ScriptInterpreterPythonImpl::ExecuteMultipleLines(char const*, lldb_private::ExecuteScriptOptions const&) + 884
90 LLDB                     0x0000000116b9a5c8 lldb_private::ScriptInterpreterPythonImpl::LoadScriptingModule(char const*, lldb_private::LoadScriptOptions const&, lldb_private::Status&, std::__1::shared_ptr<lldb_private::StructuredData::Object>*, lldb_private::FileSpec) + 1924
91 LLDB                     0x0000000116d9342c CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) + 328
92 LLDB                     0x00000001167a5ad0 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) + 656
93 LLDB                     0x000000011679c75c lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&, bool) + 2024
94 LLDB                     0x000000011679ff9c lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 828
95 LLDB                     0x00000001166d77c8 lldb_private::IOHandlerEditline::Run() + 304
96 LLDB                     0x00000001166bc0c4 lldb_private::Debugger::RunIOHandlers() + 140
97 LLDB                     0x00000001167a1178 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) + 196
98 LLDB                     0x000000011651d798 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 124
99 lldb                     0x00000001041402b8 Driver::MainLoop() + 2712
100 lldb                     0x0000000104140d14 main + 2036
101 dyld                     0x000000018145d0e0 start + 2360

[kou]

It seems that this is happen at https://github.com/protocolbuffers/protobuf/blob/976a6eb6a4f462c6bb0b2a87fcd95f0aeb17eedb/src/google/protobuf/message_lite.cc#L726 .

Is tink statically linked with its bundled Protobuf? (pyarrow does.) If tink is also linked with its bundled Protobuf, they may be conflicted.

[Inokinoki]

I started an investigation:

tink-crypto/tink-py#25 (comment)

My proposition is to upgrade the protobuf default version of pyarrow to a later one (in the CI), other than 3.21.X.

But it seems this would prevent building with OpenTelemetry.

[pitrou]

@Inokinoki I understand the explanation you give there, but it seems unexpected to me that there would be confusion between two private-linked static libraries. Especially as Python uses the default RTLD_LOCAL when loading extension modules, so there shouldn't be any name clashes when loading two different extension modules.

[pitrou]

I'm just seeing your stack trace btw:

87 Python3                  0x0000000104cb4eb4 PyRun_StringFlags + 152
88 LLDB                     0x0000000116b8f3f4 lldb_private::python::runStringMultiLine(llvm::Twine const&, lldb_private::python::PythonDictionary const&, lldb_private::python::PythonDictionary const&) + 148
89 LLDB                     0x0000000116b9503c lldb_private::ScriptInterpreterPythonImpl::ExecuteMultipleLines(char const*, lldb_private::ExecuteScriptOptions const&) + 884
90 LLDB                     0x0000000116b9a5c8 lldb_private::ScriptInterpreterPythonImpl::LoadScriptingModule(char const*, lldb_private::LoadScriptOptions const&, lldb_private::Status&, std::__1::shared_ptr<lldb_private::StructuredData::Object>*, lldb_private::FileSpec) + 1924
91 LLDB                     0x0000000116d9342c CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) + 328
92 LLDB                     0x00000001167a5ad0 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) + 656
93 LLDB                     0x000000011679c75c lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&, bool) + 2024
94 LLDB                     0x000000011679ff9c lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 828
95 LLDB                     0x00000001166d77c8 lldb_private::IOHandlerEditline::Run() + 304
96 LLDB                     0x00000001166bc0c4 lldb_private::Debugger::RunIOHandlers() + 140
97 LLDB                     0x00000001167a1178 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) + 196
98 LLDB                     0x000000011651d798 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 124
99 lldb                     0x00000001041402b8 Driver::MainLoop() + 2712
100 lldb                     0x0000000104140d14 main + 2036
101 dyld                     0x000000018145d0e0 start + 2360

Is this the Python scripting facility for LLDB? Is it where the crash is happening?

[Inokinoki]

Thanks for the update! Yes that's also my doubt. I will keep that in mind and try to continue debugging to figure out what is the case.

…

On Wed, May 7, 2025, 21:12 Antoine Pitrou ***@***.***> wrote: *pitrou* left a comment (apache/arrow#40088) <#40088 (comment)> @Inokinoki <https://github.com/Inokinoki> I understand the explanation you give there, but it seems unexpected to me that there would be confusion between two private-linked static libraries. Especially as Python uses the default RTLD_LOCAL <https://github.com/python/cpython/blob/3dfed230928de0f649061782a36691066a0ef058/Include/internal/pycore_import.h#L34-L45> when loading extension modules, so there shouldn't be any name clashes when loading two different extension modules. — Reply to this email directly, view it on GitHub <#40088 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7NEBH3ECSYOBUEYITITU325IBFLAVCNFSM6AAAAAB4TN5P26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJYGU2DOMBXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Inokinoki]

Thanks for the update! Yes that's also my doubt.

I will keep that in mind and try to continue debugging to figure out what
is the case.
…

I further investigated it. And the current conclusion is similar, it's the 2 different versions of google::protobuf::internal::OnShutdownRun which causes the issue.

But they are good in the two libraries, because eventually they are loaded implicitly using RTLD_LOCAL as you said @pitrou .

I think that it's because of the singleton data they are referring - google::protobuf::internal::ShutdownData::get()::data, which is created only once on demand:

auto shutdown_data = ShutdownData::get();

This data is expected to be different in the 2 versions (one with absl mutex as a member and another with std::mutex as a member).

I'm not sure whether it's an issue from the loader on macOS or not, but it seems that the "local" variable is shared between the different versions, which caused the issue.

---

Here are the debugging info:

So I set breakpoint on google::protobuf::internal::OnShutdownRun and then import pyarrow first. The assembly of it in libarrow is as follows:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2
    frame #0: 0x000000010732a0f4 libarrow.2000.dylib`google::protobuf::internal::OnShutdownRun(void (*)(void const*), void const*)
libarrow.2000.dylib`google::protobuf::internal::OnShutdownRun:
    0x10732a0f4 <+0>:  stp    x26, x25, [sp, #-0x50]!
    0x10732a0f8 <+4>:  stp    x24, x23, [sp, #0x10]
    0x10732a0fc <+8>:  stp    x22, x21, [sp, #0x20]
    0x10732a100 <+12>: stp    x20, x19, [sp, #0x30]
    0x10732a104 <+16>: stp    x29, x30, [sp, #0x40]
    0x10732a108 <+20>: add    x29, sp, #0x40
    0x10732a10c <+24>: mov    x20, x1
    0x10732a110 <+28>: mov    x21, x0
    0x10732a114 <+32>: adrp   x8, 1794
->  0x10732a118 <+36>: ldr    x8, [x8, #0x340]
    0x10732a11c <+40>: ldaprb w8, [x8]
    0x10732a120 <+44>: adrp   x19, 1796
    0x10732a124 <+48>: ldr    x19, [x19, #0x50]
    0x10732a128 <+52>: tbz    w8, #0x0, 0x10732a238 ; <+324>
    0x10732a12c <+56>: ldr    x22, [x19]
    0x10732a130 <+60>: add    x19, x22, #0x18
    0x10732a134 <+64>: mov    x0, x19
    0x10732a138 <+68>: bl     0x107531dd0    ; symbol stub for: std::__1::mutex::lock()
    0x10732a13c <+72>: ldp    x23, x8, [x22, #0x8]
    0x10732a140 <+76>: cmp    x23, x8
    0x10732a140 <+76>: cmp    x23, x8
    0x10732a144 <+80>: b.hs   0x10732a158    ; <+100>
    0x10732a148 <+84>: stp    x21, x20, [x23]
    0x10732a14c <+88>: add    x8, x23, #0x10 

where the marked instruction is to get the singleton data. When I get the register bank, it contains:

x8 = 0x0000000107b2b8c8  guard variable for google::protobuf::internal::ShutdownData::get()::data

We can notice that 0x10732a138 <+68> contains a direct call to standard C++ lib of mutex lock. And it's ok here because the singleton data also contains a std::mutex member.

I let it continue running and then import tink (with a newer version using absl mutex).

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.3
    frame #0: 0x0000000103f5b104 tink_bindings.cpython-310-darwin.so`google::protobuf::internal::OnShutdownRun(void (*)(void const*), void const*)
tink_bindings.cpython-310-darwin.so`google::protobuf::internal::OnShutdownRun:
    0x103f5b104 <+0>:  stp    x28, x27, [sp, #-0x60]!
    0x103f5b108 <+4>:  stp    x26, x25, [sp, #0x10]
    0x103f5b10c <+8>:  stp    x24, x23, [sp, #0x20]
    0x103f5b110 <+12>: stp    x22, x21, [sp, #0x30]
    0x103f5b114 <+16>: stp    x20, x19, [sp, #0x40]
    0x103f5b118 <+20>: stp    x29, x30, [sp, #0x50]
    0x103f5b11c <+24>: add    x29, sp, #0x50
    0x103f5b120 <+28>: mov    x20, x1
    0x103f5b124 <+32>: mov    x21, x0
    0x103f5b128 <+36>: adrp   x8, 501
->  0x103f5b12c <+40>: ldr    x8, [x8, #0xe8]
    0x103f5b130 <+44>: ldaprb w8, [x8]
    0x103f5b134 <+48>: adrp   x19, 502
    0x103f5b138 <+52>: ldr    x19, [x19, #0xaa0]
    0x103f5b13c <+56>: tbz    w8, #0x0, 0x103f5b218 ; <+276>
    0x103f5b140 <+60>: ldr    x22, [x19]
    0x103f5b144 <+64>: add    x19, x22, #0x18
    0x103f5b148 <+68>: mov    x0, x19
->  0x103f5b14c <+72>: bl     0x104026a88    ; absl::lts_20240722::Mutex::Lock()
    0x103f5b150 <+76>: ldp    x9, x8, [x22, #0x8]
    0x103f5b154 <+80>: cmp    x9, x8
    0x103f5b158 <+84>: b.hs   0x103f5b16c    ; <+104>

When I read the register, it's giving the same address (at the first arrow):

x8 = 0x0000000107b2b8c8  guard variable for google::protobuf::internal::ShutdownData::get()::data

The data is already created while importing pyarrow, and it has a member of std::mutex.

But then it calls the absl mutex lock (at the second arrow), which expects absl mutex, which can crash the program.

---

Hope that this helps!

[pitrou]

Thanks @Inokinoki . Unfortunately, I don't think there's anything we can do in Arrow against this. You might try to open a ticket at protobuf, perhaps the developers there would be able to explain why that happens exactly (I don't understand why the static variable would be "shared" between the two private protobuf instances).