Qualcomm AI Engine Direct - Support PARSeq in floating point precision

[shewu-quic]

Summary:

• Support triu
• The weight node of the linear node was the output of another node rather than being static.
• Using ctypes.CDLL(xxx, mode=ctypes.RTLD_GLOBAL) is causing unexpected behavior. The issue is still under investigation.

[ERROR] [Qnn ExecuTorch]:  <E> Failed to create InputDef for Op aten_permute_copy_default (qti.aisw::Transpose) param perm

[ERROR] [Qnn ExecuTorch]:  <E> Failed to set the params for the op: Transpose

[ERROR] [Qnn ExecuTorch]:  <E> Failed to construct common node for graph 256

[ERROR] [Qnn ExecuTorch]:  <E> Failed to add node with err 6001

[ERROR] [Qnn ExecuTorch]: Failed to add node to Qnn Graph with error: 6001
[ERROR] [Qnn ExecuTorch]: Fail to compile QNN graph

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15913

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Cancelled Jobs

As of commit e5d085e with merge base e0dda90 ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  The process '/usr/bin/sh' failed with exit code 255
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/passes/test_channels_last_tagged_reshape.py::TestChannelsLastTaggedReshapePass::test_q_dq_nodes_around_copy_are_tagged

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• pull / test-llava-runner-linux / linux-job (gh)
  ##[error]The operation was canceled.
• pull / unittest / macos / macos-job (gh)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[cccclai]

Thank you for putting up the PR! For the change in backends/qualcomm/scripts/download_qnn_sdk.py, does it mean the SDK is from pip wheel package?

[shewu-quic]

Thank you for putting up the PR! For the change in backends/qualcomm/scripts/download_qnn_sdk.py, does it mean the SDK is from pip wheel package?

Yes, I created a new virtual environment and tried to reproduce the error by running pip install executorch.
I found that I couldn't export any graph using it in the QNN Backend. It appears the issue is due to parameters not being passed correctly to QNN.
However, I'm not sure why this change would cause the prepare step to fail. Do you have any thoughts?

[shewu-quic]

Hi @cccclai,

From my experiment, the issue is related to dynamic_cast, where p_tensor_param becomes nullptr even when param is a TensorParamWrapper*.
This appears to be due to differences between clang and the GNU C++ library, which affect RTTI matching.
When PyQnnXXX is built with clang, the cast results in nullptr regardless of RTLD_GLOBAL or RTLD_LIBRARY.
By default, PyQnnXXX is built with the GNU C++ library.
If you CDLL load the clang C++ library with GLOBAL, the cast fails and returns nullptr.
However, if you CDLL load the GNU C++ library first or load the clang C++ library with LOCAL, the cast works as expected.

I also replaced dynamic_cast with the code below, and confirmed that the typeid of param matches typeid(TensorParamWrapper).

if (typeid(*param.get()) ==typeid(TensorParamWrapper) ){
    p_tensor_param = static_cast<TensorParamWrapper*>(param.get());
}

Do you have any thoughts? Or do you have any concerns about loading the library with RTLD_LOCAL?”

[cccclai]

Hmm I feel like we should find a way to build both libraries with the same compiler + same C++ standard library + same ABI settings, it's actually quite important for next release, maybe need to figure out if there is a way to do that

[shewu-quic]

Hi @cccclai,
Yes, I agree with you. I looked into this issue further and created a minimal example. It appears to be related to how RTTI/typeinfo handles differ between clang and gcc. The root cause is that the typeinfo for TensorParamWrapper exists in both PyQnnManagerAdaptor.so and PyQnnWrapperAdaptor.so (Check with nm -C --defined-only XXX.so). We create a TensorParamWrapper with PyQnnWrapperAdaptor, then pass it to PyQnnManagerAdaptor and use dynamic_cast with param in PyQnnManagerAdaptor.so. With gcc, dynamic_cast works successfully even if the definitions are across different shared objects. However, with clang, (I guess) it treats TensorParamWrapper as different types in each shared object. As a result, when we load the clang library, using dynamic_cast returns nullptr, while gcc casts as expected.

So, I think we should combine these two into one shared object to resolve the issue though it will be a big change.
How do you see it?