AirSim crashes when running against PX4 v1.8.2 SITL

[julianoes]

I'm following the guide in https://microsoft.github.io/AirSim/docs/px4_sitl/.

I've built PX4 SITL v1.8.2 and I use Blocks from: https://github.com/microsoft/AirSim/releases/tag/v1.2.0Linux

I also have QGC with a joystick configured running.

On startup it crashes. I guess it's something that the MAVLink libs used are not matching.

Backtrace:

#0  0x00007ffff75e4755 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff75cf851 in abort () from /usr/lib/libc.so.6
#2  0x0000000006854d5d in abort_message ()
#3  0x0000000006854e69 in default_terminate_handler() ()
#4  0x0000000006854c03 in std::__terminate(void (*)()) ()
#5  0x0000000006854686 in __cxa_throw ()
#6  0x0000000006227cce in mavlinkcom_impl::MavLinkConnectionImpl::prepareForSending (this=0x7fffc59899a0, msg=...)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:262
#7  0x00000000062275bd in mavlinkcom_impl::MavLinkConnectionImpl::sendMessage (this=0x7fffc59899a0, m=...)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:183
#8  0x000000000622915d in mavlinkcom_impl::MavLinkConnectionImpl::joinRightSubscriber (this=0x7fffc59899a0, connection=..., msg=...)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:347
#9  0x000000000623485c in std::__1::__invoke<void (mavlinkcom_impl::MavLinkConnectionImpl::*&)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*&, std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&, void> (__f=
    @0x7fffe6094c58: (void (mavlinkcom_impl::MavLinkConnectionImpl::*)(mavlinkcom_impl::MavLinkConnectionImpl * const, std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, const mavlinkcom::MavLinkMessage &)) 0x6229130 <mavlinkcom_impl::MavLinkConnectionImpl::joinRightSubscriber(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&)>, __a0=@0x7fffe6094c68: 0x7fffc59899a0, __args=..., __args=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/type_traits:4264
#10 std::__1::__apply_functor<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), std::__1::tuple<mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1>, std::__1::placeholders::__ph<2> >, 0ul, 1ul, 2ul, std::__1::tuple<std::__1::shared_ptr<mavlinkcom::MavLinkConnection>&&, mavlinkcom::MavLinkMessage const&> > (__f=
    @0x7fffe6094c58: (void (mavlinkcom_impl::MavLinkConnectionImpl::*)(mavlinkcom_impl::MavLinkConnectionImpl * const, std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, const mavlinkcom::MavLinkMessage &)) 0x6229130 <mavlinkcom_impl::MavLinkConnectionImpl::joinRightSubscriber(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&)>, __bound_args=..., __args=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/functional:2219
#11 std::__1::__bind<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>::operator()<std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&> (this=0x7fffe6094c58, __args=..., __args=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/functional:2252
#12 std::__1::__invoke<std::__1::__bind<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&> (__f=..., __args=..., __args=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/type_traits:4323
#13 std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&> (__args=..., __args=..., __args=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/__functional_base:349
#14 0x00000000062342cc in std::__1::__function::__func<std::__1::__bind<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::_--Type <RET> for more, q to quit, c to continue without paging--
_ph<2> const&>, std::__1::allocator<std::__1::__bind<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&), mavlinkcom_impl::MavLinkConnectionImpl*, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&> >, void (std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&)>::operator()(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>&&, mavlinkcom::MavLinkMessage const&) (this=0x7fffe6094c50, __arg=..., __arg=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/functional:1562
#15 0x000000000622d202 in std::__1::function<void (std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&)>::operator()(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, mavlinkcom::MavLinkMessage const&) const (this=0x7fffe6094c50, __arg=..., __arg=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/functional:1916
#16 0x000000000622aa0d in mavlinkcom_impl::MavLinkConnectionImpl::drainQueue (this=0x7fffc598a960)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:492
#17 0x0000000006225c68 in mavlinkcom_impl::MavLinkConnectionImpl::publishPackets (this=0x7fffc598a960)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:516
#18 0x000000000622eea4 in std::__1::__invoke<void (mavlinkcom_impl::MavLinkConnectionImpl::*)(), mavlinkcom_impl::MavLinkConnectionImpl*, , void> (
    __f=@0x7fffe73d2068: (void (mavlinkcom_impl::MavLinkConnectionImpl::*)(mavlinkcom_impl::MavLinkConnectionImpl * const)) 0x6225c30 <mavlinkcom_impl::MavLinkConnectionImpl::publishPackets()>, __a0=@0x7fffe73d2078: 0x7fffc598a960)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/type_traits:4264
#19 std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (mavlinkcom_impl::MavLinkConnectionImpl::*)(), mavlinkcom_impl::MavLinkConnectionImpl*, 2ul> (__t=...)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/thread:342
#20 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (mavlinkcom_impl::MavLinkConnectionImpl::*)(), mavlinkcom_impl::MavLinkConnectionImpl*> > (__vp=0x7fffe73d2060)
    at /home/madratman/projects/AirSim/llvm-build/output/include/c++/v1/thread:352
#21 0x00007ffff7f4757f in start_thread () from /usr/lib/libpthread.so.0
#22 0x00007ffff76a60e3 in clone () from /usr/lib/libc.so.6

This is the full startup: https://pastebin.com/D9atgQJM

And that's my config:

{
    "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
    "SettingsVersion": 1.2,
    "SimMode": "Multirotor",
    "Vehicles": {
        "PX4": {
            "VehicleType": "PX4Multirotor",
            "UseSerial": false
        }
    }
}

[JaySurplus]

Hi @julianoes, Same problem here. Do you have any solution?
Thanks

[julianoes]

I plan to debug it in the next weeks.

[julianoes]

I can reproduce this crash even without QGC.

[julianoes]

@lovettchris here is what I found debugging this:

(gdb) f 6
#6  0x0000000006227cce in mavlinkcom_impl::MavLinkConnectionImpl::prepareForSending (this=0x7fffdc62a180, msg=...)
    at /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:262
262	in /home/madratman/projects/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp
(gdb) p len
$6 = 30
(gdb) p msglen
$7 = 33
(gdb) p mavlink1
$8 = true
(gdb) p msg.msgid
$9 = 76

Note that this is with Airsim 1.2.0 so maybe it has already been fixed since February but given there hasn't been a recent Linux release, it's still there.

Edit: yes there have been changes exactly there: v1.2.0Linux...master#diff-c274834d665afd29ec99728369bc7f31

[lovettchris]

Can you also try my latest bits in this pull request?:
#2310

[julianoes]

With master this doesn't happen anymore, only with 1.2.0 on Linux.