Kernel panic after unloading and reloading driver

[ZhendanYang]

Summary:
Run the "soft/kmod_scripts/sof_bootone.sh" script, which means to do the module reload once, but after unloading and reloading driver, kernel panic happens.

[ 31.980316] sof-audio sof-audio: warn: can't find PCM with DAI ID 0
[ 31.980776] sof-audio sof-audio: tag 1 not opened!
[ 31.980817] sof-audio sof-audio: DSP core(s) enabled? 0 : core_mask f
[ 32.301721] sof-audio-pci 0000:00:1f.3: PCI DSP detected
[ 32.301952] sof-audio-pci 0000:00:1f.3: No matching ASoC machine driver found - using nocodec

[ 32.306463] Call Trace:
[ 32.306468] hda_codec_i915_init+0x19/0x60 [snd_sof_intel_hda]
[ 32.306472] hda_dsp_probe+0x458/0x8f0 [snd_sof_intel_hda_common]
[ 32.306477] sof_probe+0x145/0x320 [snd_sof]
[ 32.306483] platform_drv_probe+0x3a/0x90
[ 32.306485] ? driver_sysfs_add+0x75/0xd0
[ 32.306487] driver_probe_device+0x2fa/0x470
[ 32.306489] ? __driver_attach+0x100/0x100
[ 32.306491] bus_for_each_drv+0x66/0xb0
[ 32.306494] __device_attach+0xd9/0x150
[ 32.306496] bus_probe_device+0x8a/0xa0
[ 32.306497] device_add+0x3f7/0x620
[ 32.306500] ? _cond_resched+0x15/0x40
[ 32.306503] platform_device_add+0x10e/0x260
[ 32.306506] platform_device_register_full+0xf0/0x100
[ 32.306509] sof_create_platform_device+0x5d/0xa0 [snd_sof]
[ 32.306513] sof_pci_probe+0x232/0x346 [sof_pci_dev]
[ 32.306518] local_pci_probe+0x42/0xa0
[ 32.306520] ? pci_assign_irq+0x27/0x130
[ 32.306522] pci_device_probe+0x146/0x1b0
[ 32.306524] driver_probe_device+0x2fa/0x470
[ 32.306526] __driver_attach+0xdc/0x100
[ 32.306528] ? driver_probe_device+0x470/0x470
[ 32.306529] bus_for_each_dev+0x76/0xc0
[ 32.306531] bus_add_driver+0x161/0x260
[ 32.306533] ? 0xffffffffc05c9000
[ 32.306535] driver_register+0x5b/0xe0
[ 32.306536] ? 0xffffffffc05c9000
[ 32.306539] do_one_initcall+0x46/0x1c8
[ 32.306542] ? kobject_uevent_env+0x117/0x810
[ 32.306544] ? _cond_resched+0x15/0x40
[ 32.306546] ? kmem_cache_alloc_trace+0x3e/0x1c0
[ 32.306549] do_init_module+0x5b/0x201
[ 32.306552] load_module+0x2689/0x2e00
[ 32.306556] ? __do_sys_finit_module+0xe9/0x110
[ 32.306558] __do_sys_finit_module+0xe9/0x110
[ 32.306560] do_syscall_64+0x55/0x110
[ 32.306563] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 32.306565] RIP: 0033:0x7f6ae6e7da49
[ 32.306565] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0f b4 2c 00 f7 d8 64 89 01 48
[ 32.306592] RSP: 002b:00007ffdaeec0b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 32.306593] RAX: ffffffffffffffda RBX: 000055842d93ce90 RCX: 00007f6ae6e7da49
[ 32.306594] RDX: 0000000000000000 RSI: 000055842d1a6cee RDI: 0000000000000004
[ 32.306595] RBP: 000055842d1a6cee R08: 0000000000000000 R09: 0000000000000000
[ 32.306596] R10: 0000000000000004 R11: 0000000000000246 R12: 0000000000000000
[ 32.306597] R13: 000055842d93d3a0 R14: 0000000000040000 R15: 000055842d93ce90
[ 32.306598] ---[ end trace 37c92cd1f0a66c62 ]---

[ 32.306646] sof-audio sof-audio: error: failed to probe DSP -16
[ 32.306654] sof-audio: probe of sof-audio failed with error -16
[ 32.306933] sdhci-pci 0000:00:14.5: SDHCI controller found [8086:9df5] (rev 11)
[ 32.307753] sof-nocodec sof-nocodec: ASoC: CPU DAI SSP0 Pin not registered

Test env:
kernel: c3ba91f
sof stable-1.2: 0875f0c
soft: d77beaf
tplg: sof-apl-nocodec.tplg on APL and test-ssp0-mclk-0-I2S-volume-s16le-s16le-48k-24000k-nocodec.tplg on CNL

Log:
dmesg-apl.log
dmesg-cnl.log

[lyakh]

First of all I think the topic of this bug is not quite correct: this isn't a kernel panic, this is a kernel WARNING. Still, it has to be fixed of course.

I've tried to reproduce the bug but failed so far. The difference is, that my Up^2 has no codec. I tried both topology variants - with a codec and a "nocodec" one. In both cases kernel was reporting some errors, so, apparently the drivers didn't complete the registration, which might then explain, why a different path was taken when unloading and reloading modules, so no warnings were generated. I also noticed, that not all snd-soc modules were unloaded by the "remove" script. These modules were kept:

snd_soc_sst_bxt_pcm512x    16384  0
snd_soc_pcm512x_i2c    16384  0
snd_soc_pcm512x        32768  1 snd_soc_pcm512x_i2c
snd_seq_midi           16384  0
snd_seq_midi_event     16384  1 snd_seq_midi
snd_rawmidi            36864  1 snd_seq_midi
snd_sof                98304  0
snd_sof_xtensa_dsp     20480  0
snd_soc_core          258048  3 snd_soc_pcm512x,snd_soc_sst_bxt_pcm512x,snd_sof

I modified the script to also unload those, and still got no warning. @ZhendanYang could you maybe try to reproduce the problem with the codec board unplugged? To check, whether I've done something wrong or indeed it only occurs with a codec?

[ZhendanYang]

@lyakh
I modified the "remove" script to unload those snd-soc modules.
I add some lines:
remove_module snd_soc_hdac_hdmi
remove_module snd_soc_hdac_hda
remove_module snd_soc_core
remove_module snd_seq_midi
remove_module snd_seq_midi_event
remove_module snd_rawmidi
And when I run sof_remove.sh, system crashed. I had to reboot through the power button.

[lyakh]

@ZhendanYang do you have a kernel log from that crash? Logs, attached to this issue "only" contain a kernel warning. Have you tried with no codec?

[lyakh]

I've got a "nocodec" Up^2 configuration that seems to work - at least aplay and arecord return after some time and I'm seeing interrupts. Although the time is larger than it should be. And I've got an Oops! Investigating...

[lyakh]

I think I've found the reason for this Oops. It seems to be a case of a use after free. First the DAI driver gets freed along the lines of

sof_pcm_free()
snd_sof_free_topology()
snd_soc_tplg_component_remove()
remove_dai()

And then as resources are freed the driver's .remove method is verified, initially it was NULL, but since the memory was released, it seems to be often reused in our use-cases and the .remove pointer then contains random data, if that memory is again accessible, of course. That happens as

devm_card_release()
snd_soc_unregister_card()
soc_cleanup_card_resources()
soc_remove_dai_links()
soc_remove_link_dais()
soc_remove_dai()
dai->driver->remove()

I'm looking at ways to fix this.