Disconnecting from Wi-Fi makes it unusable

[minlexx]

When trying to disconnect from Wi-Fi (either manually, but more often this happens on shutdown) the following errors are observed in dmesg:

[ 2492.275144] wlan0: deauthenticating from 60:e3:27:30:a7:0c by local choice (Reason: 3=DEAUTH_LEAVING)
[ 2495.459554] ath10k_snoc 18800000.wifi: failed to install key for vdev 0 peer 60:e3:27:30:a7:0c: -110
[ 2495.459605] wlan0: failed to remove key (0, 60:e3:27:30:a7:0c) from hardware (-110)
[ 2495.713730] qcom-q6v5-mss 4080000.remoteproc: fatal error received: err_qdi.c:459:EX:wlan_process:1:WLAN RT:207f:PC=b00c87e0
[ 2495.713786] remoteproc remoteproc0: crash detected in 4080000.remoteproc: type fatal error
[ 2495.713873] remoteproc remoteproc0: handling crash #1 in 4080000.remoteproc
[ 2495.713888] remoteproc remoteproc0: recovering 4080000.remoteproc
[ 2495.743237] ath10k_snoc 18800000.wifi: firmware crashed! (guid 53b5f397-eeb2-43b7-ba83-2241672bd6f2)
[ 2495.743370] ath10k_snoc 18800000.wifi: wcn3990 hw1.0 target 0x00000008 chip_id 0x00000000 sub 0000:0000
[ 2495.743393] ath10k_snoc 18800000.wifi: kconfig debug 1 debugfs 1 tracing 0 dfs 0 testmode 0
[ 2495.743423] ath10k_snoc 18800000.wifi: firmware ver 1.0.0.591 api 5 features wowlan,no-4addr-pad,allows-mesh-bcast,mgmt-tx-by-reference,non-bmi,single-chan-info-per-channel crc32 c256f0d6
[ 2495.743487] ath10k_snoc 18800000.wifi: board_file api 2 bmi_id N/A crc32 00000000
[ 2495.743508] ath10k_snoc 18800000.wifi: htt-ver 3.58 wmi-op 4 htt-op 3 cal file max-sta 32 raw 0 hwcrypto 1
[ 2495.820583] qcom-q6v5-mss 4080000.remoteproc: port failed halt
[ 2495.825913] remoteproc remoteproc0: stopped remote processor 4080000.remoteproc
[ 2495.935798] qcom-q6v5-mss 4080000.remoteproc: MBA booted without debug policy, loading mpss
[ 2497.687151] remoteproc remoteproc0: remote processor 4080000.remoteproc is now up
[ 2498.527530] ath10k_snoc 18800000.wifi: failed to set 2g txpower 40: -108
[ 2498.527571] ath10k_snoc 18800000.wifi: failed to setup tx power 20: -108
[ 2498.527605] ath10k_snoc 18800000.wifi: failed to set vdev wmm params on vdev 0: -108
[ 2498.527623] ath10k_snoc 18800000.wifi: failed to set vdev wmm params on vdev 0: -108
[ 2498.527638] ath10k_snoc 18800000.wifi: failed to set vdev wmm params on vdev 0: -108
[ 2498.527654] ath10k_snoc 18800000.wifi: failed to set vdev wmm params on vdev 0: -108
[ 2498.527674] ath10k_snoc 18800000.wifi: failed to stop WMI vdev 0: -108
[ 2498.527687] ath10k_snoc 18800000.wifi: failed to stop vdev 0: -108
[ 2498.527715] ath10k_snoc 18800000.wifi: failed to flush transmit queue (skip 1 ar-state 1): 1250
[ 2498.569965] ath10k_snoc 18800000.wifi: qmi not waiting for msa_ready indicator
[ 2500.554221] ieee80211 phy0: Hardware restart was requested
[ 2500.554346] ath10k_warn: 3 callbacks suppressed
[ 2500.554354] ath10k_snoc 18800000.wifi: failed to create WMI vdev 0: -108
[ 2500.758383] ath10k_snoc 18800000.wifi: device successfully recovered

After which Wi-Fi device is lost (appears as DOWN) and can't be reconnected without system reboot

ip a

xiaomi-whyred [~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 9a:a0:60:75:95:95 brd ff:ff:ff:ff:ff:ff
    inet 172.16.42.1/16 brd 172.16.255.255 scope global noprefixroute usb0
       valid_lft forever preferred_lft forever
    inet6 fe80::1947:fbd1:ff25:2a07/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 36:25:8c:55:f7:5c brd ff:ff:ff:ff:ff:ff permaddr 3a:a6:82:2b:02:e3

nmcli

xiaomi-whyred [~]# nmcli d
DEVICE         TYPE      STATE                   CONNECTION     
usb0           ethernet  connected               USB Networking 
lo             loopback  connected (externally)  lo             
wlan0          wifi      disconnected            --             
p2p-dev-wlan0  wifi-p2p  disconnected            --

Executing

# ip link set dev wlan0 up

does not help, without any output in dmesg.

Doing

# modprobe -r ath10k_snoc ath10k_core
# modprobe ath10k_snoc

also does not help. It probes back, but wlan0 stays off

Reproducible on all my xiaomi devices I think, and probably on all 630/636/660 devices

[minlexx]

Potential fix for the first 3 error messages is:

• wifi: ath10k: only wait for response to SET_KEY

[minlexx]

I suspect the structure layout used in QMI packet might be different in mainline ath10k driver and downstream qcacld driver

• https://elixir.bootlin.com/linux/v6.16.8/source/drivers/net/wireless/ath/ath10k/wmi.h#L5066
• https://github.com/LineageOS/android_kernel_xiaomi_sdm660/blob/lineage-20/drivers/staging/fw-api/fw/wmi_unified.h#L16988

The look similar in the beginning, but the end is different, side by side comparison:

                                                                      typedef struct {
struct wmi_vdev_install_key_cmd {                                                     
                                                                               A_UINT32 tlv_header; /** TLV tag and len; tag equals WMITLV_TAG_STRUC_wmi_vdev_install_key_cmd_fixed_param */
        __le32 vdev_id;                                                        A_UINT32 vdev_id;    /** unique id identifying the VDEV, generated by the caller */
        struct wmi_mac_addr peer_macaddr;                                      wmi_mac_addr peer_macaddr;   /** MAC address used for installing   */
        __le32 key_idx;                                                        A_UINT32 key_ix;             /** key index */
        __le32 key_flags;                                                      A_UINT32 key_flags;          /** key flags */
        __le32 key_cipher; /* %WMI_CIPHER_ */                                  A_UINT32 key_cipher;         /** key cipher, defined above */
        struct wmi_key_seq_counter key_rsc_counter;                            wmi_key_seq_counter key_rsc_counter;          /** key rsc counter */
        struct wmi_key_seq_counter key_global_rsc_counter;                     wmi_key_seq_counter key_global_rsc_counter;   /** global key rsc counter */
        struct wmi_key_seq_counter key_tsc_counter;                            wmi_key_seq_counter key_tsc_counter;          /** global key tsc counter */
        u8 wpi_key_rsc_counter[16];                                            A_UINT8 wpi_key_rsc_counter[16];       /** WAPI key rsc counter */
        u8 wpi_key_tsc_counter[16];                                            A_UINT8 wpi_key_tsc_counter[16];       /** WAPI key tsc counter */
        __le32 key_len;                                                        A_UINT32 key_len;                     /** key length */
        __le32 key_txmic_len;                                                  A_UINT32 key_txmic_len;               /** key tx mic length */
        __le32 key_rxmic_len;                                                  A_UINT32 key_rxmic_len;               /** key rx mic length */
                                                                               A_UINT32 is_group_key_ix_valid;    /** specifies if the group_key_ix is valid, filled by the sender */
                                                                               A_UINT32 group_key_ix;             /** Multi group key ID */
        /* contains key followed by tx mic followed by rx mic */                                                                       /* Following this struct are this TLV. */
        u8 key_data[];                                                         A_UINT8  key_data[]; <-- actual key data; contains key followed by tx mic followed by rx mic
} __packed;                                                           } wmi_vdev_install_key_cmd_fixed_param;

2 extra fields about group key might in theory cause crash in firmware, but why only on disconnect then... and still need to trace downstream driver to verify that this is the code path that is actually used 🤔

Interesting that the same struct described in ath11k/12k driver is close match: https://elixir.bootlin.com/linux/v6.16.8/source/drivers/net/wireless/ath/ath11k/wmi.h#L3667

[minlexx]

With some debug prints added in minlexx/android_kernel_xiaomi_lavender@5364cb3:

While turning Wi-Fi on, function is called 2 times, first time with cipher 4 (WMI_CIPHER_AES_CCM), second time with 2 (WMI_CIPHER_TKIP). Nothing seems to send this one (#define WMI_CIPHER_NONE 0x0 /* clear key */)

[  375.494574] ipa ipa2_uc_reg_rdyCB:1735 bad parm. inout=          (null) 
[  375.552558] IPC_RTR: process_new_server_msg: Server 00001003 create rejected, version = 0
[  375.708576] init: starting service 'wpa_supplicant'...
[  375.711484] init: Created socket '/dev/socket/wpa_wlan0', mode 660, user 1010, group 1010
[  375.716170] init: Control message: Processed ctl.interface_start for 'android.hardware.wifi.supplicant@1.0::ISupplicant/default' from pid: 588 (/system/bin/hwservicemanager)
[  375.828072] R0: [wpa_supplicant][7282378491] [20:16:06.438052]  wlan: [4165:I :HDD] 00000000: 7f 0a 00 00 0a 82 00 40 00 00 00 01 ff 03 02 00
[  375.828086] R0: [wpa_supplicant][7282378902] [20:16:06.438073]  wlan: [4165:I :HDD] 00000010: ff dd 0a 50 6f 9a 16 03 01 03 65 01 01 dd 08 8c
[  375.828093] R0: [wpa_supplicant][7282379046] [20:16:06.438080]  wlan: [4165:I :HDD] 00000020: fd f0 01 01 02 01 00
[  375.897726] R0: [wpa_supplicant][7283715774] [20:16:06.507702]  wlan: [4165:I :HDD] 00000000: 7f 0a 00 00 0a 82 00 40 00 00 00 01 ff 03 02 00
[  375.897745] R0: [wpa_supplicant][7283716324] [20:16:06.507730]  wlan: [4165:I :HDD] 00000010: ff dd 0a 50 6f 9a 16 03 01 03 65 01 01 dd 08 8c
[  375.897754] R0: [wpa_supplicant][7283716517] [20:16:06.507740]  wlan: [4165:I :HDD] 00000020: fd f0 01 01 02 01 00
[  375.912461] IPv6: ADDRCONF(NETDEV_UP): p2p0: link is not ready
[  375.932898] IPv6: ADDRCONF(NETDEV_UP): p2p0: link is not ready
[  375.936817] R0: [wpa_supplicant][7284466314] [20:16:06.546793]  wlan: [4165:I :HDD] 00000000: 7f 0a 00 00 0a 02 00 40 00 00 00 01 ff 03 02 00
[  375.936837] R0: [wpa_supplicant][7284466909] [20:16:06.546823]  wlan: [4165:I :HDD] 00000010: ff dd 0a 50 6f 9a 16 03 01 03 65 01 01 dd 08 8c
[  375.936844] R0: [wpa_supplicant][7284467058] [20:16:06.546831]  wlan: [4165:I :HDD] 00000020: fd f0 01 01 02 01 00
[  376.729276] R0: [wpa_supplicant][7299681435] [20:16:07.339247]  wlan: [4165:I :HDD] 00000000: 7f 0a 00 00 0a 82 00 40 00 00 00 01 ff 03 02 00
[  376.729299] R0: [wpa_supplicant][7299682164] [20:16:07.339284]  wlan: [4165:I :HDD] 00000010: ff dd 0a 50 6f 9a 16 03 01 03 65 01 01 dd 08 8c
[  376.729309] R0: [wpa_supplicant][7299682365] [20:16:07.339295]  wlan: [4165:I :HDD] 00000020: fd f0 01 01 02 01 00
[  376.830258] send_setup_install_key_cmd_tlv: FEATURE_WLAN_WAPI is enabled
[  376.830275] send_setup_install_key_cmd_tlv: sending install_key: vdev_id: 0, key_idx: 0, key_cipher: 0x4
[  376.840672] send_setup_install_key_cmd_tlv: FEATURE_WLAN_WAPI is enabled
[  376.840689] send_setup_install_key_cmd_tlv: sending install_key: vdev_id: 0, key_idx: 1, key_cipher: 0x2

While disconnecting from Wi-Fi nothing is called there, only this is in log

[  339.482317] init: Service 'wpa_supplicant' (pid 4088) exited with status 0 oneshot service took 62.182999 seconds in background
[  339.482368] init: Sending signal 9 to service 'wpa_supplicant' (pid 4088) process group...
[  339.485981] libprocessgroup: Successfully killed process cgroup uid 0 pid 4088 in 0ms
[  339.500733] type=1400 audit(1761412530.109:13): avc: denied { write } for comm="wifi@1.0-servic" name="tcp_limit_output_bytes" dev="proc" ino=32648 scontext=u:r:hal_wifi_default:s0 tcontext=u:object_r:proc_net:s0 tclass=file permissive=0
[  339.501380] type=1400 audit(1761412530.109:14): avc: denied { write } for comm="wifi@1.0-servic" name="tcp_use_userconfig" dev="proc" ino=40648 scontext=u:r:hal_wifi_default:s0 tcontext=u:object_r:proc_net:s0 tclass=file permissive=0
[  339.501465] type=1400 audit(1761412530.109:15): avc: denied { write } for comm="wifi@1.0-servic" name="tcp_adv_win_scale" dev="proc" ino=32649 scontext=u:r:hal_wifi_default:s0 tcontext=u:object_r:proc_net:s0 tclass=file permissive=0

[minlexx]

New commit - minlexx/android_kernel_xiaomi_lavender@0d80b88

Got even more info - https://paste.sr.ht/~minlexx/8de23c671a69adba4e5cb3b667d86775ef430720

[minlexx]

This might be very related https://lore.kernel.org/all/20250902143225.837487-1-loic.poulain@oss.qualcomm.com/

The commit is in 6.18-rc branches, not in 6.17 (yet), might be backported since 6.17.6 ?

Disconnecting from Wi-Fi on 6.18 still crashes remoteproc though I think

[minlexx]

Sometimes my Wi-FI AP is doing something, and I see in the dmesg on (e.g whyred):

[13451.946602] wlan0: AP 60:e3:27:30:a7:0c changed bandwidth in beacon, new used config is 2462.000 MHz, width 1 (2462.000/0 MHz)
[13452.198020] qcom-q6v5-mss 4080000.remoteproc: fatal error received: err_qdi.c:459:EF:wlan_process:1:cmnos_thread.c:3285:Asserted in wlan_vdev.c:_w
[13452.198080] remoteproc remoteproc0: crash detected in 4080000.remoteproc: type fatal error
[13452.198122] remoteproc remoteproc0: handling crash #3 in 4080000.remoteproc
[13452.198136] remoteproc remoteproc0: recovering 4080000.remoteproc
[13452.206319] failed to send lookup notification
[13452.206355] failed to send lookup notification
[13452.206372] failed to send lookup notification
...
(repeated thousand times)
...
[13452.208400] failed to send lookup notification
[13452.222048] ath10k_snoc 18800000.wifi: firmware crashed! (guid d96ab90b-445f-4c6f-ab49-ea1b40cbbe7a)
[13452.222119] ath10k_snoc 18800000.wifi: wcn3990 hw1.0 target 0x00000008 chip_id 0x00000000 sub 0000:0000
[13452.222133] ath10k_snoc 18800000.wifi: kconfig debug 1 debugfs 1 tracing 0 dfs 0 testmode 0
[13452.222148] ath10k_snoc 18800000.wifi: firmware ver 1.0.0.591 api 5 features wowlan,no-4addr-pad,allows-mesh-bcast,mgmt-tx-by-reference,non-bmi,single-chan-info-per-channel crc32 401dac16
[13452.222163] ath10k_snoc 18800000.wifi: board_file api 2 bmi_id N/A crc32 00000000
[13452.222176] ath10k_snoc 18800000.wifi: htt-ver 3.58 wmi-op 4 htt-op 3 cal file max-sta 32 raw 0 hwcrypto 1
[13452.307839] qcom-q6v5-mss 4080000.remoteproc: port failed halt
[13452.316533] remoteproc remoteproc0: stopped remote processor 4080000.remoteproc
[13452.423914] qcom-q6v5-mss 4080000.remoteproc: MBA booted without debug policy, loading mpss
[13454.034591] remoteproc remoteproc0: remote processor 4080000.remoteproc is now up
[13454.036687] failed to send lookup notification
[13454.036727] failed to send lookup notification
[13454.036745] failed to send lookup notification
...
[13454.809641] failed to send lookup notification
[13454.809673] failed to send lookup notification
[13454.809690] failed to send lookup notification
[13454.910926] ath10k_snoc 18800000.wifi: qmi not waiting for msa_ready indicator
[13454.921103] failed to send lookup notification
[13454.921142] failed to send lookup notification
[13454.921159] failed to send lookup notification
[13455.056022] ath10k_snoc 18800000.wifi: failed to start WMI vdev 0: -108
[13455.056057] ath10k_snoc 18800000.wifi: failed to restart vdev 0: -108
[13455.056178] ath10k_snoc 18800000.wifi: failed to update STA 60:e3:27:30:a7:0c peer phymode 5: -108
[13456.844816] failed to send lookup notification
[13456.844851] failed to send lookup notification
[13456.844865] failed to send lookup notification
[13456.941829] ieee80211 phy1: Hardware restart was requested
[13456.996248] failed to send lookup notification
[13456.996290] failed to send lookup notification
[13456.996303] failed to send lookup notification
[13457.019887] failed to send lookup notification
[13457.019928] failed to send lookup notification
[13457.019942] failed to send lookup notification
[13457.344770] ath10k_snoc 18800000.wifi: device successfully recovered
[13457.449043] qcom-q6v5-mss 4080000.remoteproc: fatal error received: err_qdi.c:459:EX:wlan_process:1:WLAN RT:1087:PC=b00c87e0
[13457.449084] remoteproc remoteproc0: crash detected in 4080000.remoteproc: type fatal error
[13457.449142] remoteproc remoteproc0: handling crash #4 in 4080000.remoteproc
[13457.449158] remoteproc remoteproc0: recovering 4080000.remoteproc
[13457.532137] failed to send lookup notification
[13457.532176] failed to send lookup notification
[13457.532194] failed to send lookup notification

And then it's in an endless loop of restarts and failures, device drains battery and dies,
so it dies by itself from time to time even without manually disconnecting.