Mitigating hardware/firmware/driver instability on the Raspberry Pi
Sometimes, through no fault of your own, something deep down will cause your Raspberry Pi to crash. Hard.
Luckily, we can do a few things to mitigate this.
The information in this post is, to the best of my knowledge, current as of November 2023. It should work on Raspberry Pi OS versions bullseye and bookworm, at least, but I make no promises.
Use a quality power supply with plenty of capacity
Enough has been written elsewhere about this, so I’ll keep it brief: an undersized power supply, or one that produces noisy power, can cause all manner of strange hardware issues.
If in doubt, pick up an officially recommended power supply for your Raspberry Pi.
The remainder of the topics in this post are not without risk, and you should only implement any given intervention if you fully understand the change and its implications. (See my Pi Reliability post on risk vs. benefits.)
Reboot automatically after a kernel panic
Your device might, rarely, experience a kernel panic. The Raspberry Pi does not, by default, reboot after this happens, which is an obvious problem if you can’t easily power-cycle it.
You can tell the system to reboot 10 seconds after a kernel panic occurs. Place the following content in /etc/sysctl.d/90-kernelpanic-reboot.conf:
kernel.panic = 10
Use the Raspberry Pi’s hardware watchdog
The Raspberry Pi has a seldom-advertised hardware watchdog. It’s disabled by default, but once you enable and configure it, it can restart your Pi even when the OS has become nonresponsive.
I’m still learning about this feature, but as I modernize each of the long-running Pis in my life I’m enabling it with the following steps:
- Add
dtparam=watchdog=onto the[all]section in/boot/config.txt - Reboot the Pi and verify that
ls /dev/watchdog*lists/dev/watchdogand/dev/watchdog0 - Install the watchdog software via
sudo apt install watchdog - Edit
/etc/watchdog.confand set the following settings:
watchdog-device = /dev/watchdog
watchdog-timeout = 15
max-load-1 = 24
min-memory = 1
- Start the watchdog service and check its status. Enable it if it’s running without error:
sudo systemctl start watchdog
sudo systemctl status watchdog
sudo systemctl enable watchdog
See Also:
Disable HDMI if you don’t need it
Recently, one of my Pi Zeros crashed hard overnight. I hadn’t configured its hardware watchdog yet, so it stayed offline until I power-cycled it.
Thanks to remote log collection, I was able to see at least some of the crash logs. The tl;dr seemed to be “something bad happened in the HDMI hardware, firmware, or driver.”
I don’t need HDMI on this Pi, so I figured I could just disable that hardware. The only way I found to do this is via the command sudo /opt/vc/bin/tvservice -o. There are two caveats:
- The change doesn’t persist after a reboot. You’ll need to run it on each boot via however you prefer to do that on Linux.
- Using the
tvservicecommand requires switching your Pi to the “original non-GL desktop graphics driver.” You can do this via the “Advanced” section ofraspi-config.
I’ve created a .deb package that, when installed, takes care of all that automatically. Find it at cdzombak/pi-disable-hdmi on GitHub.
See Also: Considerations for a long-running Raspberry Pi, Disable or remove unneeded services and software to help keep your Raspberry Pi online.
(The crash log below is mainly here so anyone searching the web for a similar issue can find this page. You can stop scrolling here if you want; this is the end of the post.)
The full Raspberry Pi HDMI crash logs I was able to recover
[120626.734822] ------------[ cut here ]------------
[120626.734862] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120626.735610] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120626.735979] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120626.736005] Hardware name: BCM2835
[120626.736019] Workqueue: events output_poll_execute [drm_kms_helper]
[120626.736899] unwind_backtrace from show_stack+0x18/0x1c
[120626.736949] show_stack from dump_stack_lvl+0x34/0x58
[120626.736994] dump_stack_lvl from __warn+0x7c/0xf8
[120626.737037] __warn from warn_slowpath_fmt+0x60/0xcc
[120626.737066] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120626.737791] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120626.739334] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120626.741028] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120626.741937] process_one_work from worker_thread+0x34/0x53c
[120626.741975] worker_thread from kthread+0xc0/0xe4
[120626.742010] kthread from ret_from_fork+0x14/0x30
[120626.742035] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120626.742054] 9fa0: 00000000 00000000 00000000 00000000
[120626.742071] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120626.742088] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120626.742144] ---[ end trace 0000000000000000 ]---
[120626.742185] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120616.494893] ------------[ cut here ]------------
[120616.494932] WARNING: CPU: 0 PID: 451 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120616.495675] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120616.496047] CPU: 0 PID: 451 Comm: kworker/0:4 Tainted: G WC 6.1.58+ #1693
[120616.496071] Hardware name: BCM2835
[120616.496086] Workqueue: events output_poll_execute [drm_kms_helper]
[120616.496967] unwind_backtrace from show_stack+0x18/0x1c
[120616.497016] show_stack from dump_stack_lvl+0x34/0x58
[120616.497061] dump_stack_lvl from __warn+0x7c/0xf8
[120616.497105] __warn from warn_slowpath_fmt+0x60/0xcc
[120616.497133] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120616.497861] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120616.499411] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120616.501107] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120616.502010] process_one_work from worker_thread+0x34/0x53c
[120616.502047] worker_thread from kthread+0xc0/0xe4
[120616.502082] kthread from ret_from_fork+0x14/0x30
[120616.502106] Exception stack(0xdd24dfb0 to 0xdd24dff8)
[120616.502124] dfa0: 00000000 00000000 00000000 00000000
[120616.502142] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120616.502157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120616.502216] ---[ end trace 0000000000000000 ]---
[120616.502260] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120606.254999] ------------[ cut here ]------------
[120606.255038] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120606.255778] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120606.256148] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120606.256176] Hardware name: BCM2835
[120606.256190] Workqueue: events output_poll_execute [drm_kms_helper]
[120606.257071] unwind_backtrace from show_stack+0x18/0x1c
[120606.257121] show_stack from dump_stack_lvl+0x34/0x58
[120606.257164] dump_stack_lvl from __warn+0x7c/0xf8
[120606.257205] __warn from warn_slowpath_fmt+0x60/0xcc
[120606.257236] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120606.257961] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120606.259509] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120606.261196] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120606.262095] process_one_work from worker_thread+0x34/0x53c
[120606.262131] worker_thread from kthread+0xc0/0xe4
[120606.262167] kthread from ret_from_fork+0x14/0x30
[120606.262191] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120606.262210] 9fa0: 00000000 00000000 00000000 00000000
[120606.262228] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120606.262244] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120606.262302] ---[ end trace 0000000000000000 ]---
[120606.262346] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120596.014162] ------------[ cut here ]------------
[120596.014201] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120596.014797] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120596.015159] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120596.015180] Hardware name: BCM2835
[120596.015193] Workqueue: events output_poll_execute [drm_kms_helper]
[120596.015886] unwind_backtrace from show_stack+0x18/0x1c
[120596.015931] show_stack from dump_stack_lvl+0x34/0x58
[120596.015973] dump_stack_lvl from __warn+0x7c/0xf8
[120596.016016] __warn from warn_slowpath_fmt+0x60/0xcc
[120596.016043] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120596.016551] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120596.017701] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120596.018613] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120596.019123] process_one_work from worker_thread+0x34/0x53c
[120596.019157] worker_thread from kthread+0xc0/0xe4
[120596.019192] kthread from ret_from_fork+0x14/0x30
[120596.019216] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120596.019234] 9fa0: 00000000 00000000 00000000 00000000
[120596.019251] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120596.019267] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120596.019317] ---[ end trace 0000000000000000 ]---
[120596.019354] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120585.773982] ------------[ cut here ]------------
[120585.774020] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120585.774625] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120585.774987] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120585.775008] Hardware name: BCM2835
[120585.775021] Workqueue: events output_poll_execute [drm_kms_helper]
[120585.775714] unwind_backtrace from show_stack+0x18/0x1c
[120585.775760] show_stack from dump_stack_lvl+0x34/0x58
[120585.775803] dump_stack_lvl from __warn+0x7c/0xf8
[120585.775845] __warn from warn_slowpath_fmt+0x60/0xcc
[120585.775872] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120585.776379] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120585.777522] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120585.778435] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120585.778940] process_one_work from worker_thread+0x34/0x53c
[120585.778975] worker_thread from kthread+0xc0/0xe4
[120585.779009] kthread from ret_from_fork+0x14/0x30
[120585.779033] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120585.779051] 9fa0: 00000000 00000000 00000000 00000000
[120585.779070] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120585.779085] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120585.779136] ---[ end trace 0000000000000000 ]---
[120585.779174] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120575.533776] ------------[ cut here ]------------
[120575.533813] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120575.534412] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120575.534774] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120575.534795] Hardware name: BCM2835
[120575.534807] Workqueue: events output_poll_execute [drm_kms_helper]
[120575.535501] unwind_backtrace from show_stack+0x18/0x1c
[120575.535546] show_stack from dump_stack_lvl+0x34/0x58
[120575.535588] dump_stack_lvl from __warn+0x7c/0xf8
[120575.535632] __warn from warn_slowpath_fmt+0x60/0xcc
[120575.535660] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120575.536168] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120575.537319] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120575.538230] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120575.538741] process_one_work from worker_thread+0x34/0x53c
[120575.538775] worker_thread from kthread+0xc0/0xe4
[120575.538811] kthread from ret_from_fork+0x14/0x30
[120575.538834] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120575.538852] 9fa0: 00000000 00000000 00000000 00000000
[120575.538870] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120575.538885] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120575.538935] ---[ end trace 0000000000000000 ]---
[120575.538972] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120565.293667] ------------[ cut here ]------------
[120565.293704] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120565.294276] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120565.294633] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120565.294653] Hardware name: BCM2835
[120565.294665] Workqueue: events output_poll_execute [drm_kms_helper]
[120565.295356] unwind_backtrace from show_stack+0x18/0x1c
[120565.295401] show_stack from dump_stack_lvl+0x34/0x58
[120565.295444] dump_stack_lvl from __warn+0x7c/0xf8
[120565.295487] __warn from warn_slowpath_fmt+0x60/0xcc
[120565.295515] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120565.296017] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120565.297166] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120565.298073] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120565.298585] process_one_work from worker_thread+0x34/0x53c
[120565.298619] worker_thread from kthread+0xc0/0xe4
[120565.298654] kthread from ret_from_fork+0x14/0x30
[120565.298677] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120565.298696] 9fa0: 00000000 00000000 00000000 00000000
[120565.298713] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120565.298728] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120565.298778] ---[ end trace 0000000000000000 ]---
[120565.298816] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120555.053606] ------------[ cut here ]------------
[120555.053643] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120555.054216] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120555.054576] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120555.054596] Hardware name: BCM2835
[120555.054609] Workqueue: events output_poll_execute [drm_kms_helper]
[120555.055299] unwind_backtrace from show_stack+0x18/0x1c
[120555.055343] show_stack from dump_stack_lvl+0x34/0x58
[120555.055387] dump_stack_lvl from __warn+0x7c/0xf8
[120555.055430] __warn from warn_slowpath_fmt+0x60/0xcc
[120555.055458] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120555.055960] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120555.057107] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120555.058012] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120555.058521] process_one_work from worker_thread+0x34/0x53c
[120555.058555] worker_thread from kthread+0xc0/0xe4
[120555.058590] kthread from ret_from_fork+0x14/0x30
[120555.058612] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120555.058630] 9fa0: 00000000 00000000 00000000 00000000
[120555.058647] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120555.058663] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120555.058718] ---[ end trace 0000000000000000 ]---
[120555.058757] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120544.813286] ------------[ cut here ]------------
[120544.813323] WARNING: CPU: 0 PID: 32385 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120544.813897] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120544.814256] CPU: 0 PID: 32385 Comm: kworker/0:1 Tainted: G WC 6.1.58+ #1693
[120544.814276] Hardware name: BCM2835
[120544.814289] Workqueue: events output_poll_execute [drm_kms_helper]
[120544.814974] unwind_backtrace from show_stack+0x18/0x1c
[120544.815017] show_stack from dump_stack_lvl+0x34/0x58
[120544.815061] dump_stack_lvl from __warn+0x7c/0xf8
[120544.815103] __warn from warn_slowpath_fmt+0x60/0xcc
[120544.815130] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120544.815635] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120544.816784] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120544.817694] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120544.818201] process_one_work from worker_thread+0x34/0x53c
[120544.818235] worker_thread from kthread+0xc0/0xe4
[120544.818269] kthread from ret_from_fork+0x14/0x30
[120544.818292] Exception stack(0xdcc99fb0 to 0xdcc99ff8)
[120544.818309] 9fa0: 00000000 00000000 00000000 00000000
[120544.818327] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120544.818342] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120544.818395] ---[ end trace 0000000000000000 ]---
[120544.818434] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120536.733051] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -110
[120534.653005] ------------[ cut here ]------------
[120534.653038] WARNING: CPU: 0 PID: 450 at drivers/gpu/drm/vc4/vc4_hdmi.c:490 vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120534.653613] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120534.653972] CPU: 0 PID: 450 Comm: kworker/0:3 Tainted: G WC 6.1.58+ #1693
[120534.653991] Hardware name: BCM2835
[120534.654004] Workqueue: events output_poll_execute [drm_kms_helper]
[120534.654689] unwind_backtrace from show_stack+0x18/0x1c
[120534.654735] show_stack from dump_stack_lvl+0x34/0x58
[120534.654779] dump_stack_lvl from __warn+0x7c/0xf8
[120534.654821] __warn from warn_slowpath_fmt+0x60/0xcc
[120534.654848] warn_slowpath_fmt from vc4_hdmi_connector_detect_ctx+0x300/0x358 [vc4]
[120534.655373] vc4_hdmi_connector_detect_ctx [vc4] from drm_helper_probe_detect_ctx+0x48/0x134 [drm_kms_helper]
[120534.656541] drm_helper_probe_detect_ctx [drm_kms_helper] from output_poll_execute+0x118/0x220 [drm_kms_helper]
[120534.657466] output_poll_execute [drm_kms_helper] from process_one_work+0x204/0x488
[120534.657972] process_one_work from worker_thread+0x34/0x53c
[120534.658005] worker_thread from kthread+0xc0/0xe4
[120534.658039] kthread from ret_from_fork+0x14/0x30
[120534.658064] Exception stack(0xdd251fb0 to 0xdd251ff8)
[120534.658081] 1fa0: 00000000 00000000 00000000 00000000
[120534.658099] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120534.658114] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[120534.658127] ---[ end trace 0000000000000000 ]---
[120534.658155] vc4_hdmi 20902000.hdmi: Runtime PM usage count underflow!
[120532.572980] hwmon hwmon1: Failed to get throttled (-110)
[120531.534227] ---[ end trace 0000000000000000 ]---
[120531.534277] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -110
[120531.532916] ------------[ cut here ]------------
[120531.532953] WARNING: CPU: 0 PID: 428 at drivers/firmware/raspberrypi.c:67 rpi_firmware_property_list+0x1d4/0x254
[120531.533014] Firmware transaction timeout
[120531.533020] Modules linked in: brcmfmac 8021q garp stp llc brcmutil sha256_generic libsha256 vc4 cfg80211 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper snd_soc_rpi_simple_soundcard raspberrypi_hwmon syscopyarea sysfillrect sysimgblt fb_sys_fops bcm2835_codec(C) rfkill bcm2835_v4l2(C) bcm2835_isp(C) snd_soc_bcm2835_i2s v4l2_mem2mem i2c_bcm2835 snd_soc_pcm5102a regmap_mmio bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc snd_soc_core videobuf2_memops videobuf2_v4l2 videobuf2_common videodev snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm raspberrypi_gpiomem vc_sm_cma(C) mc snd_timer snd uio_pdrv_genirq uio fixed drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[120531.533379] CPU: 0 PID: 428 Comm: kworker/0:0 Tainted: G C 6.1.58+ #1693
[120531.533398] Hardware name: BCM2835
[120531.533410] Workqueue: events dbs_work_handler
[120531.533456] unwind_backtrace from show_stack+0x18/0x1c
[120531.533492] show_stack from dump_stack_lvl+0x34/0x58
[120531.533534] dump_stack_lvl from __warn+0x7c/0xf8
[120531.533577] __warn from warn_slowpath_fmt+0x94/0xcc
[120531.533605] warn_slowpath_fmt from rpi_firmware_property_list+0x1d4/0x254
[120531.533637] rpi_firmware_property_list from rpi_firmware_property+0x68/0x110
[120531.533665] rpi_firmware_property from raspberrypi_clock_property+0x48/0x74
[120531.533709] raspberrypi_clock_property from raspberrypi_fw_set_rate+0x3c/0xb0
[120531.533739] raspberrypi_fw_set_rate from clk_change_rate+0xe8/0x374
[120531.533785] clk_change_rate from clk_core_set_rate_nolock+0x168/0x194
[120531.533819] clk_core_set_rate_nolock from clk_set_rate+0x38/0x90
[120531.533851] clk_set_rate from _opp_config_clk_single+0x2c/0x8c
[120531.533893] _opp_config_clk_single from _set_opp+0x130/0x32c
[120531.533926] _set_opp from dev_pm_opp_set_rate+0x5c/0x158
[120531.533956] dev_pm_opp_set_rate from __cpufreq_driver_target+0x174/0x234
[120531.533987] __cpufreq_driver_target from od_dbs_update+0xb8/0x174
[120531.534013] od_dbs_update from dbs_work_handler+0x30/0x5c
[120531.534040] dbs_work_handler from process_one_work+0x204/0x488
[120531.534082] process_one_work from worker_thread+0x34/0x53c
[120531.534111] worker_thread from kthread+0xc0/0xe4
[120531.534144] kthread from ret_from_fork+0x14/0x30
[120531.534167] Exception stack(0xdd219fb0 to 0xdd219ff8)
[120531.534182] 9fa0: 00000000 00000000 00000000 00000000
[120531.534199] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[120531.534214] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000