Random Klipper USB disconnects

In building the MeowFire print farm we have come up against an AMD Ryzen 7 4800U micro PC that would experience random Klipper USB disconnects. After days of searching and testing we stumbled upon the issue causing these random disconnects.

First a little background. We run Kilpper and Moonraker on dedicated micro PC’s that are attached to 4 printers. The Intel based Dell 3070 have been working very well and we wanted to test how well a newer AMD based system would work.

We installed Klipper, Moonraker and Mainsail on Ubuntu on the AMD system and connected via serial over USB from the printer to the AMD Ryzen system. The first small test print finished without any issues. We started a larger test print, verified that the first layer was good and walked away from the printer. About 10 minuets into the print we were alerted that the printer has disconnected from the AMD host PC.

We power cycled the printer and it reconnected. We then restarted the test print, watched the dmesg log and print, after about 30 minuets we walked away to let the test print finish. After about 30 minuets the printer disconnected again.

As it was the end of the day we power cycled the printer, it reconnected it to the host PC and left for the night. The printer was not printing, only powered on and showing as connected in Mainsail. The next morning we found that the printer had disconnected again.

Keep in mind that the printer has a known good MCU and we has just successfully tested this printer on the Dell 3070.

Troubleshooting Klipper random USB disconnects

We were able to isolate the exact times of the disconnects by tailing the dmesg ring buffer with: dmesg -wT | grep -i usb

[Sat Apr 29 14:12:33 2023] usb 1-8: USB disconnect, device number 30
[Sun Apr 30 06:47:38 2023] usb 1-8: new full-speed USB device number 31 using xhci_hcd
[Sun Apr 30 06:47:39 2023] usb 1-8: New USB device found, idVendor=0483, idProduct=5740, bcdDevice= 0.00
[Sun Apr 30 06:47:39 2023] usb 1-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Sun Apr 30 06:47:39 2023] usb 1-8: Product: MARLIN_STM32H743VX CDC in FS Mode
[Sun Apr 30 06:47:39 2023] usb 1-8: Manufacturer: STMicroelectronics
[Sun Apr 30 06:47:39 2023] usb 1-8: SerialNumber: 397B366F3030
[Sun Apr 30 06:47:39 2023] cdc_acm 1-8:1.0: ttyACM1: USB ACM device

To make sure it was not the host PC, we connected a laptop running Ubuntu on to the printer via USB and the test print finished without disconnecting. We left the laptop connected over night with the printer on and it was still connected the next morning. For this this eliminated an issue with the printers MCU, USB cable and Klipper configuration.

With our focus on the AMD Ryzen micro PC we started out by looking at what USB chipset was used by running: lspci | grep USB

05:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01)
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
06:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1

This micro PC has external USB2 and USB3 ports that are controlled by the VIA Technologies, Inc. VL805/806 xHCI controller. The AMD Renoir/Cezanne appear to be used for internal devices. The Dell 3070 and laptop had Intel based USP controllers.

We then spent another day Googling and trying disabling USB sleep and IOMMU via kernel command line. We even installed a real time kernel, only to still have the printer disconnected the next morning when we came in.

After another 1/2 day of tweaking and testing we were no closer to figuring this out.

Turning off the light moment for USB disconnects

Frustrated we decided to get out of the lab and get some lunch. Normally we just leave and let the automatic light turn off that is above the workbench. For some reason, I decided to turn off the light manually and noticed at that moment the terminal we had tailing dmesg showed the printer disconnecting.

The printer, light and AMD Ryzen micro PC were all on the same circuit and not going through a UPS. This is the same circuit that we tested with the Dell 3070 and laptop, which had no disconnect issues.

We suspect the AMD micro PC’s power supply ( which powers the system via a USB C port ) is not well isolated, as when the overhead light would turn off ( or on via motion sensor ) this would cause all USB devices to disconnect. We never noticed the USB keyboard of mouse because they would quickly reconnect. The Dell 3070 uses a 19V DC power supply and the laptop is a Framework and is powered via USB C.

And that explains why these seemingly random Klipper USB disconnects were happening. The Ryzen micro PC would drop all USB devices connected to the VIA VL805/806 USB controller, when the florescent light, which was on the same circuit, was turned on or off. Our solution was not to use that brand of micro PC as a host PC for our printers.