[BUG] SKR Mini stop mid print

[adams79]

Bug Description

After replacing my mp select mini board with an e3 mini skr board sometimes during a print the board completely freeze with heater on and fan on. Tried both via SD and octoprint.
Replaced the board, same problem. Additionally when I run my Z-axis motor in StealthChop I've a sort of "constant" step skipping. Tried raising current but no changes. How can I track the source of problem? I'm going to change PSU as well, but I'm running out of ideas. The driver monitoring is disabled. After replacing the board I was able to print for about 5/6 hours without problems but then the problem resurfaced. Installed latest build available

Thank you

My Configurations

Required: Please include a ZIP file containing your Configuration.h and Configuration_adv.h files.

Steps to Reproduce

Start a printing from sd tr octoprint

Expected behavior:

The print should complete

Actual behavior: [What actually happens]

The print stops randomly
Archivio.zip

[swilkens]

You have:

#define DISABLE_X true
#define DISABLE_Y true
#define DISABLE_Z true

This disables the stepper motors when it's not in active use, this can cause some accuracy issues (step skipping?).

You are also not using the most recent bugfix, try the most recent - including the latest configuration files. Then do a M502 followed my a M500 after flashing.

I also notice you are using this board for a Malyan M200 - I assume you found these motor currents for the Malyan M200 in the original firmware?

[adams79]

Hi swilkens,
ok I'm going to test with latest bugfix and config. yes the currents are from the original Maylan board.
Also I will try to not disable the motors, however the step skips on Z-axis is not related to this. Once I activate stealth chop on this axis seems that there is a "fixed" amount of skips since object are always lower of about 20%. The weird thing is that by moving the z-axis with direct command the movement is right, but during print seems to skip about 20% of steps... I'm thinking of switching the z-axis motor

[adams79]

Ok, tried with latest build and config, stops after about 30 minute... trying to enable SD logging to check if some error is logged in the console...

[kain0m]

Check this one out:
#15337
It is related to SKR 1.3 (not mini), but the issue is the same. Random freeze of the print, heaters on.

My solution was a different SD card.
Also, try reformatting the SD card(s) with FAT32.
I have also read that the supplied Bigtree tech Micro SD card is poor quality in some cases.

[adams79]

This seems to not be my problem since it happens also by printing from octoprint without sd inserted

[swilkens]

Since the Malyan M200 by default does not have a fan in the electronics case, could it be that the drivers are overheating? In that case I would expect one driver to stop, rather than all of them.

Try to enable TMC driver monitoring and see if the driver reports an overtemp when it stalls. Do you still have serial communication when this happens or does the board fully lock up?

Which environment are you using the build the firmware? 256K? USB composite?

[cooldudie2]

i can confirm the exact same errors, as well as a lot of MP SELECT mini users using the Skr mini e3 1.2 mod i uploaded to thingiverse. (cooldudie2) all the users i have been in contact with have experienced the same freeze up issue using the firmware i have written, as well as their own compiled firmware based on bugfix 2.0.x branch and newest marlin 2.0.5.3 branch.

After replacing my mp select mini board with an e3 mini skr board sometimes during a print the board completely freeze with heater on and fan on.

Tried both via SD and octoprint.
Replaced the board, same problem i have tried different suppliers and same problem occurs.

someitmes it freezes sometimes it finishes prints just fine.

note the jumper cap is removed from the board as its set in firmware and not required when using uart mode some people have not taken the cap off by default but confirm the problem still persists.

i have tried a powerful and more reliable 12v 750watts ATX monitored PSU and noticed no change in voltage drop when print fails, power fluctuation of max 0.2 volts either side of 12volts when in use.

• i can confirm the PSU is not at fault. problem still persists.

*Driver monitoring doesn't report any overheat prewarn triggers.

*linear advance and hybrid theshold are turned off to rule those out completely.

*i have also tried setting the extruder to spreadcycle mode to rule out the problem where stealthchop triggers a lock up when moving to fast in one direction and then the other as suggested by others on the forum.

*256k version and 512k version show the same fault regardless of environment used.

*board fully locks up and doesnt respond to usb commands via pronterface or from the malyan m200 screen attached.

i will report back if the problem re occurs after some more prints.

Steps to Reproduce
Start a printing from sd tr octoprint

Expected behavior:

The print should complete

Actual behavior: [What actually happens]

The print stops randomly

[cooldudie2]

i have also just considered that the screen is drawing 3.3v logic power from the swd header on the board for power, i wondering if this could be causing the issue of lockup feature. whats the swd header for and if you draw power from it could it effect the other electronics

[adams79]

I’ve currently swapped the z-axis motor with a Nema 17 however I think that this will not solve the problem. Also I have a spare skr 1.4 turbo (with tmc2209) that I bought for another printer, I can make a try with this board too

[adams79]

Since the Malyan M200 by default does not have a fan in the electronics case, could it be that the drivers are overheating? In that case I would expect one driver to stop, rather than all of them.

Try to enable TMC driver monitoring and see if the driver reports an overtemp when it stalls. Do you still have serial communication when this happens or does the board fully lock up?

Which environment are you using the build the firmware? 256K? USB composite?

I have a v2 that has a case fan (that I’ve also replaced with a noctua)... tried to print without side panel and the heat sink on drivers are cool

[minosg]

From a similar thread investigating the same issue on another project.

Klipper3d/klipper#196

This correspondence with TMC is being attached which explains the lock-up issue

I think, the reason for disabling of the TMC2208 driver could be

a hard stop of the motor in stealthChop (Step frequency goes from a higher value, e.g. > 0.5 RPS to 0)

an abrupt change of motor velocity (Step frequency goes from a higher value to a low value within a single step).

When using stealthChop, please always make sure, that you use velocity ramping. A hard stop will cut away motor back-EMF at once. As stealthChop is a voltage based chopper, it cannot respond to this at once, like spreadCycle. The result is an overcurrent, and the motor driver goes to overcurrent switch off, until it becomes disabled / enabled again.

To resolve the problem, please use at least a tiny velocity ramping, when hard stopping the motor, e.g. within a few / a few ten microsteps.

To my understanding this issue could be addressed by:

• velocity ramping. ( Not certain how Linear advance will respect that )
• Setting the hybrid threshold to a value, that when the extrude changes direction suddenly it falls to spread cycle.
• Playing with the MINIMUM_STEPPER_POST_DIR_DELAY/MINIMUM_STEPPER_PRE_DIR_DELAY and overriding the driver default of 20nS

Does marlin have a velocity ramping configuration parameter?

[adams79]

Confirmed that the problem still persists after swapping the Z-axis motor with a standard Nema 17. Since other SKR mini users are not experiencing this it should be related to same specific component of this printer (this is the reason for which I've swapped the motor). I'm thinking to make a test with the display completely disconnected (I use Octopi) since @cooldudie2 is not sure that is ok to get current from SWD. Regarding @minosg reported issue in my understanding if this happens for a single driver it should be stopped only this and not the whole board.. is this correct?

[adams79]

@minosg additionally note that at least in my case steppers are on (you can check this by trying to move the head or the bed with hand) so drivers should not be stopped. Launched a print without display will report results

[adams79]

cannot be yet sure, but have printed for about 6 hours with display disconnected and no issue so far. will keep it disconnected, if someone can make the same try it will be useful to have multiple test cases... (you will need to print from pc or octoprint)

[minosg]

@adams79 The Marlin lockup of the steppers is configured and you can set it in the conf files( see DEFAULT_STEPPER_DEACTIVE_TIME and DISABLE_X/Y)

The TMC driver lockup is a totally different issue, which is triggering a failsafe component on the motor driver IC when it monitors a current spike outside of the expected thresholds. As an oversimplified explanation, when you invert the direction the momentum of the motor will generate an inverse current which depending on phase can add to the driving one and push it over the threshold. When that happens the driver chip will lock up and you need to power cycle it to unlock it. TMC engineers in the thread I posted, recommend either:

• Gradually phase out direction changes
• Add a delay between direction changes for the inverse pulse to dissipate

Following up my suggestion, I did some research and experiments.

In your conf_adv you have set up the motor currents as

  #if AXIS_DRIVER_TYPE_X(TMC26X)
    #define X_MAX_CURRENT     1000  // (mA)
    #define X_SENSE_RESISTOR    91  // (mOhms)
    #define X_MICROSTEPS        16  // Number of microsteps
  #endif

This could be an issue for a couple of reasons

• MP Select mini comes with the JK42HS34-1334 stepper motors, which are rated at 1.33A peak current. This converted to RMS which the stepper driver requires it would be 0.940A.
• The X_SENSE_RESISTOR seems a bit too large, compared to what is set-up for the ender which is what this board was created as.

Mind that most Enders runs on 2A rated motors and at 24V which make the current calculations different. This would explain why most mini users see that issue but Ender users do not.

I have set the X_MAX_CURRENT to a safe threshold of 500mA and the MINIMUM_STEPPER_POST_DIR_DELAY/MINIMUM_STEPPER_PRE_DIR_DELAY to a large arbitrary value of 1500 and it has yet to lock-up yet. I would recommend you do the same and try to see if that addresses the issue

[adams79]

hi @minosg you should check config for tmc2209 and current is set to 450mah (380 for the Xaxis motor). Regarding the MINIMUM_STEPPER_POST_DIR_DELAY this appears only related to the the driver itself so this problem should happens for all users of this board... However please let us know is you have no other freeze. Unfortunately I've to confirm that with display connected and powered with a sperate line I still experienced a freeze. The only case with no freeze for almost 10 ours has been with display completely disconnected...

[minosg]

@adams79 apologies I was looking at the wrong section of the attached config.

As for the the problem appearing to every user, that should not be the case. BTT SKR e3 board has the r26 and r27 resistors set to .11Ω which according to chapter 8 of the TMC2209 datasheet is for sensing a motor of 1.7A and higher. While this setting is usefull for sensing the end point break, it is indicative that Select Mini is using a considerably lower power motor than the board was tested on. If that is operating at its threshold a lot of bizarre things can happen.

My testing is showings that this delay does matter, and gets to make the issue less frequent. You can also isolate the extruder motor as the suspect, because if you do not use filament and run it as a dry run it should never freeze.

The other good question is why is the watchdog not kicking in and resetting the board?

[adams79]

I’m just trying to swap the board with an skr 1.4 turbo that I got for another printer, it has 4 tmc 2209... I’m curious to check if the same problem will arise (however it will require some work to mount the board in the little case)... Il giorno 5 giu 2020, alle ore 19:00, minosg <[email protected]> ha scritto:  @adams79 <https://github.com/adams79> apologies I was looking at the wrong section of the attached config. As for the the problem appearing to every user, that should not be the case. BTT SKR e3 board has the r26 and r27 resistors set to .11Ω which according to chapter 8 of the TMC2209 datasheet is for sensing a motor of 1.7A and higher. While this setting is usefull for sensing the end point break, it is indicative that Select Mini is using a considerably lower power motor than the board was tested on. If that is operating at its threshold a lot of bizarre things can happen. My testing is showings that this delay does matter, and gets to make the issue less frequent. You can also isolate the extruder motor as the suspect, because if you do not use filament and run it as a dry run it should never freeze. The other good question is why is the watchdog not kicking in and resetting the board? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18117 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOCVK4G7RH6CR5KYIZRBDLRVEQD5ANCNFSM4NKBOL3A> .

[minosg]

To answer my own question and to offer a workaround on the freezing part, the reason is that the watchdg is not enabled on this cpu HAL

#18226

If you uncomment this line the board will now reset when the crash occurs which is more safe than crashing with the heaters on.

As to why that stop happens, I am still investigating it since it has been reported for months, and there is no apparent solution.

What I have confirmed so far.

• It is not directly related with stealthchop. It gets worse by it but it can trigger on builds with Spread cycle on motors only.
• It is somehow related with the extruder motor. Running prints without filament always complete.
• It triggers in both prints from SD and Host enabled
• It can occur in both 512 and 256KB builds.
• Setting the extuder jerk to 2.0 or lower can increase the frequency.

When it happens the board is lost. You have lost UART debugging, sd logging etc

This means that most likely we are seeing an interrupt race condition, or disabled interrupts. But on the same time the heaters are running in set temperature, so the pwm loop for the heater is working, which makes it less likely to be a hardfault.

[adams79]

HI @minosg,
thank you for the update. In the meantime I've managed to mount the skr 1.4 turbo board and start printing, until now (about 4 hours) I've had no freeze. I was not able to connect the Maylan lcd onto this board (tried to connect onto the tft serial but unfortunately on this serial I see the standard console output and not the encoded one that should come from the maylan ext ui... can you help?)

[adams79]

@minosg I can confirm that is related to the extruder, I see that printing at high speed with high layer height and fast retract the freeze occurs very often.

[minosg]

HI @minosg,
thank you for the update. In the meantime I've managed to mount the skr 1.4 turbo board and start printing, until now (about 4 hours) I've had no freeze. I was not able to connect the Maylan lcd onto this board (tried to connect onto the tft serial but unfortunately on this serial I see the standard console output and not the encoded one that should come from the maylan ext ui... can you help?)

That is a separate issue but I would be looking at the serial declaration. The Malyanlcd serial is set in
extui_malyan_lcd.cpp

You need to make sure the Serial is available and properly set in the appropriate pins, ie for BTT Skr Mini you can find it on pins_BTT_SKR_MINI_E3.h

Last but not least you need to make sure the host serial is properly selected in config.h

[minosg]

Also does the SKR turbo have hardware or software serial? I have come to notice that SKR mini 1.2 is using software serial to talk to the drivers, and I am not quite confident that the pulse timings are withing acceptable threshold.

But could a broken serial channel on the TMC driver commns break the firmware? I cannot see how.

[adams79]

@minosg after more than 12 hours printing on the skr 1.4 board I experienced no freeze with the mini. Note that I use the same drivers (TMC 2209) on UART, so seems to me that the problem is not directly related to the drivers or the motors... However I've not connected the display. Have you tried using the printer with the display detached? In my case I was able to print without freeze on the mini this way...

[taragor]

I've just run into the issue exept the Z skipping.
I'm using the SKR mini V2.0 (STM32F1, TMC2209) on an ender 3. However I have printed without any issues for probably 30 hours so far on 50mm/s.
Yet the printer freezes randomly with heaters on as soon as i try printing at 80mm/s (Had one freeze after 1h and one after 5h, same gcode).
Anyway I'm not sure that the issue comes from sudden direction changes since both prints failed at curves/circles in walls (cura wall speed: 40mm/s)

[minosg]

@taragor according to my experiments, it is more related to the acceleration than the maximum printing speed. You can trigger it quite consistently if you use any high retraction model ( ie Flexi Articulated Gecko Keychain ) and set:

• Maximum E-Jerk 2.0 or less
  -Maximum Feed-rate E 100mm/s or more
• Maximum Accel E to 6000+
• Maximum Accell E when extruding to 1200+

As long as you have filament in the printer it will get frozen in the next 5 prints even when printing at 40 or 35mm/s speed.

Other things we have confirmed so far is that it still happens when you:

• Try older release tags of marlin.
• Reduce the TMC serial speed to 115200. Mini e3 is using software serial which is not ideal.
• Enable DISABLE_MULTI_STEPPING
• Try with a higher watt power supply ( 220W )
• Work with a higher voltage threshold 24V instead of 12V ( that is what you just confirmed )
• Disable steathchop and run the drivers in Spreadcycle mode ( less frequent but it happens )
• Disable Linear Advance, Hybrid threshold altogether
• Set minimum square pulse to 2, and 4
• Enable Square Wave Stepping
• Enabled velocity ramping by increasing the ihold_delay using tmc_adv()

Basically every workaround on tickets relating to the same issue proposed in the last few months has been tested, and even though it makes it better, it will not fix this issue.

The question now is weather the issue we are facing is:

• A hard to reproduce long standing, creeping marlin issue
• A board design issue, if it only happens on SKR mini. BTT have omitted the recommended coil ripple protection capacitors in the driver outputs ( 470pf/100V). But as I see from this bug report 17161 something similar is happening in LPC1716X
• A physical limitation. If you try to do this the CPU cannot keep up for the fp math, but then it should either caught up in sanity checks or graciously crash rather than halt with heaters on.

I am pretty much at a loss.

[sawaguna]

I have the same problem with my SKR 1.3
It's really random.
It can works for days without issues, and then it can freeze during a print, with heaters on.

No idea where the problem is.
I had it happening with both SD and USB (Repetier-Server)

First time I had this problem, it was a file with 30% gyroid infill.
Each time I was attempting to print it, it would freeze the printer (being via SD or USB).
I just changed the infill type to rectangular, and it worked.

So I thought it was maybe a cache issue.
But I really don't know, and it's probably not related

[taragor]

@minosg
Wait, do you mean it will litteraly crash for the next 5 print, regardless of the file?

As long as you have filament in the printer it will get frozen in the next 5 prints even when printing at 40 or 35mm/s speed.

If so I think I witnessed that. After my first failed print I had some issues restarting it:
I was printing live from octoprint and octoprint lost connection during G29:
It heats the bed and then does G28 followed by G29 (as specified in my start Gcode), however it stops sending temperature status and Octoprint will just time out. G29 however completes normally
I didn't think much of that since I occasionally had that issue before so I'm not even sure if it's connected to this.
I had this a few times before but this time it happened a few times in a row. I can't recall how often exactly but since i tried powercycling both the printer and the raspberry I guess it was 3-5 times.

[minosg]

@minosg
Wait, do you mean it will litteraly crash for the next 5 print, regardless of the file?

As long as you have filament in the printer it will get frozen in the next 5 prints even when printing at 40 or 35mm/s speed.

I believe you need a high retraction model. Articulated models are perfect for that. The reason that his issue has been creeping for over a year is that is quite random and hard to reproduce.

What I meant above is that if you use that model, and those settings in prusaslicer it will make it more apparent.

My gecko gcode, will crash the firmware in maximum 5 prints, regardless of the setting combinations I described above.

But if you run it without filament in the extruder, it will never crash.

[taragor]

Do you know if it is possible to debug Marlin using JTAG? If so it might give a clue whether it's the m3 just halting or marlin waiting for some kind of event/data.

[minosg]

Just an update on this issue, having been testing for it in the last week.

When it happens, the Bed and Hotend PWM is working as intended, which means both the interrupts, and the PWM are functioning. Also the DIAG pin on each of the drivers appears to be low, and the Index pin pulsing as expected.

Considering that the issue becomes more visible with setting high acceleration settings in M204 and under the assumption that it could not be the consumer's fault ( tcm_stepper) I started looking into the producer (planner.cpp).

If you disable the Malyan_LCD and print using SD card commands (M21 & M24) I have yet to have a freeze in two consecutive days of printing the test file. This could also be the case for #18315

BTT have pushed a change like that on their display driver which could indicate a possible workaround
bigtreetech/BIGTREETECH-TouchScreenFirmware@5777f41

Could the people affected please share:

• What display are you using
• What happens if you run it in headless mode by disabling that display

[adams79]

@minosg In my case I was using the Malyan display when experienced the issue.. Unfortunately I've now swapped both board and display so I don't know If using the e3 mini with the the new display would solve the problem. It's possible however that when using high retraction/acceleration would cause to send multiple updates to the display and this cause issues (race condition on the serial port?)

[minosg]

You don't have the original board to try?

[adams79]

No unfortunately I've returned it.

[taragor]

@minosg I've tested with the display disabled in Marlin, having the TFT35 in BTT mode, therefore just running as an serial GCODE terminal, like octoprint or pronterface.
I've just had another freeze with the following setup:
-TFT35 connected using the TFT connector (Serial 2 in Marlin)
-Raspberry connected via USB (Serial -1) and was printing live of Octoprint
-Display disabled in Marlin, ribbon cable to EXP3 on the screen disconnected
-Object was sliced in cura, speed 80mm/s, despite that pretty much default settings.

[minosg]

@taragor pleaee disable displays on firmware, not just disconnect them and print using sd commands. You can do that with pronterface. The issue i am seeing are serial interrupts firing into each other. Printing though octoprint or even enabling host keep alive will trigger it.

[taragor]

@minosg I have disabled #define CR10_STOCKDISPLAY and some other things according to sanity check (i.e. M73). I'm currently printing with octoprint disconnected, and only the TFT35 in BTT mode (for my understanding that should be the same as using pronterface, please correct me if I'm wrong here). I'm currently printing live of the TFT, using a USB drive in the TFTs USB Slot, but will try printing of the printers onboard SD once that finishes or fails.

EDIT: just to clarify, you suggest starting the print through pronterface (or octoprints terminal for that matter) using M24/27 and then disconnecting from the printer?

[minosg]

I think that printing from sd is m21, m24 commands but pronterface has an sd button to automate it. And yes that will make sure the only interrupts firing are the tsteppet ones. Btt mode is still a serial

[boelle]

Anyone that has the board mentioned:

Please test the bugfix-2.0.x branch to see where it stands.

[minosg]

@boelle I have the BTT SKR mini e3 which is the one mentioned. I also have a cheetah board, using the same STM32F1 chipset, and I have ordered another one. All of them are using this chipset and TCM uart's in serial mode and exhibit the same issue

I can confirm that the issue is still there on latest bugfix and the latest TMC_Stepper library ( v0.7 )

[adams79]

I confirm that I'm seeing that on SKR E3 Mini with latest bug fix but not on SKR 1.4 (LPC chipset)

[minosg]

@adams79 are you using malyan_lcd on SKR1.4? What mode are you using the steppers on?

[adams79]

@minosg no I'm not able to use the Malyan LCD, I'm using a TFT24 connected BOTH in serial and LCD12865 emulation mode. I'm using all steppers in stealthchop, the drivers are tmc2209. Print dozens of hours with no problem with this setup

[drakaz]

Hello,

Same issue with a SKR mini v2.0, board completely freeze randomly with heater on and fan on, printing from Octoprint.

[lexter12]

Same issue with SKR Mini V2, but can't print more than 5 minutes without freeze - pretty consistent, never managed to print over 10 minutes without hitting the freeze.

[Depz42]

SKR Mini V2, downloaded a fresh copy of Marlin bug fix on 6/29 and the freezing issue persists with no rhyme or reason. I had a failed 20 minute print in the first few layers, successful 8 hour print, failed 30 minute print about halfway through. I've been able to save one of the prints because it tells you which layer it was on, but still a pretty serious bug.

[minosg]

I have posted a possible workaround, which involves patching the libmaple library.

#18358 (comment)

Can you please try it and let us know in #18358 if it has resolved your issues?

[Depz42]

I uploaded the change and testing a print now. I tried printing the same thing 3 times and it failed twice, so it's not 100% reproducible. I'll monitor the next few prints and keep you updated.

[Depz42]

Just ran a 1:45 hour print with no issues.

[minosg]

Sd card print or host (octoprint?). Anyhow if cause was what I described in case 5 you shouldn't get any more prints.

Remember you may need to reapply the patch if you upgrade the platfromio

[Depz42]

Octoprint on a raspberry pi 3B+. Thanks for the quick response and your time fixing this.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label / comment or this will be closed in 5 days.

[droneando]

I have posted a possible workaround, which involves patching the libmaple library.

#18358 (comment)

Can you please try it and let us know in #18358 if it has resolved your issues?

I have posted a possible workaround, which involves patching the libmaple library.

#18358 (comment)

Can you please try it and let us know in #18358 if it has resolved your issues?

Hi:
This workaround works for me! And also locks issues related with M500 use in terminal are fixed!
Thank you very much!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.