AVR Freaks

Hi @spachner, @limpkin, @0Pal and @ekalyvio,

I apologize for not responding earlier. I did not received notifications about posts to this thread :(

The only "work around" I could find was implementing the watch dog early warning interrupt. It wakes up the SAMD21 every 8 seconds which then "kicks the dog". If the SAMD21 does not wake up, the watch dog will "bite" and reset it 8 seconds later. It works, but restoring the firmware state to the same condition before reset and keeping accurate RTC time is problematic (it loses up to a second during reset).

I used a test farm of 4 units and by trail and error got a version of demonstration firmware (using the watch dog early warning interrupt) that was able to regularly produce one failure within 4 days (usually within 24 hours). It seems to be a hardware related race condition as any changes in the code makes the problem less likely (you have to wait much longer to see a failure). I sent boards and demo firmware to (then) Atmel, but they were unable to discover the root cause as they were approaching the problem only from a pure firmware perspective :(

Recently I have been tasked to look at this problem again, because some units have been resetting up to 12 times a day! I found the "ST STM32F40x and STM32F41x Errata sheet", section 2.1.3 and 2.1.5 which has provided me with a new idea to test. I am running a new test on a test farm of 7 units and should know by the end of the weekend if I am on to something.

I will keep you posted. If you would like to find out the early details, you may PM me. Please let me know if you are still experiencing the problem and if you have found something.

Regards,

Pieter

https://piconomix.com/contact/

Hi everyone,

I hope it's not premature, but I think I found a solution. Please test and post feedback on this thread. Here it is:

RAMFUNC void main_standby_sleep(void);

void main_standby_sleep(void)
{
    // Disable interrupts
    __disable_irq();
    __DMB();
    // Select STANDBY sleep mode
    SCB->SCR |=  SCB_SCR_SLEEPDEEP_Msk;
    // Wait for Interrupt
    __DSB();
    __WFI();
    // Enable interrupts
    __DMB();
    __enable_irq();
}

I replaced the ASF version of the code with my own version and placed the function in SRAM (“RAMFUNC” attribute). If you look in the ASF version, you will see:

static inline enum status_code system_set_sleepmode(
    const enum system_sleepmode sleep_mode)
{

#if (SAMD20 || SAMD21 || SAMR21)

    /* Get MCU revision */
    uint32_t rev = DSU->DID.reg;

    rev &= DSU_DID_REVISION_Msk;
    rev = rev >> DSU_DID_REVISION_Pos;

#if (SAMD20)
    if (rev < _SYSTEM_MCU_REVISION_E) {
        /* Errata 13140: Make sure that the Flash does not power all the way down
         * when in sleep mode. */
        NVMCTRL->CTRLB.bit.SLEEPPRM = NVMCTRL_CTRLB_SLEEPPRM_DISABLED_Val;
    }
#endif

#if (SAMD21 || SAMR21)
    if (rev < _SYSTEM_MCU_REVISION_D) {
        /* Errata 13140: Make sure that the Flash does not power all the way down
         * when in sleep mode. */
        NVMCTRL->CTRLB.bit.SLEEPPRM = NVMCTRL_CTRLB_SLEEPPRM_DISABLED_Val;
    }
#endif

My hypothesis is that Atmel fixed the Flash power down problem (errata 13140), but not perfectly. There are still (rare) timing circumstances where the Flash does not power up fast enough after the ARM core wakes up and feeds it garbage for the next instruction to execute after the "WFI" (Wait for Interrupt) assembly instruction, which causes a hard fault. By placing the code in SRAM this problem is avoided, because the SRAM is always powered and ready to supply the next instruction.

Regards,

Pieter

https://piconomix.com/contact

Hi everyone,

It would also be useful to determine if the SAMD21 locks up while it is trying to execute from SRAM or FLASH. Here is a proposed debug trick: Mark debug counter variables as "NO_INIT" so that their values will not be reset to zero when the FW starts up. Increment the value before exiting FLASH, entering SRAM and then again after WFI, exiting SRAM and entering FLASH again. When the FW starts up and detects that a WatchDog reset occurred, it will wait for you to connect a debug probe, so that you can inspect the debug counter values. The values will then give you a better indication were it locked up.

Here is the (almost complete) pseudo code:

NO_INIT static uint32_t standby_sleep_enter_counter;
NO_INIT static uint32_t standby_sleep_exit_counter;

NO_INIT static uint32_t wfi_enter_counter;
NO_INIT static uint32_t wfi_exit_counter;

RAMFUNC void main_standby_sleep(void);

void main_standby_sleep(void)
{
    // Disable interrupts
    __disable_irq();
    __DMB();
    // Select STANDBY sleep mode
    SCB->SCR |=  SCB_SCR_SLEEPDEEP_Msk;
    // Increment WFI enter counter (executing from SRAM)
    wfi_enter_counter++;
    // Wait for Interrupt
    __DSB();
    __WFI();
    // Increment WFI exit counter (executing from SRAM)
    wfi_exit_counter++;
    // Enable interrupts
    __DMB();
    __enable_irq();
}

static void wdt_early_warning_handler(void)
{
}

static void main_watchdog_enable(void)
{
    struct wdt_conf config;

    wdt_get_config_defaults(&config);
    config.clock_source         = GCLK_GENERATOR_3;
    config.early_warning_period = WDT_PERIOD_8192CLK;
    config.timeout_period       = WDT_PERIOD_16384CLK;
    wdt_set_config(&config);

    wdt_register_callback(wdt_early_warning_handler, WDT_CALLBACK_EARLY_WARNING);
    wdt_enable_callback(WDT_CALLBACK_EARLY_WARNING);

    wdt_reset_count();
}

static void main_watchdog_disable(void)
{
    struct wdt_conf config;

    wdt_get_config_defaults(&config);
    config.enable = false;
    wdt_set_config(&config);
}

static void main_watchdog_restart(void)
{
    wdt_reset_count();
}

int main (void)
{
    // Watchdog reset?
    if(PM->RCAUSE.reg & PM_RCAUSE_WDT)
    {
        main_watchdog_disable();
             // Wait here for debug session to inspect debug counter values

        for(;;)
        {
            led_red_on();
            delay_ms(100);
            led_red_off();
            delay_ms(100);
        }
    }

    // Enable watchdog
    main_watchdog_enable();
    // Reset debug counters
    standby_sleep_enter_counter = 0x80000000;
    standby_sleep_exit_counter  = 0x80000000;
    wfi_enter_counter           = 0x80000000;
    wfi_exit_counter            = 0x80000000;

    while (true)
    {
        // Increment enter standby sleep counter (executing from FLASH)
        standby_sleep_enter_counter++;
        // Enter STANDBY Sleep
        main_standby_sleep();
        // Increment exit standby sleep counter (executing from FLASH)
        standby_sleep_exit_counter++;
        // Restart watchdog
        main_watchdog_restart();
    }
}

Hi everyone!

Microchip support finally responded over the December holidays with a concrete hardware related explanation and fix. Here is their response verbatim (I hope that's OK):

We have identified the issue with WDT reset. It happens due to SysTick timer.

Issue is that WDT initiates the wake up, but then SysTick interrupt starts to get handled first before system is actually ready. Basically, what happens is that SysTick interrupt does not wait for the RAM to properly wake up from sleep.

So if you wake up from WDT, the system will wait for the RAM, but the core clock will actually be running, so SysTick interrupt may happen too. SysTick interrupt does not wait on the RAM, so the core attempts to run the SysTick handler and fails, since RAM is not ready. This causes a Hard Fault (in our testing SRAM is so slow to wake up even Hard Fault handler).

This mean that device wakes up, getting into the Hard Fault, stay there until WDT fully expires.

You can reproduce the issue quicker by running SysTick timer faster, and WDT wake ups also quicker.

The solution for the customer is to disable SysTick interrupt before going to sleep and enable it back after the sleep.

// Disable systick interrupt

SysTick->CTRL &= ~SysTick_CTRL_TICKINT_Msk;

// Deep sleep
sleepmgr_sleep(SLEEPMGR_STANDBY);

// Enable systick interrupt
SysTick->CTRL |= SysTick_CTRL_TICKINT_Msk;

I have been running a test for a few days and so far no lock ups. Please try the fix and respond to this thread to let us know if it works for you (or not).

Regards,

Pieter

Hi Rob,

I'm not familiar with the MK1000WiFi board or the "RTCZero.cpp" file. I use the ASF (Atmel Software Framework) Library that comes standard with ATmel Studio projects. The "sleepmgr_sleep(SLEEPMGR_STANDBY);" function call is to a function provided in the ASF. This function puts the microcontroller into Standby Sleep Mode. The "SysTick->CTRL &= ~SysTick_CTRL_TICKINT_Msk;" lines write directly to the SysTick peripheral's memory mapped registered. You can consult the datasheet to understand how it work.

Best of luck!

Pieter