Is there a hardware bug in SAMD devices waking up from standby -- specifically I2C Slave

Go To Last Post
6 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have a product that needs to act as an I2C slave and wake up from sleep, I'm using a SAMD11. I've spent a long time trying to track down the problem, which only happens sometimes. I've instrumented my code to drive GPIO to show what is happening.

 

This is what happens when it works. For testing I have added a resistor in series in the sda and sdc so the slave can pull the line lower than the master and have included analog captures. For reference the production product has better capacitance/resistance than the debug setup.

Working behaviour:

  1. The samd11 starts in 'standby sleep mode' it is set to wake on a matching I2C address.
  2. A matching address occurs, and on the 9th low transition the samd11 holds the SCL line (shown on channel 1) to clock stretch.
  3. The samd11 enters the interrupt handler, which sends an ack on the I2C and a variable is set to signals to the main loop to not sleep again until data is processed. The rest of the I2C transactions complete and after recieving a STOP, the device enters sleep again.

This is all correct behaviour. However, apparently randomly sometimes the samd11 doesn't wake up as shown here:

Broken Behaviour

Here, it can be seen that that the samd11 doesn't hold the SCL, nor is the interrupt handler entered. Even more odd, about 10ms later the chip hard faults.

As can be seen from an overview, the faults are sporadic.

 

 

To trace what was going on, I tried to make sure that the first instruction in the interrupt handler and the first instruction after the WFI instruction both are to drive the GPIO line to signal in the traces.  I've double checked the driver code (using both ASF and then writing my own interrupt routines) and can see no problem. In addition, the samd11 behaves correctly when the same code runs but only IDLE2 sleep rather than STANDBY is used.

 

This issue has taken a lot of time, but I'm beginning to think it's a bug in the hardware related to wake from STANDBY.

I've checked the forum and can't see any other solutions, though other people report problems when coming out of sleep and I wonder if they're related -- e.g. https://community.atmel.com/foru..., https://community.atmel.com/foru...

In my I2C slave example we can see the events leading up to the chip waking up, where a timer is internal. From the behaviour I observe I'm struggling to think how it can be anything other than a hardware bug -- no code appears to get run that could fail, yet a hardfault occurs.

 

Obviously I'm keen to try and get this sorted especially if it sheds any light on other similar problems with wake -- thanks in advance for any other suggestions.

This topic has a solution.
Last Edited: Tue. Feb 12, 2019 - 10:26 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi @darrenw,

 

For reference, see this thread:

https://community.atmel.com/foru...

 

For me it was counter-intuitive, but after reading about it online and in the forums, it makes good sense. For the ARM Cortex core you can disable all interrupts, select STANDBY sleep mode, call WFI, and after the core has woken up enable interrupts again. This is great, because you can take a debug GPIO pin high just before WFI and take the GPIO pin low just after WFI and be assured that the core did try to execute interrupt code and failed. If the debug GPIO pin is stuck high, it demonstrates that the core was unable to execute any code after WFI.

 

I implemented two atomic debug counters. Incrementing one just before WFI and incrementing the other one just after WFI. If the watch dog resets the micro, I have startup code that detects the watchdog reset condition and starts an infinite loop flashing an LED to signal the error condition. I am then able to halt the micro and inspect the counters in SRAM, because it is not reset to zero by the CRT initialization code. I hope you get my drift.

 

What do you mean that a hard fault occurs? What kind of hard fault? How did you determine this? I was unable to halt the micro to see if a hard fault occurred and I added special debug code to signal any faults, but it did not trigger :(

 

Regards,

Pieter

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi Pieter, thanks for your suggestions.

pieterc wrote:

 For the ARM Cortex core you can disable all interrupts, select STANDBY sleep mode, call WFI, and after the core has woken up enable interrupts again. This is great, because you can take a debug GPIO pin high just before WFI and take the GPIO pin low just after WFI and be assured that the core did try to execute interrupt code and failed. If the debug GPIO pin is stuck high, it demonstrates that the core was unable to execute any code after WFI.

 

Have done this and yes core definitely doesn't execute any code after WFI. I even went as far as polling system_interrupt_is_pending(SERCOM1_IRQn) and then 'manually' dispatched the interrupt handler so I didn't need interrupts enabled at all. As such it appears it is the wake up from deep sleep which is broken.

 

pieterc wrote:

What do you mean that a hard fault occurs? What kind of hard fault? How did you determine this? I was unable to halt the micro to see if a hard fault occurred and I added special debug code to signal any faults, but it did not trigger :(

 

I added the following in src\ASF\sam0\utils\cmsis\samd11\source\gcc\startup_samd11.c to indicate a hardfault.
 

void HardFault_Handler       ( void ) __attribute__ ((weak, alias("mDummy_Handler")));

void mDummy_Handler(void)
{
    while(1) {
        port_pin_set_output_level(EXT1_PIN_GPIO_3, 1);
        port_pin_set_output_level(EXT1_PIN_GPIO_3, 0);    
    }
}

and then was able to reattach the debugger to confirm I'd got here/check the approriate status registers. Rewinding the stack indicated the PC had been pointing to an instruction just after the the WFI. It appeared it wasn't always the same instruction (possibly only varying against compile time changes and not varying at run time -- I must admit at the time it didn't seem consistent but I was so confused about what was going on I changed a few things including clock config) and one of the instructions was a cmp -- suggesting it wasn't related to a bad data load. However, this process was definitely inconsistent; sometimes the debugger would attach and sometimes it wouldn't. I wasn't even sure if the act of attaching the debugger prodded some state. The NVIC bit for sercom1 and the sercom interrupts were set though. Unfortunately I've now got to the stage where it doesn't catch it and it now resets.

 

At the time I had multiple wakeup sources too, so I'm now worried how they will interact after reading the ST errata. Basically I'd just like someone to give me confidence that my products aren't going to lock up in the field. Suppose it'll come down to relying on the WDT and then being careful with reset state. Good idea with regard persisting SRAM, had pondered about using that as part of a workaround.

 

So in conclusion, I can't think of any other explanation than this is a hardware bug when waking from SYSTEM_STANDBY. Annoying in itself and annoying no mention of any errata.

 

I'll try Microchip support, see if there's any joy and post back results.

This reply has been marked as the solution. 
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

For everyone finding this thread, I have posted my possible solution in this thread to avoid duplication:

https://community.atmel.com/foru...

 

Regards,

Pieter

https://piconomix.com/contact

Last Edited: Wed. Nov 7, 2018 - 01:12 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Just to draw this thread to conclusion,  I can confirm this sorted things:

 

https://community.atmel.com/comm...

 

Many thanks to everyone who helped get to bottom of this, particularly @pieterc for persistence!

Last Edited: Tue. Feb 12, 2019 - 10:25 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi Darren!

 

My pleasure :) I'm glad that I can finally close that chapter in my life! It was like solving a missing person's case. It will haunt you forever until you finally get to the truth.

 

Kind regards,

Pieter