Allocating a receiver buffer for variable length of data from host in USB communication.

Go To Last Post
31 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm working USB CDC on SAM D21 Xplained Pro. My application is to read the string of data send by the host to the device and send back the same  data to the host.

-I'm working using atmel studio 7.0, the code which i'm using is USB  Communication Device Class (CDC)  for ATSAMD21.

 

Here i'm using udi_cdc_read_buf(buf,size) to read the data. The problem which i'm facing is here in this call we need to set the size of data to read.

If the received data length is less or greater than the size which i set. The data loss will occur.

 

In my application the host send data with variable length every time. How can i fix this issue.

 

I was tracking down the read call iram_size_t udi_cdc_multi_read_buf(uint8_t port, void* buf, iram_size_t size) in this function i'm getting complete string from the host in "udi_cdc_rx_buf[port][buf_sel][pos] "   in this call--> memcpy(ptr_buf, &udi_cdc_rx_buf[port][buf_sel][pos], copy_nb);  

 

-->It will only copy the allocate size in the udi_cdc_read_buf(buf,size) to the "ptr_buf".

 

->Here i need to read  entire string received in the "udi_cdc_rx_buf" in the read call without set the size in read call how can i do this?

->How can i read the entire data received from the host ?

 

 

Any help will be appreciated ,

Thanks in advance.

 

 

This topic has a solution.
Last Edited: Wed. Jan 9, 2019 - 02:09 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Data should be lost ONLY if the message is longer than the  size of the allocated buffer. Why is it lost if the buffer is smaller?

 

The standard strategy is to make the buffer at least as large as the longest message. You really cannot allocate dynamically, because you don't know how big a given message will be until it is finished. You have to have a buffer to store it long enough to determine the length, so why not just make it big enough? Done.

 

Jim

Jim Wagner Oregon Research Electronics, Consulting Div. Tangent, OR, USA http://www.orelectronics.net

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hi  ka7ehk ,

  Thank you for your reply. I cant use malloc  or callaloc here.  How can i do this here can you help me with the code? Here in debug i can see the entire data in the 3D BUFFER .  i  get that data from there to main application?

Last Edited: Wed. Jan 9, 2019 - 05:58 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Why do you have  to get it from 3D BUFFER to anywhere else? Can't you use 3D BUFFER directly? Sorry, I don't know SAM D21 but this seems to me to be really basic programming. 

 

If you need to access 3D BUFFER by a different name, just create a pointer of the desired name that points to 3D BUFFER. Why wouldn't that work?

 

By the way, what is 3D BUFFER? The common practice is to use all capitals for macros. But, that is not a valid name for a macro or an array or anything. How is it defined?

 

Jim

Jim Wagner Oregon Research Electronics, Consulting Div. Tangent, OR, USA http://www.orelectronics.net

Last Edited: Wed. Jan 9, 2019 - 08:38 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0
p
please check the code flow

udi_cdc_read_buf((void *)usb_data,len); ---->Code flow

-------->Code flow
iram_size_t udi_cdc_read_buf(void* buf, iram_size_t size)
{
	return udi_cdc_multi_read_buf(0, buf, size);
}


-------->Code flow
iram_size_t udi_cdc_multi_read_buf(uint8_t port, void* buf, iram_size_t size)
{
    irqflags_t flags;
    uint8_t *ptr_buf = (uint8_t *)buf;
    iram_size_t copy_nb;
    uint16_t pos;
    uint8_t buf_sel;
    bool again;
#if UDI_CDC_PORT_NB == 1 // To optimize code
    port = 0;
#endif

udi_cdc_read_buf_loop_wait:
    // Check available data
    flags = cpu_irq_save();
    pos = udi_cdc_rx_pos[port];
    buf_sel = udi_cdc_rx_buf_sel[port];
    again = pos >= udi_cdc_rx_buf_nb[port][buf_sel];
    cpu_irq_restore(flags);
    while (again) {
        if (!udi_cdc_data_running) {
            return size;
        }
        goto udi_cdc_read_buf_loop_wait;
    }
    // Read data
    pos=0;
copy_nb = udi_cdc_rx_buf_nb[port][buf_sel] - pos;
if (copy_nb>size) {
        copy_nb = size;
    }
    memcpy(ptr_buf, &udi_cdc_rx_buf[port][buf_sel][pos], copy_nb); // BUFFER WITH DATA
    udi_cdc_rx_pos[port] += copy_nb;
    ptr_buf += copy_nb;
    size -= copy_nb;
    udi_cdc_rx_start(port);
    if (size) {
        goto udi_cdc_read_buf_loop_wait;
    }
    return 0;
}

Here in the memcpy it is copying the data to the ptr_buf. 

 copy_nb-->Size of buffer to read.  It is set in the udi_cdc_read_buf(buff,size) (Please check the code flow)

IS there any way to copy the entire  data in udi_cdc_rx_buf[port][buf_sel][pos]  to another buffer without losing any data.

Can you please help me with a code.

 

Last Edited: Wed. Jan 9, 2019 - 09:49 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Surely your problem actually starts before the code segment shown? It's when:

udi_cdc_read_buf((void *)usb_data,len); ---->Code flow

is invoked that the len of "usb_data" is already set to a fixed size. As Jim said:

ka7ehk wrote:
The standard strategy is to make the buffer at least as large as the longest message.
Why can't you do that?

 

Also note that:

    pos=0;
copy_nb = udi_cdc_rx_buf_nb[port][buf_sel] - pos;

looks like very suspicious code. Given that pos is always set to 0 on the first line what is the purpose of it appearing on the next line. Surely this is just:

copy_nb = udi_cdc_rx_buf_nb[port][buf_sel];

??

 

Anyway you have posted this ARM thread in an AVR forum so I am going to move it.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  clawson ,

  

   What changes i need to made in this code?If i'm not changing  the pos=0 i'm able to read the data from the host.   Can you help me how can i get the complete data without setting the size?

 

Thanks

 

Last Edited: Wed. Jan 9, 2019 - 10:38 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Use udi_cdc_get_nb_received_data to find out how much is available.

/Lars

 

Edit: looks like I already told you this

https://community.atmel.com/comm...

 

Last Edited: Wed. Jan 9, 2019 - 12:12 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi Lajon

 

   Thank you for your reply. . I'm  using this call for reading and checked using the call  udi_cdc_read_buf(buf,size) it is not having any connection to the function to the function udi_cdc_get_nb_received_data();, From there i'm not getting any size of received data.

 

As you suggested i set a flag in notify function & i'm reading the flow is like void uart_rx_notify(uint8_t port)-->udi_cdc_read_buf(buf,size)-->iram_size_t udi_cdc_read_buf(void* buf, iram_size_t size)-->iram_size_t udi_cdc_multi_read_buf(uint8_t port, void* buf, iram_size_t size).

 

From this calls how can i get the size of data received?

-->I'M getting the entire data in buffer  udi_cdc_rx_buf[port][buf_sel][pos] . I'M not able to process it. Please help me with a suggestion.

Last Edited: Wed. Jan 9, 2019 - 12:43 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I think you will have to show what the code looks like now. I certainly thought you should get the number of available bytes from that function when I wrote the example code:

        const iram_size_t n_rec = udi_cdc_get_nb_received_data();
        uint8_t buf[n_rec];
        udi_cdc_read_buf(buf, n_rec);

are you saying you get 0 from it or what?

/Lars

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon ,

 

    That was error in my side . I'M able to get the total number of bytes received.

 

 iram_size_t n_rec;
 uint8_t buf[n_rec];
while (true) {

if(data_ready){
 n_rec = udi_cdc_get_nb_received_data();        
udi_cdc_read_buf(buf,n_rec);
}

 Here i'm getting the size of received data. But i'm not getting the data in buffer it is showing buf[0] .

Can you please point out the mistake.

 

Thank you

 

This reply has been marked as the solution. 
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You are allocating a buffer of random size. Try

    while (true) {
        if (data_ready) {
            iram_size_t n_rec = udi_cdc_get_nb_received_data();    
            uint8_t buf[n_rec];
            udi_cdc_read_buf(buf, n_rec);
            data_ready = false;
        }
    }

/Lars

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hi  Lajon ,

   Thank you for your great help. Every thing is running fine now.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi,

 

    In my application the host is sending the data. In  between the transfer  the host will check the device is connected or not by continuous pinging in background (receive data from host  and responds back) .

It is for host to check the connection status.

 

I tried to implement this with the help of solution suggested    Lajon  . But if host is s sending other data the device is getting disconnected "stopping transfer to check device connection status " .

Why it is happening? It is not possible to perform multiple transfers at a time?

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

It's impossible to understand what is going on from this description. Is it the host that terminates the connection because there was no answer to the ping? What does a normal message look like? What does a ping message look like? Is it possible that what you receive contains both a normal message and a ping message in the same buffer?

/Lars

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hi  Lajon ,

 

    I used to send string to check the connection status every instant i send "1234". The device is connected  i have to send back the same  same string "1234" if the device is connected. It need to happened  at every instant. If host send the other data the connection breaks.

I'm receiving all data to the same buffer i set a flag and reading from main. How can i resolve this.

 

Thanks & Regards

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Look for "1234" in the buffer, if found send the "1234" response and remove "1234" and process the rest of the message normally. You might need to consider where the buffer the "1234" is also (to send the "1234" response in the correct sequence).

I sure hope "1234" is something that will only occur in the ping message, you did not answer the question  "What does a normal message look like?" 

/Lars

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi Lajon ,

 

    The normal messages i'm sending is all in  numbers ( string ) like numbers from  2000 to 3000 (i'm doing such a way that if i receive 2000 perform operation:1 if  receive 2001 perform:2). For pinging I'm doing in such way w that if i receive "1234" sending the same data back to the host.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon,

    Actually why its causing this problems, because i'm receiving all the data in a single buffer?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon,

 

 My application is to read the command  from the host to the device and perform the operation suggested by the host and send  acknowledgement back to the host from device.

 

 

 

  1.Device Waiting for second request from host for sending response for first request.

 

     

1st request ---->No acknowledgement received from device.

2nd request---->Host received  acknowledgement for 1st request .

3rd request----->Host received  acknowledgement 2nd request.

 

--->Why this issue is happening? How can i resolve this issue i'm stuck with this any help will be appreciated.

 

Thanks & Regards

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Is the host waiting for the response before making a new request, i.e., how much time passes between requests? I think you need to find out if the first ack is received eventually or if it in fact is not received at all when there is no additional ack sending done from the D21.

/Lars

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon ,

 

" I think you need to find out if the first ack is received eventually or if it in fact is not received "--->I checked the traffic using wire shark the ACK is receiving to the device.

 

 

   I tested this with the basic application code. Read data from the host and send the same data back to host. It is also giving the same issue. 

 

CODE SNIPPET:


void uart_rx_notify(uint8_t port)
{
UNUSED(port);
data_ready=true;
} 

-->Here when the data is received i'll send back the same data to the GUI.
-->Here i'm reading data from main.

 

while (true) {
if(data_ready)
{
udi_cdc_read_buf(rec_buffer,3);
udi_cdc_write_buf(rec_buffer,3);
} 

-->In this code i'm not making any changes i made changes only for this two functions

 

ISSUE:

In this same application also i'm facing the same issue. In the first case the GUI is sending 111 then 222. Here if i receive 111 i need to send back 111. In my case the device is sending 111 when 222 is received. Even the first response as the response for second data .

 

 

LOG:

byteArr55-->Data from host
received data ->Data send back from device.
 


libusb initialised
handle-->libusb device handle 0x7f263c173a60
device found

count----------------------------------------------------count-->1
byteArr55-->111

3-->bytes sent to device
read00
capacity-->64
------------------------------------ bytes read from device----->0
received data ->

byteArr55-->222

3-->bytes sent to device
read00
capacity-->64
------------------------------------ bytes read from device----->3
received data ->111

count----------------------------------------------------count-->2
byteArr55-->111

3-->bytes sent to device
read00
capacity-->64
------------------------------------ bytes read from device----->3
received data ->222 

Every time it is not sending the response for 1st data. It is sending it as the response for 2nd data.

 

Please have a look at this issue.

 

Thanks & Regards

Last Edited: Thu. Jan 24, 2019 - 12:11 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I don't think you have answered the question "Is the host waiting for the response before making a new request", I don't know how much time passes between the two requests here:

3-->bytes sent to device
read00
capacity-->64
------------------------------------ bytes read from device----->0
received data ->

byteArr55-->222

3-->bytes sent to device

Find out if the first ack would eventually be received if there is no other activity. If not that is a serious error, but if it is this can be expected (i.e., some delay before the USB stack actually sends something).

/Lars

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hi  Lajon ,

 I faced this issue on my application now i'm checking this on the code i mentioned above there also i got the same error.(Recv and  Snd back the same data).

 

==> "Is the host waiting for the response before making a new request"

 

      --->Here i did in such a way that host is continuously sending 111 & 222 alternatively.   Host wont wait for any request i will send continuously in both cases.

 

==> Find out if the first ack would eventually be received if there is no other activity

 

--->No other data transfer is happening in between i checked traffic using the wireshark. 

 

---->Device not ready to respond instantly it wait for another data to come to  send response. If i debug i can see the data on the buffer which is using for send data it wait until next request from host.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

So you still have to

Find out if the first ack would eventually be received if there is no other activity

/Lars

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon ,

  I tried with sending data one time to host without the loop. When i click send then only host transfer the data.

First time the device will send back the data which is received. 

2nd data it wont send anything back. When the 3rd data is send by the host ,device will send response for the second data. How this happens?

 

Best Regards

Last Edited: Fri. Jan 25, 2019 - 06:41 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I don't know, the attached project works for me with this python script as the host side:

import serial
from time import sleep
with serial.Serial('COM17', 115200, timeout=0) as ser:
    n = 1
    nBad = 0
    while True:
        sendstr = 'r' + '{:04}'.format(n)
        sendbytes = sendstr.encode()
        ser.write(sendbytes)
        sleep(0.0011)
        s = ser.read(size=64)
        if (s != sendbytes):
            nBad += 1
        print("Got:", s, 'Bad:', nBad, '/', n)
        n += 1

With 1 ms or less delay after send there will be "errors", this is expected due to buffering in the USB device implementation, i.e., it will wait for more data to send but not longer than the start of frame (SOF) period (1 ms).

I write "errors" because there is no loss of data, it only means a read call will get nothing and the next read call will get two responses.

/Lars

 

Attachment(s): 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

hi  Lajon ,

I checked  the code you send with mu GUI it is giving the same error. I'M  unable to check with the python code.

I'm attaching the host code below please check this code and verify.

 

 

Thanks & Regards

Attachment(s): 

Last Edited: Tue. Jan 29, 2019 - 09:00 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well you could try a >1ms delay after write before reading. But I would think the 100 (ms?) TIMEOUT would already provide this needed wait.

                write2(handle,byteArr);
                // delay here
                read(handle);

/Lars

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon ,

I added the delay as you suggested and checked it with delay of 100 and 1000.

 

 

for(int i=1;i<=1;i++)
			{

				System.out.println();
				System.out.println("count----------------------------------------------------count-->"+i);
				write2(handle,byteArr);
                                Thread.sleep(1000);
				read(handle);
				write2(handle,byteArr1);
                                Thread.sleep(1000);
				read(handle);}

 

=>First time host will send 111 & 222 respectively one time

 

  •   Able to receive both data correctly.

 

=>2-Sending  111 & 222 respectively one time:

 

  • First data -No response 
  • 2nd data- 111 AS RESPONSE

 

=>3rd Sending  111 & 222 respectively one time:

 

  • First data - 222 AS RESPONSE
  • 2nd data- 111 AS RESPONSE

 

 

If i tried this i'm getting this response .

      ==>If i restarting the device before every receive this issue is not happening.

      ==>After receiving some data for the next sequence this issue is coming.

      ==>If i send this data in for loop ten times first 10 time data will receive and send back to the host correctly.

              After completing the loop 10 times , if i tried to send again in loop i=1(data-111 No response), i=2(data-222 resp -111) ,i=3(data-111 resp-222) like this happens.

 

I think you are more clear about the issue now. Please help me with this

 

Best Regards

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi  Lajon ,

    The issue i'm facing is happening may be because of if host send a data device is sending the data in the response buffer. 

 

  • In my application if host send a particular ID only the device responds. For each ID seprate functions are there. 

 

For Ex: host send 1000 device responds with 2000

           host send 1001 device responds with 2001

 

  • The read calls i'm using is  in respective functions corresponds to the ID.
  • If device receive correct ID only it should send the data from respective ID.

 

Here in every functions i'm memset the send buffer before send the data. I think that it is storing the data somewhere else before sending so when next ID is coming it is sending data from there not from the function.Any help regarding this will be appreciated.