Serial handling at high data rates with HW flow control on ST platforms

[RobMeades]

Description

• Type: Possible Bug?
• Priority: Minor

Target
STM

I'm dealing with an application on an ST platform in which the serial port is being run at quite a high rate (460800 bits/s). In order to receive characters reliably at the MCU with this kind of rate, HW flow control is being employed. While investigating some character loss I'm seeing in this scenario I noticed something odd: once the serial port buffer is sufficiently full for the RTS line to be raised, it remains in that state; the RTS line is raised after every character, like the serial port's HW buffer is never being emptied.

Looking at the code from UARTSerial() down:

• UARTSerial::rx_irq() calls SerialBase::_base_getc() in a loop while SerialBase::readable() is true.
• ST's serial_readable() (in targets/TARGET_STM/serial_api.c) just checks for the flag UART_FLAG_RXNE being set for the UART.
• However, the interrupt handler (uart_irq() in targets/TARGET_STM/TARGET_STM32F4/serial_device.c) clears the UART_FLAG_RXNE flag.

So if the interrupt has gone off and flow control has stopped any more characters being received, the flag will not be set and UARTSerial::rx_irq() won't go around its loop at all, which would lead to the buffer never being emptied. I think...?

Aside from the fact that this would be inefficient, I wonder if it might be implicated in the character loss I'm seeing.

Steps to reproduce
Run serial port via UARTSerial class (e.g. the new Cellular API) on STM part at a data rate high enough that the interrupt can't be serviced within the usual interrupt latency (e.g. 460800 bits/s) and watch how quickly RTS is raised once a few 10's of characters have been received.

cc: @kjbracey-arm , @hasnainvirk , @adustm , @LMESTM , @jeromecoutant

[RobMeades]

OK, so my logic above is flawed: the interrupt handler only clears the UART_FLAG_RXNE flag after it has called UARTSerial::rx_irq(), so the flag will be set when serial_readable() is called. But the effect remains. I'll poke at it more tomorrow and report back on whether I'm seeing phantoms.

[LMESTM]

@RobMeades Hi Rob - would be great if you can share a a basic sample code, using PC serial for instance and the same APIs you're using in your program

[RobMeades]

Understood, though that might take a while as I would have to understand how UARTSerial worked first. I am actively investigating the original data-loss-at-high-data-rates issue (which may be related) with @kjbracey-arm at the moment, let me continue with that and I will return to this either (a) when we've sorted it or (b) when we think that resolving this might resolve that, IYSWIM.

[kjbracey]

Hi - I had a look at the driver, and it occurred to me that the manual clearing of RXNE looked dubious. Rob confirms that removing the manual clearing from uart_irq solves the problem.

The act of reading from the data register clears RXNE automatically. If the IRQ handler clears it manually, then it can drop a byte of data that arrived during execution of the handler.

It's also possible that the corresponding manual clearing of TC for Tx could be an issue - losing an interrupt if somehow the a transmit triggered by the handler completes.

On another note, Rob's traces show that output is slow - it doesn't maintain full line rate.

I believe that's because TxIrq is attached to TC "transmit complete" rather than TXE "transmit register empty". So we see a funny pattern where the interrupt handler writes 2 bytes at a time, then a delay before the next 2. Don't know if it's worth adjusting that.

[RobMeades]

FYI, ref. Kevin's point about the manual clearing of TC for Tx, after commenting out the manual clearing of RXNE in uart_irq() I was able to successfully receive data at the serial port without any loss at 460800.

However my setup is end to end to a remote server (a modem is attached to the serial port) and I found that, when I switched to 460800, the server received no data whatsoever, yet it was fine when I ran the modem interface at 230400. My logic analyzer showed that there was no Tx data on the serial lines at all. Speculatively, I commented out the manual clearing of TC for Tx, in uart_irq() and my transmit sprang to life. So I think Kevin is probably right on that count also.

[kjbracey]

That may in turn be the answer for the odd use of TC rather than TXE. If it was TXE, and the clear was in the same place, it would have jammed at any baud rate.

[RobMeades]

FYI, speculatively (not claiming any science here) I tried attaching the Tx interrupt to TXE rather than TC. This worked fine but my end-to-end uplink throughput (my data is 100% uplink UDP packets) dropped by around 20% (180 kbits/s versus 220 kbits/s).

[kjbracey]

I assume that would be the effect of doing 1 byte per interrupt rather than 2, so having increased CPU load. The bursty thing worked out in your favour. That was why I wasn't sure about adjusting it.

Devices with FIFOs usually have water-mark settings to counteract that, eg generating an interrupt with 4 bytes or space available in the buffer, else after a timeout.

This lacks FIFOs, so nothing you can do on RX, but using TC effectively lets you do 2 at a time - once TC triggers you know the next write will vacate the TX register immediately, allowing the interrupt handler to write 2.

[kjbracey]

A data rate that high at one-byte-per-interrupt is really pushing it. The original point of the FIQ interrupt and dedicated registers on the ARM was to achieve 250 kbit/s 1-byte-at-a-time to the floppy disk drive, meaning the ARM could do the job itself without needing a DMA controller.

Managing the same with normal IRQs is a stretch, even with the faster CPU core speeds we have 30 years later (memory and peripherals haven't got proportionately faster).

[kjbracey]

So what I'm saying is we would need to start worrying about the DMA/"multibuffer" mode here to get near line rate. Whereas a UART with FIFOs and water marks would work fine and transparently.

Not sure if its possible to hide that DMA/multibuffer inside the HAL's normal serial_getc/putc API.

UARTSerial doesn't support the separate asynch API at the moment, but even if it did the STM driver doesn't seem to DMA for asynch anyway.

[RobMeades]

@LMESTM I'd be happy to support such work as we have modems connected to this serial port that support far higher data rates than 460800. How can I help?

[LMESTM]

@RobMeades I'll have a look asap, hopefully next Monday - what would help is a simple sample code with Serial to PC to reproduce the issue

[RobMeades]

Thought you might say that, have been working on it, will post something here when I have it.

[RobMeades]

@LMESTM : I've created a repo containing a Greentea test of serial port performance. Its checks for zero character loss and an expected throughput, repeating the test at increasing baud rates up to a configurable maximum (I've proposed a throughput of no more than 20% below the line rate at up to 460800 bits/s). It requires an mbed board with a spare serial port on which the Tx line can be looped back to the Rx line and the RTS line to the CTS line. It is fully configurable and can be run under a debugger if required.

You can find the test here: https://github.com/u-blox/baud-rate-test. I hope this helps you make us a nice fast DMA'ed serial implementation for the STM32F4 :-).

[LMESTM]

@RobMeades - thanks I'll have a look. Short term objective is to make the setup stable and fix the interrupts management.

@kjbracey-arm said:

UARTSerial doesn't support the separate asynch API at the moment, but even if it did the STM driver doesn't seem to DMA for asynch anyway

so is there a point in supporting DMA as of now ? This is not in our plans for now.