Python asyncio protocol support

[jameshilliard]

I was thinking it might make sense to wire this up somehow to a python asyncio protocol.

I wrote something that acts as a virtual serial port proxy using a pty and asyncio tcp protocol with ser2net but that of course is missing some features this has.

Something like this is handy if one wants to run a serial based application on a development machine when the serial device is attached to a separate embedded system(which may not have the right environment to run said application itself) in addition to allowing for debugging/manipulating of commands bidirectionally between the application and device(think of it like a serial based mitmproxy).

serialproxy.py

I'm wondering, was there a reason for using swig for the python interface of serialsim? Wouldn't it be easier to just use normal python ioctl calls?

[cminyard]

On Wed, Oct 05, 2022 at 06:09:16PM -0700, James Hilliard wrote: I was thinking it might make sense to wire this up somehow to a python asyncio protocol. I wrote something that acts as a virtual serial port proxy using a pty and asyncio tcp protocol with ser2net but that of course is missing some features this has.

Yes, otherwise it's almost impossible to automate testing of what is happening on a serial port.

Something like this is handy if one wants to run a serial based application on a development machine when the serial device is attached to a separate embedded system(which may not have the right environment to run said application itself) in addition to allowing for debugging/manipulating of commands bidirectionally between the application and device(think of it like a serial based [mitmproxy](https://mitmproxy.org/)).

I can see that. What's missing for that application is immediate notification that something has changed. At least if you want to propagate baud rate and other settings. That's possible, I just didn't need it.

[serialsim.py](https://gist.github.com/jameshilliard/c10eb210ee1fbcabe68f5af1c1fdbd13) I'm wondering, was there a reason for using swig for the python interface of serialsim? Wouldn't it be easier to just use [normal python ioctl calls](https://docs.python.org/3/library/fcntl.html#fcntl.ioctl)?

Well, it was actually easier to use swig if you know it, and from that python doc: This function is identical to the fcntl() function, except that the argument handling is even more complicated. :-) I am not sure you could get the termios stuff to work properly in a portable manner. I suppose it's possible, but I took the path of least resistance for me. I'm fairly surprised that no one has discovered this until now and wanted to use it for some different application. I'm certainly open to extending it to make it more useful. My testing frameworks are tied around the swig interface, but I'm not against another one. It would be more convenient to be pure python.

…

-corey

[jameshilliard]

I can see that. What's missing for that application is immediate notification that something has changed. At least if you want to propagate baud rate and other settings. That's possible, I just didn't need it.

Yeah, I was configuring stuff manually on the ser2net side.

Well, it was actually easier to use swig if you know it, and from that
python doc: This function is identical to the fcntl() function, except
that the argument handling is even more complicated. :-)

Swig stuff tends to be a bit annoying to build I guess in some cases, and is harder to distribute(ie can't just upload a simple pure python wheel to pypi).

I'm fairly surprised that no one has discovered this until now and
wanted to use it for some different application. I'm certainly open to
extending it to make it more useful. My testing frameworks are tied
around the swig interface, but I'm not against another one. It would be
more convenient to be pure python.

Yeah, I could probably give porting it to pure python a shot, have any good examples of something consuming the swig interface that I could use for checking that there's no regressions when migrating to pure python?

Probably easier to extend once migrated to pure python IMO.

By the way what's the difference between this and your v2 patch? Did that go any further in regards to getting accepted upstream?

[jameshilliard]

I am not sure you could get the termios stuff to work properly in a portable manner. I suppose it's possible, but I took the path of least resistance for me.

Yeah, looks like I need to special case types for a few archs for the termios2 structure but this seems to be mostly working:

import ctypes
import fcntl
import termios

tcflag_t = ctypes.c_uint
cc_t = ctypes.c_ubyte
speed_t = ctypes.c_uint
NCCS = 19

class termios2(ctypes.Structure):
    _pack_ = 1
    _fields_ = [("c_iflag", tcflag_t),
                ("c_oflag", tcflag_t),
                ("c_cflag", tcflag_t),
                ("c_lflag", tcflag_t),
                ("c_line", cc_t),
                ("c_cc", cc_t * NCCS),
                ("c_ispeed", speed_t),
                ("c_ospeed", speed_t)]

UNCCS = 32

def _IOR(ty, nr, size):
    return (2 << 30) | (ord(ty) << 8) | (nr << 0) | (size << 16)

TIOCSERGREMTERMIOS = _IOR('T', 0xe7, ctypes.sizeof(termios2))

def getspeed(baudrate):
    return getattr(termios, 'B{}'.format(baudrate))

def get_remote_termios(fd):
    ktermios = termios2()
    rv = fcntl.ioctl(fd, TIOCSERGREMTERMIOS, ktermios);
    user_c_cc = []
    for i in range (0, UNCCS):
        if i == termios.VTIME or i == termios.VMIN:
            user_c_cc.append(ktermios.c_cc[i])
        elif i < NCCS:
            user_c_cc.append(chr(ktermios.c_cc[i]))
        else:
            user_c_cc.append(chr(0))
    return (
        ktermios.c_iflag,
        ktermios.c_oflag,
        ktermios.c_cflag,
        ktermios.c_lflag,
        getspeed(ktermios.c_ispeed),
        getspeed(ktermios.c_ospeed),
        tuple(user_c_cc),
    )

[cminyard]

On Wed, Oct 05, 2022 at 08:46:17PM -0700, James Hilliard wrote: > I can see that. What's missing for that application is immediate notification that something has changed. At least if you want to propagate baud rate and other settings. That's possible, I just didn't need it. Yeah, I was configuring stuff manually on the ser2net side. > Well, it was actually easier to use swig if you know it, and from that python doc: This function is identical to the fcntl() function, except that the argument handling is even more complicated. :-) Swig stuff tends to be a bit annoying to build I guess in some cases, and is harder to distribute(ie can't just upload a simple pure python wheel to pypi). > I'm fairly surprised that no one has discovered this until now and wanted to use it for some different application. I'm certainly open to extending it to make it more useful. My testing frameworks are tied around the swig interface, but I'm not against another one. It would be more convenient to be pure python. Yeah, I could probably give porting it to pure python a shot, have any good examples of something consuming the swig interface that I could use for checking that there's no regressions when migrating to pure python? Probably easier to extend once migrated to pure python IMO.

Yes, probably so. The only users I know are the ser2net and gensio tests.

By the way what's the difference between this and your [v2 ***@***.***/)? Did that go any further in regards to getting accepted upstream?

Greg KH had a bunch of comments, and I asked some questions and never got a response. Using kernel threads are frowned upon, so I'd probably have to switch to a timer, and the sysfs interface probably needs to go since parsing strings in the kernel is also frowned upon. I was waiting for interest, really. No one else has chimed in.

…

-corey

[cminyard]

On Thu, Oct 06, 2022 at 01:17:42AM -0700, James Hilliard wrote: > I am not sure you could get the termios stuff to work properly in a portable manner. I suppose it's possible, but I took the path of least resistance for me. Yeah, looks like I need to special case types for a few archs for the termios2 structure but this seems to be mostly working: ```python import ctypes import fcntl import termios tcflag_t = ctypes.c_uint cc_t = ctypes.c_ubyte speed_t = ctypes.c_uint NCCS = 19 class termios2(ctypes.Structure): _pack_ = 1 _fields_ = [("c_iflag", tcflag_t), ("c_oflag", tcflag_t), ("c_cflag", tcflag_t), ("c_lflag", tcflag_t), ("c_line", cc_t), ("c_cc", cc_t * NCCS), ("c_ispeed", speed_t), ("c_ospeed", speed_t)]

I'm not a python expert, but I think that should work. It's all chars, so that should be pretty straightforward.

…

-corey

UNCCS = 32 def _IOR(ty, nr, size): return (2 << 30) | (ord(ty) << 8) | (nr << 0) | (size << 16) TIOCSERGREMTERMIOS = _IOR('T', 0xe7, ctypes.sizeof(termios2)) def getspeed(baudrate): return getattr(termios, 'B{}'.format(baudrate)) def get_remote_termios(fd): ktermios = termios2() rv = fcntl.ioctl(fd, TIOCSERGREMTERMIOS, ktermios); user_c_cc = [] for i in range (0, UNCCS): if i == termios.VTIME or i == termios.VMIN: user_c_cc.append(ktermios.c_cc[i]) elif i < NCCS: user_c_cc.append(chr(ktermios.c_cc[i])) else: user_c_cc.append(chr(0)) return ( ktermios.c_iflag, ktermios.c_oflag, ktermios.c_cflag, ktermios.c_lflag, getspeed(ktermios.c_ispeed), getspeed(ktermios.c_ospeed), tuple(user_c_cc), ) ``` -- Reply to this email directly or view it on GitHub: #2 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[jameshilliard]

I was waiting for interest, really. No one else has chimed in.

Having this upstream does sound handy, then anyone wanting to use it would only need to have the pure python userspace serialsim module.

I'm not a python expert, but I think that should work. It's all chars,
so that should be pretty straightforward.

There seemed to be a number of extraneous intermediary operations going on with the swig module that didn't seem to be needed, for example this user_termios structure didn't seem to be doing anything useful.

I just removed some layers of indirection and simplified the logic to return the same value.

The returned structure for get_remote_termios seems rather strange, I mean it was easy enough to replicate the format in pure python but I'm not really sure what the purpose of the weird nested tuple return format is in the first place.

[cminyard]

I was waiting for interest, really. No one else has chimed in.

Having this upstream does sound handy, then anyone wanting to use it would only need to have the pure python userspace serialsim module.

I'm not a python expert, but I think that should work. It's all chars,
so that should be pretty straightforward.

There seemed to be a number of extraneous intermediary operations going on with the swig module that didn't seem to be needed, for example this user_termios structure didn't seem to be doing anything useful.

user_termios is there because the representation of termios you get from the kernel is different than the main termios structure, you get from glibc and there is an include nightmare getting both of them. This code was originally written to support C code, not python, too.

I just removed some layers of indirection and simplified the logic to return the same value.

The returned structure for get_remote_termios seems rather strange, I mean it was easy enough to replicate the format in pure python but I'm not really sure what the purpose of the weird nested tuple return format is in the first place.

That is the normal "termios" structure that you get from the python termios calls. I made it match that so it could be compared easily. See https://docs.python.org/3/library/termios.html

Anyway, thanks for your work on this. It's a big improvement.

[jameshilliard]

user_termios is there because the representation of termios you get from the kernel is different than the main termios structure, you get from glibc and there is an include nightmare getting both of them. This code was originally written to support C code, not python, too.

Ah, that makes sense, wasn't sure of the history there.

That is the normal "termios" structure that you get from the python termios calls. I made it match that so it could be compared easily. See https://docs.python.org/3/library/termios.html

Oh, hadn't realized that's what termios was returning(I haven't worked with it much directly).

By the way, is there a good way to get notified when a serial application say configures the serialsim serial port parameters without polling the ioctl's?

For asyncio protocol reading/writing I'm thinking I should probably use a fd watcher like I used with my pty based script or maybe a pipe protocol read/writer, what do you think would work best there?

[cminyard]

On Fri, Oct 14, 2022 at 08:05:58PM -0700, James Hilliard wrote: > user_termios is there because the representation of termios you get from the kernel is different than the main termios structure, you get from glibc and there is an include nightmare getting both of them. This code was originally written to support C code, not python, too. Ah, that makes sense, wasn't sure of the history there. > That is the normal "termios" structure that you get from the python termios calls. I made it match that so it could be compared easily. See https://docs.python.org/3/library/termios.html Oh, hadn't realized that's what termios was returning(I haven't worked with it much directly). By the way, is there a good way to get notified when a serial application say configures the serialsim serial port parameters without polling the ioctl's?

Not currently, that's what I was mentioning earlier; that's probably something you would need. It would require a kernel change.

For asyncio protocol reading/writing I'm thinking I should probably use a [fd watcher like I used with my pty based script](https://docs.python.org/3/library/asyncio-eventloop.html#watching-file-descriptors) or maybe a [pipe protocol read/writer](https://docs.python.org/3/library/asyncio-eventloop.html#working-with-pipes), what do you think would work best there?

I'm not sure at that level. At the driver level, you have two basic options: 1) Add an ioctl that will block until the other end changes something. or the device closes. 2) Add an ioctl that will return an fd that will watch for changes. #1 is probably the simplest to implement, but you can't use poll() on it, which is fairly inconvenient. #2 is probably the right thing to do, but harder to implement. You could also do something with signals, like async I/O, but that's a real pain to use. You might be able to implement #2 with a sysfs file. I don't know how sysfs handles poll(), though. The trick there is finding the file from the device, which would not be straightforward.

…

-corey