Netdev: RFC for a network device driver model

[scottfeldman]

Hi, I'd like to present an RFC for a network device driver model for Tinygo
I've been working on. I call it netdev. It incorporates a socket-like API
idea from @justin A Wilson
.
ref: https://github.com/scottfeldman/tinygo/tree/netdev
ref: https://github.com/scottfeldman/tinygo-drivers/tree/netdev

Goal

Provide a subset of functionality of the Go "net" package for Tinygo.
Higher-level protocols, applications, and tests that depend on Go "net" will
"just work" on Tinygo, as long as they use the subset.

The subset is:

    type Addr
    type Conn
            Dial(network, address)
    type IP
    type Listener
            Listen(network, address)
    type TCPAddr
    type TCPConn
            DialTCP(network, laddr, raddr)
            (more)
    type UDPAddr
    type UDPConn
            DialUDP(network, laddr, raddr)
            (more)

Additional limitations:

    TCPConns are: IPv4 + TCP + STREAM
    UDPConns are: IPv4 + UDP + DGRAM

    No IPv6 support
    No Unix socket support

Issues with current model

• Duplication of code for higher-level protocols. See wifinina/http.go and
  rtl8720dn/http.go. Both implement "net/http" but do so in a
  device-dependent manner. "net/http" should be device-agnostic and built on
  "net" TCPConn, which hides the underlying device-specific code.

• Duplication of code in tests and applications to initialize the network
  device. See examples/wifinina/tcpclient and examples/rtl8720dn/tcpclient.
  Same test, but with device-specific code for device initialization. There
  should be a single tcpclient test which uses "net" TCPConn.

• Duplication of code in tests and applications to connect the network.
  Again, see examples/wifinina/tcpclient and examples/rtl8720dn/tcpclient.
  Same test, but with device-specific code for connecting to AP.

Introducing Netdev Driver Model

A netdev driver implements the Netdever interface:

// A Netdever is a network device driver for Tinygo; Tinygo's network device
// driver model.
type Netdever interface {

        // Probe initializes the network device and maintains the connection to
        // the network.  For example, Probe will maintain the connection to the
        // Wifi access point for a Wifi network device.
        Probe() error

        // GetHostByName returns the IP address of either a hostname or IPv4
        // address in standard dot notation.
        GetHostByName(name string) (IP, error)

        // Socketer is Berkely Sockets-like interface
        Socketer
}

Which includes a Socketer interface:

// Berkely Sockets-like interface.  See man page for socket(2), etc.
type Socketer interface {
        Socket(family AddressFamily, sockType SockType, protocol Protocol) (Sockfd, error)
        Bind(sockfd Sockfd, myaddr SockAddr) error
        Connect(sockfd Sockfd, servaddr SockAddr) error
        Listen(sockfd Sockfd, backlog int) error
        Accept(sockfd Sockfd, peer SockAddr) error
        Send(sockfd Sockfd, buff []byte, flags SockFlags, timeout time.Duration) (int, error)
        SendTo(sockfd Sockfd, buff []byte, flags SockFlags, to SockAddr,
                timeout time.Duration) (int, error)
        Recv(sockfd Sockfd, buff []byte, flags SockFlags, timeout time.Duration) (int, error)
        RecvFrom(sockfd Sockfd, buff []byte, flags SockFlags, from SockAddr,
                timeout time.Duration) (int, error)
        Close(sockfd Sockfd) error
        SetSockOpt(sockfd Sockfd, level SockOptLevel, opt SockOpt, value any) error
}

Higher protocols, applications and tests that want to use Go "net" package import the
standard "net" package and silently the "netdev" package:

import (
        "net"
        _ "tinygo.org/x/drivers/netdev"
)

Network devices move to tinygo-drivers/netdev:

netdev/
├── netdev.go
├── netdev_wifinina.go
├── netdev_rtl8720dn.go
├── wifinina
│   ├── wifinina.go
│   └── (more)
└── rtl8720dn
    ├── rtl8720dn.go
    └── (more)

The netdev driver will register with a call to UseNetdev() in init() to set
the active netdev. Which network device is registered depends on Tinygo build
tags. See netdev/netdev_wifinina.go, for example, which registers wifinina only
for these boards:

// +build: pyportal nano_rp2040 arduino_nano33 metro_m4_airlift arduino_mkrwifi1010 matrixportal_m4

Tests move to tinygo-drivers/examples/net. Note that the tests are agnostic of the network device.

examples/net/
├── mqttclient
│   └── main.go
├── tcpclient
│   └── main.go   // this is the only test I have working so far, with wifinina
├── webclient
│   └── main.go
└── webserver
    └── main.go

Finally, netdev settings such as Wifi SSID, passphrase are passed in with tinygo -ldflags option:

tinygo flash -target pyportal -ldflags '-X "tinygo.org/x/drivers/netdev.ssid=xxxx" -X "tinygo.org/x/drivers/netdev.pass=xxxxxxxx"' examples/net/tcpclient/main.go

These settings are at the netdev level, and not the individual netdev driver (wifinina) level. So the command line above could target wioterminal by changing only the -target. The test to compile and the settings passed in don't change.

[soypat]

Thank you so much for the work you put into this! I'm very excited about where this leads tinygo in the future.

A few comments:

net API change

It seems wise to minimize the surface change in net's exported API. The exported Netdev interface can go under tinygo-org/drivers.

Personally I would leave the exposed surface at this:

// Mirrors tinygo drivers.Netdev. Unexported to prevent use outside of tinygo
type netdev interface {
   // drivers.Netdev methods
}

// UseNetdev sets the currently active network device for use with tinygo.
func UseNetdev(n netdev) {
	netdev = n
}

Reasons for this:

0. packages that use Netdev can be compiled with native Go compiler. This may open doors in the future to a whole new area of Go development in embedded systems.
1. Netdev is a driver interface. The drivers package is perfectly well suited for it.
2. Not a fan of stutter in net.Netdever name. drivers.Netdev, drivers.Socket is not that bad 👀

How to develop for lower level hardware?

Say I wanted to use a lower level Ethernet hardware like the ENC28J60. This IC requires the user to build packets from scratch basically, from the Ethernet to the TCP frame. I'm worried that the proposed API will be too high level for these devices. Back when I developed the driver for the ENC28J60 I came up with the following drivers interface:

// Datagrammer represents a reader/writer of data packets received over stream.
// These packets are low level representations of what could be an Ethernet/IP/TCP transaction.
// An example of an IC which implements this is the ENC28J60.
type Datagrammer interface {
	PacketWriter
	PacketReader
}

// Packet represents a handle to a packet in an underlying stream of data.
type Packet interface {
	io.Reader
	// Discard discards packet data. Reader is terminated as well.
	// If reader already terminated then it should have no effect.
	Discard() error
}

// PacketReader returns a handle to a packet. Ideally there should be no more
// than one active handle at a time.
type PacketReader interface {
	// Returns a Reader that reads from the next packet.
	NextPacket(deadline time.Time) (Packet, error)
}

// PacketWriter handles writes to buffer. Writes are not sent over stream until Flush is called.
type PacketWriter interface {
	io.Writer
	// Flush writes buffer to the underlying stream.
	Flush() error
}

Why compiler flags?

Why can't people simply start their main program with net.UseNetdev?

[scottfeldman]

Thank you so much for the work you put into this! I'm very excited about where this leads tinygo in the future.

You're welcome and thank you for the feedback. I have an itch, and it's not netdev, but I need something like netdev to give me a device-independent, robust networking stack.

net API change

It seems wise to minimize the surface change in net's exported API. The exported Netdev interface can go under tinygo-org/drivers.

Personally I would leave the exposed surface at this:

// Mirrors tinygo drivers.Netdev. Unexported to prevent use outside of tinygo
type netdev interface {
   // drivers.Netdev methods
}

// UseNetdev sets the currently active network device for use with tinygo.
func UseNetdev(n netdev) {
	netdev = n
}

Yes, less exposure good. I'm not following how the mirroring works. Does this require putting a dependency in tinygo on tinygo/drivers? Maybe you can flesh it out a bit to help me?

I think I'm going to learn some new tricks...at first I put netdev over in tinygo itself thinking the network device was kind of part of the machine. Wrong approach, didn't feel right.

How to develop for lower level hardware?

Say I wanted to use a lower level Ethernet hardware like the ENC28J60. This IC requires the user to build packets from scratch basically, from the Ethernet to the TCP frame. I'm worried that the proposed API will be too high level for these devices.

Right, that's the kind of device I'm used to dealing with personally, at least in a full OS stack setting. These embedded devices with the fw exposing a full TCP/IP stack interface is new to me.

With the proposed socket API, the ENC28J60 netdev driver would have to include a TCP/IP/Eth stack. So "upper edge" is socket API, the "lower edge" is the HW FIFOs.

Back when I developed the driver for the ENC28J60 I came up with the following drivers interface:

Right, I see. This would be an alternative to the sockets API? As long as there is a mapping between TCPConn and UDPConn to the API, it doesn't matter the API. The sockets API mapped really well with TCPConn, from what I've seen so far. I guess that should be expected since the real Go TCPConn sits on top of socket syscalls.

Did you have a mapping from TCPConn to Datagrammer?

Why compiler flags?

Why can't people simply start their main program with net.UseNetdev?

I guess they could, but then you need to name the particular netdev to use (wifinina, etc). That's something I was trying to get away from: the app having a dependency on a particular netdev.

[soypat]

I'm not following how the mirroring works. Does this require putting a dependency in tinygo on tinygo/drivers?

As of your proposal now tinygo/drivers will import tinygo-org/net package and this import will break native Go programs since golang/go/net does not have a Netdev type. This is why I suggest moving the Netdev and Socket types into tinygo-org/drivers/socket.go. This will make them usable and testable from native Go programs, which I believe is invaluable for the tinygo and Go ecosystem.

As for tinygo-org/net, it would then be left with only the UseNetdev function, which is the bare minimum needed to link hardware with net.

The problem here is what type does UseNetdev take as an argument? It can't take a drivers.Netdev since this would cause a circular dependency between the tinygo-org/tinygo and tinygo-org/drivers packages. If we try to define Socketer in the drivers package and define a mirrored, unexported Socketer type in net we run into a similar problem:

// type alias mirrors github.com/tinygo-org/drivers Socketer interface
type socketer = interface {
	Socket(family drivers.AddressFamily, sockType drivers.SockType, protocol drivers.Protocol) (drivers.Sockfd, error)
        // ... more methods...
}

We still have to import drivers due to the types the interface methods take! What a conundrum! It would seem as though leaving the types in the net package is the only way forward, but there is an elegant solution to all this...

Create a new tinygo-org level packge for interfaces. @aykevl @deadprogram

This new package would solve the aforementioned problem, lets call it tinyio for tiny-io. It would contain the top-level contents of the drivers package. types like I2C, SPI, UART, Socketer and Netdev among others. This package will contain no imports to derisk the possibility of another split. It will be a self-contained assortment of interfaces that define the tinygo ecosystem, and hopefully the embedded native Go ecosystem in the future. As this package contains no imports it will be importable into any project. This would allow tinygo to import tinyio and the drivers package to import it as well, solving the cyclic dependency problem.

This would also solve the issue with of mirroring interfaces. Since now all packages have a single source of truth of interfaces.

With the proposed socket API, the ENC28J60 netdev driver would have to include a TCP/IP/Eth stack. So "upper edge" is socket API, the "lower edge" is the HW FIFOs.

Sounds reasonable!

Did you have a mapping from TCPConn to Datagrammer?

Nope, rolled my own TCP stack in github.com/soypat/ether-swtch. Not very pleasant, but alas was the first shot I got at networking protocols.

I guess they could, but then you need to name the particular netdev to use (wifinina, etc). That's something I was trying to get away from: the app having a dependency on a particular netdev.

Aha! Yeah, I got it now. Its a boon to microcontroller cross compilation!

[bgould]

I guess they could, but then you need to name the particular netdev to use (wifinina, etc). That's something I was trying to get away from: the app having a dependency on a particular netdev.

Aha! Yeah, I got it now. Its a boon to microcontroller cross compilation!

I kind of prefer the explicit "mounting" of a network device in Go code rather than using build tags for at least 2 reasons:

• Even if a board has some built in network peripheral ... maybe I want to use a custom one, or wrap the device driver in some Go interface to modify its behavior, etc.
• The flip side of that also poses a problem ... what if I want to use wifinina on some custom board that is not in TinyGo ... if it is based on build tags, I don't think I would be able to do that very easily.

The point about user-friendliness of having to specifically initialize hardware is valid ... however in TinyGo that kind of already comes with the territory so I don't think it would be too bad. For the syntactic sugar of automatically picking the right device for the board based on build tags that you mentioned ... I think in theory it would be simple enough to maintain that functionality in a separate package that users could opt into, perhaps by doing import _ "tinygo.org/x/drivers/netdev/auto" or something similar.

[bgould]

Create a new tinygo-org level packge for interfaces. @aykevl @deadprogram

This new package would solve the aforementioned problem, lets call it tinyio for tiny-io. It would contain the top-level contents of the drivers package. types like I2C, SPI, UART, Socketer and Netdev among others. This package will contain no imports to derisk the possibility of another split. It will be a self-contained assortment of interfaces that define the tinygo ecosystem, and hopefully the embedded native Go ecosystem in the future. As this package contains no imports it will be importable into any project. This would allow tinygo to import tinyio and the drivers package to import it as well, solving the cyclic dependency problem.

Providing a set of interfaces like this in TinyGo itself in a package that is importable in standard Go programs as well might be a worthy goal IMO.

[scottfeldman]

I kind of prefer the explicit "mounting" of a network device in Go code rather than using build tags for at least 2 reasons:

Thanks for the feedback. I think there are solutions to these 2 issues you raise:

• Even if a board has some built in network peripheral ... maybe I want to use a custom one, or wrap the device driver in some Go interface to modify its behavior, etc.

For this case, import "net" but not "tinygo.org/x/drivers/netdev". Now, call net.UseNetdev(), passing in the custom netdev before using anything from "net". The custom netdev could wrap a stock netdev, modifying it's behavior, if desired, or be a completely new custom driver.

import (
    "net"
    "mynetdev"
)

func main() {
    net.UseNetdev(mynetdev.New(...))
    // continue with net.Dial(), etc
}

• The flip side of that also poses a problem ... what if I want to use wifinina on some custom board that is not in TinyGo ... if it is based on build tags, I don't think I would be able to do that very easily.

There is a solution: kind of same as above, import "net", but don't include "tinygo.org/s/drivers/netdev". Do import "tinygo.org/x/drivers/netdev/wifinina", and call net.UseNetdev:

import (
    "net"
    "tinygo.org/x/drivers/netdev/wifinina"
)

func main() {
    net.UseNetdev(wifinina.New("myssid", "mypass"))
    // continue with net.Dial(), etc
}

Just for completeness, the suggested default mode is to let the build tags select the correct netdev:

import (
    "net"
    _ "tinygo.org/x/drivers/netdev"
)

func main() {
    // continue with net.Dial(), etc
}

[scottfeldman]

Create a new tinygo-org level packge for interfaces. @aykevl @deadprogram

This new package would solve the aforementioned problem, lets call it tinyio for tiny-io. It would contain the top-level contents of the drivers package. types like I2C, SPI, UART, Socketer and Netdev among others. This package will contain no imports to derisk the possibility of another split. It will be a self-contained assortment of interfaces that define the tinygo ecosystem, and hopefully the embedded native Go ecosystem in the future. As this package contains no imports it will be importable into any project. This would allow tinygo to import tinyio and the drivers package to import it as well, solving the cyclic dependency problem.

This is more ambitious than what I've proposed with netdev RFC, but something to think about. I'll defer to higher pay-grades.

[soypat]

So after some thought and use I've got some opinions:

Opinions

The API is too abstracted in my opinion. I was trying to get the HTTP server example to work and I ended up in what seemed like a dead end at a glance. My program used net.Connect to connect to a network. If my device was not succesfully connecting to the network or something was wrong the only thing that seems exposed by the current API is the error from the Connect function. This seems problematic to me. There seems to be too much magic at work here.

Having net.Connect seems off. Original net package does not have Connect. It is not net's repsonsability to connect to the network, net handles connections over an already existing established network.

The aims of this proposal seem too ambitious. My understanding is that his proposal aims to provide Gophers with a way to almost directly port main package Go programs to TinyGo. Although this is a noble goal and would be nice to have I feel it may lead us into bad design for embedded systems. I plan to use this package in a professional setting. I enjoy abstractions when they do not leave me with less control. I feel this proposal as it stands today makes it harder to understand what is going on under the hood of the microcontroller.

This is not only a problem for experienced embedded engineers, but also novices. I mentioned earlier I was having trouble understanding what exactly happens at net.Connect since there is no state other than the error. I've no idea how to acquire my IP address, MAC, or anything else for that matter by simply looking at this code. This is why I propose we take a step back and first expose more internals of netdev before jumping on the high level abstraction train. I think this would allow us all to use the netdev package and understand it and take a better decision in the future on how to reach perfect Go<->TinyGo portability with packages that use net.

Proposed changes

My proposed changes to the actual implementationare then summarized as:

• If at all possible do not export package-level types, variables, functions in the net package that are not part of net package
• Ideally we should use only a netdev.SetNetdev function to interface with net. This would solve a couple problems I see with this proposal:
  • Avoid exposing parts of net package that do not exist in upstream Go. TinyGo programs are also correct Go programs!
  • The magical import _ "tinygo.org/x/drivers/netdev" becomes plain old "tinygo.org/x/drivers/netdev".
  • If we need an exposed function in the net package we should not use it in examples or programs, but rather the netdever package should wrap it.
• I think there should be a netdev.Netdev global variable that contains the microcontroller's internet peripheral. This would go a long way to making the netdever seem like the central piece to all of this, which it very much is!
• Can we add a netdev.SetSSID(ssid,password string) function that sets the SSID and password in the netdever package?

Great work Scott, this is looking really promising! I am ecstatic about where this goes!

Again, @aykevl @deadprogram : It'd be great to talk about tinygo-org/tinyio package from my RFC? It would go a long way to aiding a reasonable API design for netdev package and all drivers packages

[scottfeldman]

@soypat Thank you Patricio for the feedback, I appreciate it. I think you're right about the API being too abstract and too controlling; I see your points. I'll look at incorporating your proposals into my next version. I just read your RFC and I think I follow. I'll defer to you and others on APIs as that's not really my forte. The bulk of the work is in the driver itself and the integration with "net"; neither which should change if the APIs change. The anchoring API for netdev is the sockets API, so there is some safety in that.

[scottfeldman]

@soypat Hi Patricio!

Ok, I've made some updates and I'd like you to take a look if you would please. Only net.SetNetdev function is added to net package. Everything else moved to drivers/netdev. The examples import wifinina directly and call net.UseNetdev(wifinina.New()). No more magic import. I'm not sure what to do about passing in ssid/pass...see how the wifinina.Config feels to you. It's wifinina-specific, but I think that's not an issue. Each netdev would have its own bespoke Config. I'm not sure I understand your netdev.Netdev global variable. Thanks again for your help.

[soypat]

Alright, I might get around to this during the weekend or next week!

[sago35]

@scottfeldman
I have checked some examples on wioterminal+rtl8720dn.
It is working very well.
Excellent.

I hope to be able to release it at the timing of TinyGo 0.28 (not 0.27).

[deadprogram]

Any comments on how golang/go@18e17e2 impacts this proposal?

[deadprogram]

Also for that matter any new commits to https://github.com/golang/go/commits/release-branch.go1.21/src/net

cc @scottfeldman @soypat @bgould @sago35

[scottfeldman]

Any comments on how golang/go@18e17e2 impacts this proposal?

No impact; the patch doesn't touch any files included with tinygo-net.

[scottfeldman]

Also for that matter any new commits to https://github.com/golang/go/commits/release-branch.go1.21/src/net

Working on it...

[deadprogram]

Closing as completed in the most recent release. Thank you!