[WIP] 802.1q (vlan) support, fix some layering issues

[hannesm]

the goal of this PR is to start a discussion about network layering. My initial goal was to add virtual local networks (using 802.1q) support to TCP/IP in order to be able to have a separate access and a management network - taking into account our current constraint on solo5 to only have a single network interface.

There are multiple ways I can think of to organise multiple ethernet layers: one question is whether they should share a common arp layer, another whether each incoming ethernet frame is parsed once per vlan or not. One design choice is as well whether to use a common TCP/IP stack or separate ones -- atm this PR targets the latter solution (though the branchname indicates that I initial planned the former).

Now, this is WIP - the goal is achieved (look at https://gist.github.com/hannesm/be63dd6bfecc9839df46b72bfd70293d), though I modified main.ml manually (still lacking some knowledge of mirage.ml how to properly extend this). This example unikernel configures two vlans, id 23 (hardcoded as ethernet2 in main.ml) with IP address 10.23.42.3/24 on stack s2 (in unikernel.ml), and vlan id 5 (via config.ml). Alluding to the questions above: these TCP/IP stacks are two separate ones (i.e. no routing table, depending on which stack you use for communication it will reach either one or the other vlan). The netif is shared, but already the ethernet (and arp, and ip) layers are distinct. This is IMHO a good setup for a unikernel-service that wants to emit syslog and monitoring to a private network (vlan 23), and provide a public service (vlan 5). Only the log reporter and the metrics reporter actually need access to the vlan 23 stack.

Now, in brief some changes I did

• mirage-protocols is adjusted (at 802.1q (vlan) support mirage-protocols#13):
  • rename ETHIF to ETHERNET (there are still some spurious Ethif leftover)
  • introduce Ethernet.Proto module (providing type t, pp, compare)
  • introduce IP.Proto module (same as above)
  • added register to Ethernet and Ip module signatures register: t -> Proto.t -> callback -> (unit, [ `Conflict ]) result
  • intoduce Ethernet.allocate_frame : ?size -> t -> buffer (and Ethernet.header_size)
  • input could now be removed (but is still used for tests) ;)
• vlan (802.1q) support mirage#951
  • rename Ethif to Ethernet
  • ipv4_keyed_conf receives a ?ip:(Ipaddr.V4.Prefix.t * Ipaddr.V4.t) (the tuple which is the result of Ipaddr.V4.Prefix.of_address_string_exn) instead of separate ip and network, which lead to some churn in static_ipv4.ml
  • the stack used to get network, ethif, arp, ipv4 and built up the stack. it now only gets the ipv4.
  • this is missing a vlan device / object, to act as a user interface for configuring vlans
• in this PR:
  • rename Ethif to Ethernet
  • move the Ethernet encapsulation and decapsulation into the ethernet layer (instead of having it in IPv4 and ARP)
  • call register in Static_ipv4.connect and Arp.connect -- no need for Tcpip_stack_direct to do this
  • fix Arp and IPv4 to only use the Ethernet module, rather than calling directly Ethernet_wire.sizeof_ethernet
  • add tcpip.vlan which implements 802.1q
  • removed users of io-page (now that allocate_frame is exposed)
  • write functions now receive protocol and destination explicitly

Now, the change is rather big, and there are some not-so-nice corners which I'd like to discuss:
(a) before this PR MirageOS stacking works by using connect from bottom to top (i.e. Netif.connect device : Netif.t Lwt.t), which gets some arguments, and allows to construct a layer which is then passed to the layer above (i.e. Ethernet.connect : Netif.t -> Ethernet.t Lwt.t). Processing of incoming frames is done by Netif.listen with a callback to evaluate for each received frame. This callback is a pattern matching of the distinct protocols on the layers, and calling the input functions of each layer (see https://github.com/mirage/mirage-tcpip/blob/v3.5.1/src/stack-direct/tcpip_stack_direct.ml#L91-L124). This function as well calls to (Lwt.async listen netif), which seems to be executed multiple times over the lifetime of a unikernel.
(b) the issue with that is that Tcpip_stack_direct knows all the layers, and since I'd like to replace Ethernet with Vlan, and have multiple stacks, this doesn't feel right (who is supposed to call Netif.listen?).
(c) to deal with this a new function register : t -> proto -> callback -> (unit, [`Conflict]) result is introduced at each layer. registering the same protocol twice leads to a conflict. Each layer calls register on its lower layer(s) during connect -- i.e. this is a top to bottom, different from connect. The implementation uses a mutable map from proto to callback. IMHO this is fine since it is contained mutable state in each layer.
(d) Netif.listen is now called by Ethernet.connect and Vlan.connect (in the end, it should likely be called by Netif in connect -- or is there a use for a unikernel to have a network interface but not listen on it?
(e) so, packets are received once the connect calls are executed, and will be directed to the specific layer

What are the issues?

• finding the handler for a frame requires a lookup in a Map (used to be a direct pattern match), I don't fully understand the performance implications thereof
• the whole layer registration and connection is mainly done at configure time (is it really?), so it feels stupid to have yet another dynamic registration mechanism
• the register uses mutable state, there is no unregister

What is left to do?

• IPv6, I just don't have sufficient knowledge neither a test network to try it out. the changes I did here was only to get it to compile
• remove input functions, and rebase the test cases onto the register protocol
• if the above makes sense, Netif should as well provide a register function (without protocol since Netif doesn't know anything about protocols), and once that is called (someone is interested in netif-frames), a task which calls the listen function should be created (then ethernet and vlan don't need to bother with this).

what do you think? (I'm especially interested in other proposals in respect to "how does an upper layer tell its lower layer which callback to execute for packets that are of interest to the upper layer".)

[talex5]

(b) the issue with that is that Tcpip_stack_direct knows all the layers, and since I'd like to replace Ethernet with Vlan, and have multiple stacks, this doesn't feel right (who is supposed to call Netif.listen?).

I don't understand vlan in much detail, but I would expect:

• Each TCP stack calls listen and input on its Ethernet device (as it does now).
• These Ethernet devices are virtual ones provided by the vlan module.
• vlan makes a single call to listen/input on its underlying network device.

I guess my question is: why can't the vlan module itself handle all this internally? It would mean vlan keeping separate queues of frames for each virtual device, because one tcp stack being slow shouldn't prevent another one from getting data, but I guess we'd want to do that anyway.

[hannesm]

@talex5 thanks for your comment.

Each TCP stack calls listen and input on its Ethernet device (as it does now).

I don't quite follow you here. At the moment, Tcpip_stack_direct calls out to Netif.listen. As mentioned, this will not work for the 802.1q case: there is a single "physical" netif, but multiple ethernet layers on top - i.e. imho there should only be one call to Netif.listen, not every Ethernet layer should do this. This is done in this PR, see the Ethernet.connect and Vlan.connect implementations.

There never was any listen on the Ethernet layer. Now, the Netif interface is listen : t -> callback -> ..., and as implemented here, the callback differs from the Vlan and Ethernet use case.

This PR removes the netif, ethernet, and arp from the Tcpip_stack_direct - since it's hardcoded setup of network layers does not scale to vlans.

I guess my question is: why can't the vlan module itself handle all this internally? It would mean vlan keeping separate queues of frames for each virtual device, because one tcp stack being slow shouldn't prevent another one from getting data, but I guess we'd want to do that anyway.

Uhm? I don't understand your remark. What should all this internally mean? I'm not entirely sure what you mean by "separate queues" either. To recap, in the current setting Netif.listen creates a separate task (using Lwt.async) for each received frame. Does this clarify your comment?

[talex5]

there is a single "physical" netif, but multiple ethernet layers on top

I think perhaps that having the TCP stack take both a Netif and an Ethif was a mistake. It appears that the only thing Tcpip_stack_direct does with the Netif is to call listen (and log some stats when it finishes), which Ethif could do if it provided a listen call. The default implementation could just proxy to Netif.listen, as now.

So, I guess I'm thinking of a system where:

1. You create one physical Netif.
2. You wrap it with vlan, which provides virtual Ethernet devices (one for each vlan ID).
3. You give one virtual eth device to each tcpip stack (but not the netif).

The vlan module would be responsible for dealing with the fact that it will get one listen on each of its virtual devices, but only needs to call listen once on its own network device.

[hannesm]

ok, we're getting closer.

I think perhaps that having the TCP stack take both a Netif and an Ethif was a mistake.

I concur. This PR removes netif, ethif, and arp from stack-direct, leaving the IPv4, TCP, and UDP layers in place.

You give one virtual eth device to each tcpip stack (but not the netif).

Why should the tcpip stack be interested in eth? This limits the tcpip stack to communicating via ethernet, while there are still various other setups deployed (PPPoE, PPP, IP-over-DNS, ...).

It is still unclear why Ethif Ethernet should have a listen function. Is there any useful unikernel that has a network device but does not listen on it?

[hannesm]

NB: I'll split this PR into smaller ones to allow for easier review; the "one Lwt task per received packet" is the current state, I'm not sure whether this is the best solution. I'm also not sure what the best API is on the receiving and decapsulating side -- I read some foxnet papers that function calls / callbacks were too expensive for their stack (but I'll need to re-read them, and their evaluation may not hold for modern CPUs).

The connect and register as proposed here works - but I'm not sure what the performance is (Map lookup vs pattern match for each frame (and their protocol field)), and how possible alternatives look like and what their tradeoffs are. If anyone has some pointers apart from foxnet, I'd be interested to look into some network layer papers. Does the NEC clicks unikernels have such a design document or document what they're actually doing to achieve performance?

[avsm]

I've not had a chance to look at this in detail, but I wanted to point out a good source of design inspiration: https://www.freebsd.org/cgi/man.cgi?query=netgraph&sektion=4

FreeBSD Netgraph provides a dynamically configurable networking stack based on a dataflow connectivity model. They cover a lot of different link layers (incl. Bluetooth and so on), so it might provide guidelines for us to structure the modular stack.

[hannesm]

this is full of conflicts, I separated a big chunk of this diff into #394 (also, arp and ethernet were removed from this repository).