Including unmanaged devices in lab topology

[ssasso]

Discussed in #538

^{Originally posted by ssasso October 4, 2022}
I have a setup when I want netlab to take care of some network gear equipment config (using the external provider), but at the same time I need to establish peerings with their counterparts which should not be configured by netlab.
(i.e. configure a BGP session on a device controlled by netlab)

I was thinking to use the none device type for that kind of stuff, with sth like:

vrfs:
  RED:
  BLUE:

vlans:
  site1red:
    mode: route
    id: 41
    vrf: RED
    bgp: false
  site1blue:
    mode: route
    id: 61
    vrf: BLUE

module: [ vlan,vrf,ospf,bgp,mpls ]
bgp.as: 65000

nodes:
  pe1:
  site1a:
    device: none
    module: [ vlan,vrf,bgp ]
    bgp.as: 65101
  site1b:
    device: none
    module: [ vlan,vrf,bgp ]
    bgp.as: 65101

links:
- pe1:
    ifname: eth3
    vlan.trunk:
      site1red.ipv4: 172.24.111.33/28
      site1blue.ipv4: 10.211.211.1/28
  site1a:
    vlan.trunk:
      site1red.ipv4: 172.24.111.34/28
      site1blue.ipv4: 10.211.211.11/28
  site1b:
    vlan.trunk:
      site1red.ipv4: 172.24.111.35/28
      site1blue.ipv4: 10.211.211.12/28
  vlan.trunk: [ site1red, site1blue ]

(for this specific example, I need pe1 to have two configured bgp sessions towards 10.211.211.11 and 10.211.211.12.)

But unfortunately the none device does not support vlan, vrf, and bgp.

Do you think it would be a good idea to allow the none device type to support all modules,
or should I use a real platform on all the devices, and the rely on a group of configurable devices to limit the ansible config with -l group?

[ipspace]

It looks like the best way to handle this would be to have a special devices.features flag (maybe ghost: True) and remove nodes with that device feature from final list of nodes (or some such trick), so they don't get into Ansible inventory or Vagrantfile/clab.yml.

A one-stop solution might be to remove "ghost" devices at the end of data transformation process, unless we want to see them in graphs and Graphite, in which case we have to clean up the nodes dictionary in inventory and provider output modules. Should we go for the latter option, I'd recommend adding a shared routine to augment/nodes.py (how about "ghost_buster") and calling it from every provider module that needs this functionality.

Just keep in mind that a user might chain a bunch of output modules in sequence, so we have to make sure the original topology data is intact for the next module. Not sure whether we already make a copy of the data before doing anything with it in provider and inventory output modules -- if not, the "ghost_buster" routine could make a copy of the topology with Box(topology,...parameters...) and then work on that.

[ssasso]

I'm starting to work on this.

Shouln't be better, at this point, to simply have a parameter like ghost: true (or unmanaged: true) at node level, instead of having it as device features (or creating a new device type)?

At the end of the day, any kind of device should be able to support ghost mode.

I'll create a branch and start working on some code, so we can see how it can be.

[ssasso]

Preview: ssasso@1ac2f3e

Note: it could be great to add a test case for that with the "expected" topology output. But this means I need to modify also the YAML output provider, since its output is used for the test itself. Could that be ok for you?

Note 2: I tested it with both libvirt and clab providers.

[ipspace]

Looks good to me (maybe we should use "unmanaged" attribute while talking about ghosts in behind the scenes ;).

As for a test case: of course it would be nice to be able to test this functionality. Tests don't use output provider (at least the ones working with topology/[input,expected]) but create YAML output in utils.py:transformation_results_yaml. We could add a flag into topology defaults that would be checked by that module. I could do that after you're done with the interesting parts of this project ;) -- just create a test case or two and I'll make sure the correct thing happens in the testing process.

[ipspace]

Looks good to me

Should have said "using node attributes makes more sense than a weird device type and allows us to do all sorts of interesting stuff in the future" ;) I'm coming down with fever, so maybe I should stop typing...

[ipspace]

Implemented in ed38601