ACL logs with MicroOVN do not work because of hard coded file path

[escabo]

Required information

• Found with a MicroCloud setup using LXD 5.20

Issue description

• The lxc network acl show-log bla command always produces this error: Error: Only OVN log entries may be retrieved at this time

Seems like this line https://github.com/canonical/lxd/blob/main/lxd/network/acl/driver_common.go#L754 should be snap aware as if I manually create a link on the servers of the cluster to the snap location of the file (when using MicroOVN), the error disappears.

[tomponline]

@escabo the HostPath() call in line https://github.com/canonical/lxd/blob/main/lxd/network/acl/driver_common.go#L734 is snap aware.

Was this tested only using a microovn installation or also with a normal Ubuntu based OVN setup (such as the one described here https://documentation.ubuntu.com/lxd/en/latest/howto/network_ovn_setup/#set-up-a-standalone-ovn-network)?

If it works with the standard OVN setup and not with the microovn setup then its likely an issue with the snap packaging in LXD not setting up the correct symlinks internally for the microovn log directory.

[escabo]

This was with MicroOVN (in a MicroCloud setup) and I reported it because when I added symlinks, the error message disappeared.

[tomponline]

@masnax want to take a look at this one given you've done the work around microovn content interfaces previously?

[simondeziel]

I think LXD and MicroOVN snaps should have a plug/interface to allow for easy retrieval of OVN logs by LXD. On the MicroOVN side, all I could find is for local log files.

We currently have a couple of interfaces/connections between the 2:

# snap connections lxd | grep ovn
content[ovn-certificates]  lxd:ovn-certificates  microovn:ovn-certificates  -
content[ovn-chassis]       lxd:ovn-chassis       microovn:ovn-chassis       -

# snap connections microovn | grep lxd
content[ovn-certificates]  lxd:ovn-certificates          microovn:ovn-certificates  -
content[ovn-chassis]       lxd:ovn-chassis               microovn:ovn-chassis       -

I think an eventual ovn-logs interface should only be manually connected as we would not always want to receive OVN logs. On the LXD side, this should also be conditional to having core.syslog_socket=true.

This would be a joint effort with the MicroOVN team.

[tomponline]

Heres one for you @gabrielmougard

[gabrielmougard]

After looking at it, it seems a bit complex... Since canonical/microovn#94, the OVN controller logs are directly sent to syslogs inside the MicroOVN snap. I don't think using a snap content interface would be ideal in this situation: since each snap has its own /dev/log by default, sharing the syslog socket file between snaps via the content interface seems not practical. Plus, I believe that the philosophy of the content interface is designed to share static files and directories, not active sockets or FIFOs.

Instead, couldn't we just add the log-observe interface in LXD and read the OVN controller syslog like :
journalctl -f -u snap.microovn.ovn-controller.service >> $SNAP_COMMON/logs/ovn-controller.log

I think we'd also need a plug in MicroOVN (I like ovn-logs) with the corresponding slot in LXD so that when the connect hook is fired, we can start this write logic + logrotate logic as a daemon task (ovn-log-stream.start) and create a dummy file like /etc/microovn_ovn_controller_log_stream (to indicate that a log streaming task is active and to tell LXD that it is possible to consume it) and stop it when the disconnect hook is fired (ovn-log-stream.stop) which also remove the /etc/microovn_ovn_controller_log_stream file.

Then, inside LXD:

• We could decide to create a fifo named pipe between this ovn-controller.log and the syslog.socket (defined in StartSyslogListener, if core.syslog_socket=true of course). This would give the advantage of monitoring the snap confined OVN controller logs through lxc monitor --type=ovn
• If core.syslog_socket is not set to true, we'd still have the journalctl stream writing to $SNAP_COMMON/logs/ovn-controller.log which should allow lxc network acl show-log bla to work (we just won't have the OVN controller logs as part of the LXD snap syslogs)

This solution is non-invasive for MicroOVN (litteraly just adding a dummy 'ovn-logs' plug to trigger the connect hook on the LXD side)

However, the snap.microovn.ovn-controller.service is dynamically started by the the MicroOVN daemon, so we'd still need to perform a snapctl check in the LXD connect hook to know if the service has been started.

@tomponline @simondeziel how does it sound?

(Having an ovn-controller.log file persisted in MicroOVN would have been much easier 😅 but I get why they decided to get rid of it..)

[tomponline]

what would be involved with getting microovn to log to the the LXD syslog listener, as per

lxd/doc/howto/network_ovn_setup.md

Line 181 in 5386396

## Send OVN logs to LXD

[tomponline]

@masnax does the microovn interface expose any unix sockets to the ovn DBs btw?

[gabrielmougard]

what would be involved with getting microovn to log to the the LXD syslog listener, as per

lxd/doc/howto/network_ovn_setup.md

Line 181 in 5386396

## Send OVN logs to LXD

We can't really update the OVN controller parameters from outside the snap (unlike setting OVN_CTL_OPTS=" --ovn-controller-log='-vsyslog:info --syslog-method=unix:/var/snap/lxd/common/lxd/syslog.socket'" in the link you mention) . Maybe it is possible to manually edit the systemd service file (generated from commands/chassis.start in MicroOVN) but it seems a bit hacky and could potentially have side effects since this service is already managed by the MicroOVN daemon itself programmatically (see https://github.com/canonical/microovn/blob/main/microovn/snap/snap.go)

[gabrielmougard]

The proposed solution from above would require three interactions:

lxc config set core.syslog=true
snap connect lxd:log-observe
snap connect lxd:ovn-logs microovn:ovn-logs

[tomponline]

We can't really update the OVN controller parameters from outside the snap (unlike setting OVN_CTL_OPTS=" --ovn-controller-log='-vsyslog:info --syslog-method=unix:/var/snap/lxd/common/lxd/syslog.socket'" in the link you mention) . Maybe it is possible to manually edit the systemd service file (generated from commands/chassis.start in MicroOVN) but it seems a bit hacky and could potentially have side effects since this service is already managed by the MicroOVN daemon itself programmatically (see https://github.com/canonical/microovn/blob/main/microovn/snap/snap.go)

Can we update the microovn snap to take an option to send logs to a different unix socket using snap set microovn ..., or have them support an interface LXD can plug into?

[gabrielmougard]

I don't know if the OVN controller can directly output its logs to a UNIX socket. If this is possible, then we could share this socket (which would be LXD's syslog.socket right ?) through a snap content interface. But then will the logs appear in the syslogs of the MicroOVN snap?

[tomponline]

I don't know if the OVN controller can directly output its logs to a UNIX socket. If this is possible, then we could share this socket (which would be LXD's syslog.socket right ?) through a snap content interface. But then will the logs appear in the syslogs of the MicroOVN snap?

It can, see https://documentation.ubuntu.com/lxd/en/latest/howto/network_ovn_setup/#send-ovn-logs-to-lxd

[tomponline]

But then will the logs appear in the syslogs of the MicroOVN snap?

What do you mean?

[gabrielmougard]

If I understood correctly the snap architecture, each snap has its own /dev/log device to contain its syslogs, so if we decide to redirect these OVN controller logs to a websocket LXD is using (syslog.socket), these logs won't be shown in journalctl snap.microovn.ovn-controller.service` but in journalctl snap.lxd.daemon.service right ?

[gabrielmougard]

Plus, this solves half the problem: we still don't have ovn-controller.log in the LXD snap that cause the original issue. We'll need to have a writer thread in StartSyslogListener that output ovn type events to this file only if the MicroOVN snap is connected (if this is builtin OVN, we don't want to do that, hence the idea of creating a /etc/microovn_ovn_controller_log_stream to know in which use case we are)

[tomponline]

@gabrielmougard think we need a meeting about this as im not following you.

[masnax]

@masnax does the microovn interface expose any unix sockets to the ovn DBs btw?

Just the openvswitch socket. NB/SB/controller sockets are not exposed.

[tomponline]

Just the openvswitch socket. NB/SB/controller sockets are not exposed.

Ack. But at least it shows that one can expose sockets over interfaces, which is what I was wondering. Thanks

[tomponline]

OK so after a quick discussion on MM @gabrielmougard and I are considering this approach:

1. MicroOVN's ovn-controller is logging to journalctl already.
2. LXD has host system access and so should be able to access journalctl -u snap.microovn.ovn-controller.service already.
3. We don't want to be redirecting this output to an intermediate text file as this would then require rotation management.
4. Instead we will explore using an invocation of journalctl -u snap.microovn.ovn-controller.service when lxc network acl show-log bla is run in order to retrieve the X (to be defined) most recent controller logs.