Failed to find any network interface to listen on

[norg]

Hi,

I played around with bandwhich a bit and stumpled upon the issue, that on some interface it prints:

sudo bandwhich -i enp216s0f1
Error: Failed to find any network interface to listen on.

While other interfaces on this system work. The main difference between those interface is that the working one is the onboard NIC and the other one is a dedicated Intel XL710 card with 10GE SFPs connected and is using i40e driver which is also the same as the onboard NIC is using.
Other tools like iftop (which I want to replace with bandwhich), tcpdump etc. work on that interface. I also don't see any other error messages. In dmesg I just see the missing output that the interface went into promiscuous mode.

I also made sure the interface is up. I also tested in on some other machines with dedicated NICs, where I could also see it happen on normal 1GE cooper cards with e1000e driver. But I also had one system where it worked with a dedicated NIC. So I'm a bit lost on how to narrow it down. Any hint would be nice as bandwhich looks like a very nice replacement for the old and slow iftop :)

The main test system is a debian stretch with kernel 4.19.

[imsnif]

Hi @norg, thanks for reporting this - and sorry for the trouble.
We unfortunately swallow the exact error we get when we fail to listen on an interface (there's an on-going PR to address this at the moment). Would you be willing to help us troubleshoot this?

What I'd ask you to do first is to clone this repository and build+run it locally (if you are not familiar with rust, there are instructions on how to do this here: https://github.com/imsnif/bandwhich#contributing).

Then, I'd like you to add this debugging line: println!("iface_error: {:?}", iface_error.kind()); above this line: https://github.com/imsnif/bandwhich/blob/master/src/os/shared.rs#L126

Would you be comfortable doing that?

[norg]

I tested it and it doesn't even reach the point where you wanted me to add the additional print statement. I added additional ones and while the test print statement after if available_network_frames.is_empty() is seen the one below for (_, iface) in network_frames doesn't show up.

I'm not that familiar with Rust, but if the is_empty() condition is true (which explains why I see the first println statement and the bail afterwards), is there even a chance to use a for loop on network_frames if there aren't any?
This doesn't explain the actual issue but at least why I don't see the later print statements.

I can also reproduce it when I use -i on my local wireless interface.

Thanks for the fast reply!

[imsnif]

Right, sorry - my bad! We recently moved stuff around. :)

How about if instead of these lines: https://github.com/imsnif/bandwhich/blob/master/src/os/shared.rs#L41-L48

you do this:

    match datalink::channel(interface, config) {                                                                                                                                                           
        Ok(Ethernet(_tx, rx)) => Ok(rx),
        Ok(_) => {
            println!("unsupported type?!");
            Err(std::io::Error::new(
                ErrorKind::Other,
                "Unsupported interface type",
            ))  
        },  
        Err(e) => {
            println!("e {:?}", e); 
            Err(e)
        }   
    }   

[norg]

I added those and some more print statements, I can see that in get_input the match get_interface is correct, as I did:

Some(interface) => {
    println!("what {}", interface);
    vec![interface]
}

I added a simple print before and after let network_frames = network_interfaces... to make sure it's also passed. But within get_datalink_channel none of the print statements shows anything.
Even those before and after those lines in the get_datalink_channel function don't show up:

fn get_datalink_channel(
    interface: &NetworkInterface,
) -> Result<Box<dyn DataLinkReceiver>, std::io::Error> {
    println!("enter get_datalink_channel!");
    let mut config = Config::default();
    config.read_timeout = Some(time::Duration::new(1, 0));
    println!("got config in get_datalink_channel!");

    match datalink::channel(interface, config) {
        Ok(Ethernet(_tx, rx)) => Ok(rx),
        Ok(_) => {
            println!("unsupported type?!");
            Err(std::io::Error::new(
                ErrorKind::Other,
                "Unsupported interface type",
            ))
        },
        Err(e) => {
            println!("e {:?}", e);
            Err(e)
        }
    }
}

[imsnif]

So just to reiterate - nothing inside get_datalink_channel is printed? Seems like we don't even get to this function?

Before we get to this function, we filter the interfaces by interfaces that are up and interfaces that have an IP address. Maybe something goes wrong there.

Could you swap this line: https://github.com/imsnif/bandwhich/blob/master/src/os/shared.rs#L103

with:

        .filter(|iface| {
            println!("interface: {:?} is_up {:?}, ips {:?}", iface.name, iface.is_up(), iface.ips);
            iface.is_up() && !iface.ips.is_empty()
        })

And see what you get?

(also - thanks for the debugging help! This is really helpful)

[norg]

Before we get to this function, we filter the interfaces by interfaces that are up and interfaces that have an IP address.

This is the reason for my mistake :)
The interfaces I want to watch don't have an IP as they are just monitoring traffic.
So I have interface: "enp216s0f1" is up true, ips [] which explains why it's not processed further.

So this is not a pure bug IMHO. It might be an idea to add another error output like found no configured ip and in addition it might be a nice feature to add an option to force listening on such interfaces.

[imsnif]

Aha! Glad we got to the bottom of this. :)

Also - this is totally not your mistake. I don't think it's reasonable for a user to know this! We should definitely add a more descriptive error message here. I'll open an issue for it.

Force listening on such an interface might be a problem, since we use the interface's IP to classify traffic to processes (eg. we can't know what interface firefox is using without knowing the interface's IP, since that's the info we get from /proc or lsof).

I think I encountered a similar issue (with bond interfaces) recently that might be coming from the same root cause. I'd definitely like to address this in one way or another: could you tell me a little bit more about your setup so that I can understand the use case for such interfaces better?

[norg]

It would be an idea to disable other features (like those with process related traffic) if you want to just listen to what traffic is on an interface. In that scenario I'm mostly interested in Utilization by connection.

The use case in detail is that you have a switch (or more advanced packetbroker, tap device, span port) with a mirror port and you attach this to the system where you want to analyze all the traffic that is mirrored from the switch. In such scenarios you use monitoring tools, like an IDS system (Suricata for example), but tcpdump, wireshark etc. work as well.
So you see traffic and the interface is passive as it's only receiving and not sending any traffic.
Tools like iftop help to find some specific high traffic flows, but I had some issues with the performance of iftop thus I tried bandwhich.
Another sidenote, that also works if the interface is shown as DOWN. It can still receive such monitoring traffic. A typical example is fiber connection where you only connect the RX cable and not the TX part.

[imsnif]

@norg - that makes sense. Thanks for your help! Would you be able to work around this meanwhile by listening to the mirrored interfaces instead?

@zhangxp1998 - I have some ideas about how to address this, but I'd very much like to hear your thoughts first. What say you?

[norg]

@norg - that makes sense. Thanks for your help! Would you be able to work around this meanwhile by listening to the mirrored interfaces instead?

That's the interface I want to run bandwhich on :) So the cable leaves the mirror port from the switch and is connected to the interface enp216s0f1 on the linux system which is the only place I would see the traffic I want to check.

But it's not urgent, just me playing around with alternatives to iftop. So if this will be supported some day that would be nice as bandwhich seems to have better performance.

[zhangxp1998]

@norg - that makes sense. Thanks for your help! Would you be able to work around this meanwhile by listening to the mirrored interfaces instead?

@zhangxp1998 - I have some ideas about how to address this, but I'd very much like to hear your thoughts first. What say you?

Originally, I filter out interfaces that are not up or don't have IP to save CPU usage. As these interfaces probably won't have any packets going through. Since we create a separate to probe each interface, filtering out some useless interfaces can save quite some resources. However, some of the assumptions I made are clearly false now. Mirroring interfaces(or potentially bond/bridge interfaces) might not be up(or have IP), but they can still transmit packet. Maybe we could just remove the interface filter? Or have an option that allow the user to override it.

[imsnif]

Maybe we could just remove the interface filter? Or have an option that allow the user to override it.

I started following the code path of doing that and the place I see some issues is where we classify the packets in the network sniffer. If the interface doesn't have an IP, we won't know if this is an incoming or outgoing packet. Maybe for such cases we can cross-reference the bound connections to find a match?

[zhangxp1998]

Maybe we could just remove the interface filter? Or have an option that allow the user to override it.

I started following the code path of doing that and the place I see some issues is where we classify the packets in the network sniffer. If the interface doesn't have an IP, we won't know if this is an incoming or outgoing packet. Maybe for such cases we can cross-reference the bound connections to find a match?

We could also use MAC addresses? Sounds simpler to me. Not sure if the special interfaces mentioned above have MAC addresses. @norg What say you?

[imsnif]

Also, actually - coming to think about it - nothing from /proc or lsof is relevant here... so definitely MAC addresses. Joining your question about whether they are also mirrored or not then.

[imsnif]

I've been reading a little more about promiscuous mode, and I think even the MAC address solution won't work for us in this case. Seems like this mode is mostly intended for packet sniffing of another interfaces, just like @norg is doing.

IMO - the best thing we can do here is add a feature called "promiscuous mode". We would essentially be starting bandwhich with a --promiscuous flag which would essentially just show the connections table, but in it show both the source IP and destination IP (forgoing the network interface, as it's likely less relevant for this use case).

The upshot of this would be that this would be a great infrastructure for us to support piping .pcap captured files into the app in the future - and then allowing users to walk through them in the terminal rather than opening them in wireshark (if they're only interested in connection/bandwidth information).

@norg - would such a feature serve your purpose? Would you like anything else there?
@zhangxp1998 - what do you think of this idea? Maybe you have another one?

[norg]

This sound close to what I would expect. In such scenarios you're mostly interested in the network flows and how much bandwidth they use. I don't know how iftop does this but you can still distinguish between the different flow directions. So you have two lines IP A -> IP B and IP B -> IP A and the amount of traffic for each direction of the flow. But you don't need any infos about the local interface and it does nothing except sniffing the traffic.

[imsnif]

So you have two lines IP A -> IP B and IP B -> IP A and the amount of traffic for each direction of the flow. But you don't need any infos about the local interface and it does nothing except sniffing the traffic.

I think in our case, you would see this as 1 line and get both the up and down traffic. We would just randomly decide which IP to put at the beginning of the line, since in that mode we would have no concept of "local" or "remote" ips. Would that make sense?

[norg]

Of course, since there is no real local/remote part in that scenario it's not relevant. If IP A => IP B has UP 1Gbit/s Down 100Mbit/s it's the same as the output line IP B => IP A with UP 100Mbit/s Down 1Gbit/s in such a scenario.

[imsnif]

Awesome. I'll make an issue tonight and tag you so you let us know your thoughts (if you want to). If someone doesn't pick this up, I'll likely get to it myself sometime in the next few weeks.

[imsnif]

I'm closing this issue in favor of: #141
@norg, @zhangxp1998 - if you have anything to add to the other issue, that would be great.

[Dentrax]

I just installed from brew install bandwhich and get this error. Thats why it brings me here. :(

[imsnif]

@Dentrax - when running with sudo as well?

[zhangxp1998]

@Dentrax - when running with sudo as well?

Maybe we should print underlying errors instead of swallow them?

[Dentrax]

I haven't sudo privileges because of this is not my PC. It my company's.

[imsnif]

@zhangxp1998 - agreed completely. There's an ongoing PR for this: #104

@Dentrax - ah, sorry friend. The tool requires elevated privileges to run.

[zhangxp1998]

I haven't sudo privileges because of this is not my PC. It my company's.

As far as I know, on a Mac, you need sudo privilege to run bandwhich

[Dentrax]

Oh... :( Maybe you think about add the sudo information on the README file. :) In addition, you can throw a sudo exception message instead of "Failed to find..." information. This would be more clear. :)

[zhangxp1998]

Oh... :( Maybe you think about add the sudo information on the README file. :) In addition, you can throw a sudo exception message instead of "Failed to find..." information. This would be more clear. :)

We did mention sudo in README, and we will try to address the error message "Failed to find...". That could certainly be made better. Thanks for your suggestion!