AWS ENA driver not reporting all statistics

[AndrewNeudegg]

Description

The ENA driver installed with flatcar does not produce all of the metrics that AWS advertise as being available. The documentation states that metrics like conntrack_allowance_available should be available, but they are not present.

Output of ethtool -S eth0:

ethtool -S eth0
NIC statistics:
     tx_timeout: 0
     suspend: 0
     resume: 0
     wd_expired: 0
     interface_up: 1
     interface_down: 0
     admin_q_pause: 0
     bw_in_allowance_exceeded: 77193
     bw_out_allowance_exceeded: 8487
     pps_allowance_exceeded: 0
     conntrack_allowance_exceeded: 0
     linklocal_allowance_exceeded: 0
..continues..

Output of modinfo ena:

modinfo ena
filename:       /lib/modules/5.15.106-flatcar/kernel/drivers/net/ethernet/amazon/ena/ena.ko.xz
license:        GPL
description:    Elastic Network Adapter (ENA)
author:         Amazon.com, Inc. or its affiliates
alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
retpoline:      Y
intree:         Y
name:           ena
vermagic:       5.15.106-flatcar SMP mod_unload
sig_id:         PKCS#7
signer:
sig_key:
sig_hashalgo:   unknown
signature:

The instance and the AMI supports ENA metrics:

aws ec2 describe-instances --instance-ids i-03caf708273b6aced --query "Reservations[].Instances[].EnaSupport"
[
    true
]

 aws ec2 describe-images --image-id ami-041894d5bf5ede441  --query "Images[].EnaSupport"
[
    true
]

OS Information:

cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3510.2.1
VERSION_ID=3510.2.1
BUILD_ID=2023-04-24-2100
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3510.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3510.2.1:*:*:*:*:*:*:*"

This is likely due to the ENA driver being version older than 2.8.1.

Impact

The impact is that we are unable to export all the metrics that should be available to us limiting our ability to detect and resolve issues.

Environment and steps to reproduce

Flatcar running on an m6i.2xlarge EC2 instance within AWS. When querying ethtool -S eth0 | grep conntrack_allowance_available you should get a result.

Expected behavior

The full set of ENA metrics should be reported.

Additional information

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html#network-performance-metrics
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances

This works as expected on Amazon Linux 2 on the same instance type.

[jepio]

Thanks for reporting this, it looks like the ENA driver in the linux kernel tree is at v2.1.0. I wonder why AWS stopped updating it.

[jepio]

@AndrewNeudegg do you see the same output as Flatcar on Ubuntu on AWS?

[AndrewNeudegg]

Hi @jepio,

The output from Ubuntu:

ethtool -S eth0

root@ip-10-0-130-111:/home/ubuntu# ethtool -S eth0
NIC statistics:
     rx_gso_checksum_fixup: 0

modinfo ena

root@ip-10-0-130-111:/home/ubuntu# modinfo ena
filename:       /lib/modules/5.19.0-1025-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
license:        GPL
description:    Elastic Network Adapter (ENA)
author:         Amazon.com, Inc. or its affiliates
srcversion:     735F4868374051CB9CB7B43
alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:        
retpoline:      Y
intree:         Y
name:           ena
vermagic:       5.19.0-1025-aws SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        10:32:81:35:71:B2:A0:27:4A:05:FA:A8:CE:01:38:62:B4:DC:59:A4
sig_hashalgo:   sha512
signature:      26:0A:DE:FA:72:EA:13:FE:24:9E:96:98:C3:3C:AC:B8:06:21:94:50:
                E2:A3:7D:90:56:19:0F:9D:6C:1F:0F:B1:09:B7:5A:60:57:FE:DC:59:
                6D:72:FC:A6:45:F9:2F:41:13:7A:58:34:B1:9E:94:A9:AF:AA:EC:7D:
                68:8D:ED:64:DF:3D:30:F0:1D:CB:AC:BC:67:E3:5D:98:31:07:48:D5:
                38:90:4E:A7:25:D5:D0:2E:DD:E5:DE:1A:8D:2E:9B:49:46:11:6D:A5:
                B4:EB:E9:D1:1F:EE:44:78:36:BF:D9:68:7F:0D:8A:8E:3F:A4:32:62:
                EB:C6:88:4E:CE:35:32:22:25:15:91:3D:5F:4C:F2:F0:CF:9A:CE:AD:
                3B:82:2E:89:2E:2A:9F:81:AD:0F:47:F1:DF:DA:F1:B6:0F:FA:AD:6B:
                7F:93:C2:AE:C7:16:D6:80:25:FB:00:1A:99:0F:A1:24:9E:1E:CF:1B:
                4A:D6:CB:3A:29:BE:F3:BF:84:07:64:81:73:70:2F:48:4E:12:1A:F5:
                56:7B:BA:DF:B5:C8:8C:5D:63:A9:11:A2:2C:DE:2B:CB:B7:B2:31:4D:
                B5:54:CF:1E:B9:42:CF:5A:D4:3F:C9:07:23:5D:7B:A3:8A:94:3F:C7:
                3A:32:AC:A4:2C:93:DF:94:D6:97:B0:FA:5D:A7:9E:E5:4E:51:F4:E5:
                D8:AF:4F:26:0A:5A:3B:30:F9:C1:63:67:EB:0F:A4:D0:28:65:81:82:
                55:07:C9:66:E2:50:D3:0D:F3:D5:26:70:9C:0E:B5:71:36:38:D1:6B:
                AE:6F:8A:F0:3C:15:5A:51:7D:AC:52:91:91:1C:07:9C:DD:83:87:EE:
                E5:21:CF:35:BF:59:07:3B:CD:74:AF:F9:23:78:A0:A5:F7:51:37:DE:
                D9:D0:FB:B8:E9:BC:42:14:15:39:E5:65:53:C7:BD:25:21:AC:26:D1:
                B4:46:9D:4D:BC:67:D8:6A:2A:CE:8E:58:7D:8C:84:12:D5:DE:3B:7D:
                19:D5:BC:94:B7:77:AB:68:67:04:66:DB:CF:1C:5E:FC:A4:06:99:B6:
                7C:9A:C6:43:DA:A9:4F:C4:12:40:0F:E8:2C:66:AD:2B:6F:8A:15:A7:
                53:28:18:C2:EC:D8:70:49:19:C7:9A:A6:F8:9C:EF:30:BB:F3:6E:81:
                36:8D:ED:5E:21:EF:EB:BC:B1:BF:45:92:C5:68:26:0C:F1:8E:33:45:
                38:29:AD:1F:6D:9B:42:21:7A:03:9A:3E:E3:B7:5C:7E:23:30:3C:99:
                D4:5E:4D:C6:E9:A0:E9:B8:BC:3E:38:D4:62:7A:6D:98:FF:4B:74:FE:
                58:B8:F2:2B:53:66:81:34:FA:D3:B1:14

OS Information

root@ip-10-0-130-111:/home/ubuntu# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• instance had ena support
• ami had ena support