Skip to content

Commit

Permalink
Merge branch 'net-flash-modees-firmware' into main
Browse files Browse the repository at this point in the history
Danielle Ratson says:

====================
Add ability to flash modules' firmware

CMIS compliant modules such as QSFP-DD might be running a firmware that
can be updated in a vendor-neutral way by exchanging messages between
the host and the module as described in section 7.2.2 of revision
4.0 of the CMIS standard.

According to the CMIS standard, the firmware update process is done
using a CDB commands sequence.

CDB (Command Data Block Message Communication) reads and writes are
performed on memory map pages 9Fh-AFh according to the CMIS standard,
section 8.12 of revision 4.0.

Add a pair of new ethtool messages that allow:

* User space to trigger firmware update of transceiver modules

* The kernel to notify user space about the progress of the process

The user interface is designed to be asynchronous in order to avoid RTNL
being held for too long and to allow several modules to be updated
simultaneously. The interface is designed with CMIS compliant modules in
mind, but kept generic enough to accommodate future use cases, if these
arise.

The kernel interface that will implement the firmware update using CDB
command will include 2 layers that will be added under ethtool:

* The upper layer that will be triggered from the module layer, is
 cmis_ fw_update.
* The lower one is cmis_cdb.

In the future there might be more operations to implement using CDB
commands. Therefore, the idea is to keep the cmis_cdb interface clean and
the cmis_fw_update specific to the cdb commands handling it.

The communication between the kernel and the driver will be done using
two ethtool operations that enable reading and writing the transceiver
module EEPROM.
The operation ethtool_ops::get_module_eeprom_by_page, that is already
implemented, will be used for reading from the EEPROM the CDB reply,
e.g. reading module setting, state, etc.
The operation ethtool_ops::set_module_eeprom_by_page, that is added in
the current patchset, will be used for writing to the EEPROM the CDB
command such as start firmware image, run firmware image, etc.

Therefore in order for a driver to implement module flashing, that
driver needs to implement the two functions mentioned above.

Patchset overview:
Patch #1-#2: Implement the EEPROM writing in mlxsw.
Patch #3: Define the interface between the kernel and user space.
Patch #4: Add ability to notify the flashing firmware progress.
Patch #5: Veto operations during flashing.
Patch #6: Add extended compliance codes.
Patch #7: Add the cdb layer.
Patch #8: Add the fw_update layer.
Patch #9: Add ability to flash transceiver modules' firmware.

v8:
	Patch #7:
	* In the ethtool_cmis_wait_for_cond() evaluate the condition once more
	  to decide if the error code should be -ETIMEDOUT or something else.
	* s/netdev_err/netdev_err_once.

v7:
	Patch #4:
		* Return -ENOMEM instead of PTR_ERR(attr) on
		  ethnl_module_fw_flash_ntf_put_err().
	Patch #9:
		* Fix Warning for not unlocking the spin_lock in the error flow
          	  on module_flash_fw_work_list_add().
		* Avoid the fall-through on ethnl_sock_priv_destroy().

v6:
	* Squash some of the last patch to patch #5 and patch #9.
	Patch #3:
		* Add paragraph in .rst file.
	Patch #4:
		* Reserve '1' more place on SKB for NUL terminator in
		  the error message string.
		* Add more prints on error flow, re-write the printing
		  function and add ethnl_module_fw_flash_ntf_put_err().
		* Change the communication method so notification will be
		  sent in unicast instead of multicast.
		* Add new 'struct ethnl_module_fw_flash_ntf_params' that holds
		  the relevant info for unicast communication and use it to
		  send notification to the specific socket.
		* s/nla_put_u64_64bit/nla_put_uint/
	Patch #7:
		* In ethtool_cmis_cdb_init(), Use 'const' for the 'params'
		  parameter.
	Patch #8:
		* Add a list field to struct ethtool_module_fw_flash for
		  module_fw_flash_work_list that will be presented in the next
		  patch.
		* Move ethtool_cmis_fw_update() cleaning to a new function that
		  will be represented in the next patch.
		* Move some of the fields in struct ethtool_module_fw_flash to
		  a separate struct, so ethtool_cmis_fw_update() will get only
		  the relevant parameters for it.
		* Edit the relevant functions to get the relevant params for
		  them.
		* s/CMIS_MODULE_READY_MAX_DURATION_USEC/CMIS_MODULE_READY_MAX_DURATION_MSEC
	Patch #9:
		* Add a paragraph in the commit message.
		* Rename labels in module_flash_fw_schedule().
		* Add info to genl_sk_priv_*() and implement the relevant
		  callbacks, in order to handle properly a scenario of closing
		  the socket from user space before the work item was ended.
		* Add a list the holds all the ethtool_module_fw_flash struct
		  that corresponds to the in progress work items.
		* Add a new enum for the socket types.
		* Use both above to identify a flashing socket, add it to the
		  list and when closing socket affect only the flashing type.
		* Create a new function that will get the work item instead of
		  ethtool_cmis_fw_update().
		* Edit the relevant functions to get the relevant params for
		  them.
		* The new function will call the old ethtool_cmis_fw_update(),
		  and do the cleaning, so the existence of the list should be
		  completely isolated in module.c.
===================

Signed-off-by: David S. Miller <[email protected]>
  • Loading branch information
davem330 committed Jun 28, 2024
2 parents 8fda537 + 32b4c8b commit c977ac4
Show file tree
Hide file tree
Showing 22 changed files with 1,963 additions and 11 deletions.
55 changes: 55 additions & 0 deletions Documentation/netlink/specs/ethtool.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ definitions:
name: header-flags
type: flags
entries: [ compact-bitsets, omit-reply, stats ]
-
name: module-fw-flash-status
type: enum
entries: [ started, in_progress, completed, error ]

attribute-sets:
-
Expand Down Expand Up @@ -1004,6 +1008,32 @@ attribute-sets:
-
name: burst-tmr
type: u32
-
name: module-fw-flash
attributes:
-
name: header
type: nest
nested-attributes: header
-
name: file-name
type: string
-
name: password
type: u32
-
name: status
type: u32
enum: module-fw-flash-status
-
name: status-msg
type: string
-
name: done
type: uint
-
name: total
type: uint

operations:
enum-model: directional
Expand Down Expand Up @@ -1764,3 +1794,28 @@ operations:
name: mm-ntf
doc: Notification for change in MAC Merge configuration.
notify: mm-get
-
name: module-fw-flash-act
doc: Flash transceiver module firmware.

attribute-set: module-fw-flash

do:
request:
attributes:
- header
- file-name
- password
-
name: module-fw-flash-ntf
doc: Notification for firmware flashing progress and status.

attribute-set: module-fw-flash

event:
attributes:
- header
- status
- status-msg
- done
- total
70 changes: 70 additions & 0 deletions Documentation/networking/ethtool-netlink.rst
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@ Userspace to kernel:
``ETHTOOL_MSG_PLCA_GET_STATUS`` get PLCA RS status
``ETHTOOL_MSG_MM_GET`` get MAC merge layer state
``ETHTOOL_MSG_MM_SET`` set MAC merge layer parameters
``ETHTOOL_MSG_MODULE_FW_FLASH_ACT`` flash transceiver module firmware
===================================== =================================

Kernel to userspace:
Expand Down Expand Up @@ -274,6 +275,7 @@ Kernel to userspace:
``ETHTOOL_MSG_PLCA_GET_STATUS_REPLY`` PLCA RS status
``ETHTOOL_MSG_PLCA_NTF`` PLCA RS parameters
``ETHTOOL_MSG_MM_GET_REPLY`` MAC merge layer status
``ETHTOOL_MSG_MODULE_FW_FLASH_NTF`` transceiver module flash updates
======================================== =================================

``GET`` requests are sent by userspace applications to retrieve device
Expand Down Expand Up @@ -2041,6 +2043,73 @@ The attributes are propagated to the driver through the following structure:
.. kernel-doc:: include/linux/ethtool.h
:identifiers: ethtool_mm_cfg

MODULE_FW_FLASH_ACT
===================

Flashes transceiver module firmware.

Request contents:

======================================= ====== ===========================
``ETHTOOL_A_MODULE_FW_FLASH_HEADER`` nested request header
``ETHTOOL_A_MODULE_FW_FLASH_FILE_NAME`` string firmware image file name
``ETHTOOL_A_MODULE_FW_FLASH_PASSWORD`` u32 transceiver module password
======================================= ====== ===========================

The firmware update process consists of three logical steps:

1. Downloading a firmware image to the transceiver module and validating it.
2. Running the firmware image.
3. Committing the firmware image so that it is run upon reset.

When flash command is given, those three steps are taken in that order.

This message merely schedules the update process and returns immediately
without blocking. The process then runs asynchronously.
Since it can take several minutes to complete, during the update process
notifications are emitted from the kernel to user space updating it about
the status and progress.

The ``ETHTOOL_A_MODULE_FW_FLASH_FILE_NAME`` attribute encodes the firmware
image file name. The firmware image is downloaded to the transceiver module,
validated, run and committed.

The optional ``ETHTOOL_A_MODULE_FW_FLASH_PASSWORD`` attribute encodes a password
that might be required as part of the transceiver module firmware update
process.

The firmware update process can take several minutes to complete. Therefore,
during the update process notifications are emitted from the kernel to user
space updating it about the status and progress.



Notification contents:

+---------------------------------------------------+--------+----------------+
| ``ETHTOOL_A_MODULE_FW_FLASH_HEADER`` | nested | reply header |
+---------------------------------------------------+--------+----------------+
| ``ETHTOOL_A_MODULE_FW_FLASH_STATUS`` | u32 | status |
+---------------------------------------------------+--------+----------------+
| ``ETHTOOL_A_MODULE_FW_FLASH_STATUS_MSG`` | string | status message |
+---------------------------------------------------+--------+----------------+
| ``ETHTOOL_A_MODULE_FW_FLASH_DONE`` | uint | progress |
+---------------------------------------------------+--------+----------------+
| ``ETHTOOL_A_MODULE_FW_FLASH_TOTAL`` | uint | total |
+---------------------------------------------------+--------+----------------+

The ``ETHTOOL_A_MODULE_FW_FLASH_STATUS`` attribute encodes the current status
of the firmware update process. Possible values are:

.. kernel-doc:: include/uapi/linux/ethtool.h
:identifiers: ethtool_module_fw_flash_status

The ``ETHTOOL_A_MODULE_FW_FLASH_STATUS_MSG`` attribute encodes a status message
string.

The ``ETHTOOL_A_MODULE_FW_FLASH_DONE`` and ``ETHTOOL_A_MODULE_FW_FLASH_TOTAL``
attributes encode the completed and total amount of work, respectively.

Request translation
===================

Expand Down Expand Up @@ -2147,4 +2216,5 @@ are netlink only.
n/a ``ETHTOOL_MSG_PLCA_GET_STATUS``
n/a ``ETHTOOL_MSG_MM_GET``
n/a ``ETHTOOL_MSG_MM_SET``
n/a ``ETHTOOL_MSG_MODULE_FW_FLASH_ACT``
=================================== =====================================
57 changes: 57 additions & 0 deletions drivers/net/ethernet/mellanox/mlxsw/core_env.c
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,63 @@ mlxsw_env_get_module_eeprom_by_page(struct mlxsw_core *mlxsw_core,
}
EXPORT_SYMBOL(mlxsw_env_get_module_eeprom_by_page);

int
mlxsw_env_set_module_eeprom_by_page(struct mlxsw_core *mlxsw_core,
u8 slot_index, u8 module,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack)
{
struct mlxsw_env *mlxsw_env = mlxsw_core_env(mlxsw_core);
u32 bytes_written = 0;
u16 device_addr;
int err;

if (!mlxsw_env_linecard_is_active(mlxsw_env, slot_index)) {
NL_SET_ERR_MSG_MOD(extack,
"Cannot write to EEPROM of a module on an inactive line card");
return -EIO;
}

err = mlxsw_env_validate_module_type(mlxsw_core, slot_index, module);
if (err) {
NL_SET_ERR_MSG_MOD(extack, "EEPROM is not equipped on port module type");
return err;
}

device_addr = page->offset;

while (bytes_written < page->length) {
char mcia_pl[MLXSW_REG_MCIA_LEN];
char eeprom_tmp[128] = {};
u8 size;

size = min_t(u8, page->length - bytes_written,
mlxsw_env->max_eeprom_len);

mlxsw_reg_mcia_pack(mcia_pl, slot_index, module, page->page,
device_addr + bytes_written, size,
page->i2c_address);
mlxsw_reg_mcia_bank_number_set(mcia_pl, page->bank);
memcpy(eeprom_tmp, page->data + bytes_written, size);
mlxsw_reg_mcia_eeprom_memcpy_to(mcia_pl, eeprom_tmp);

err = mlxsw_reg_write(mlxsw_core, MLXSW_REG(mcia), mcia_pl);
if (err) {
NL_SET_ERR_MSG_MOD(extack, "Failed to access module's EEPROM");
return err;
}

err = mlxsw_env_mcia_status_process(mcia_pl, extack);
if (err)
return err;

bytes_written += size;
}

return 0;
}
EXPORT_SYMBOL(mlxsw_env_set_module_eeprom_by_page);

static int mlxsw_env_module_reset(struct mlxsw_core *mlxsw_core, u8 slot_index,
u8 module)
{
Expand Down
6 changes: 6 additions & 0 deletions drivers/net/ethernet/mellanox/mlxsw/core_env.h
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ mlxsw_env_get_module_eeprom_by_page(struct mlxsw_core *mlxsw_core,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack);

int
mlxsw_env_set_module_eeprom_by_page(struct mlxsw_core *mlxsw_core,
u8 slot_index, u8 module,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack);

int mlxsw_env_reset_module(struct net_device *netdev,
struct mlxsw_core *mlxsw_core, u8 slot_index,
u8 module, u32 *flags);
Expand Down
15 changes: 15 additions & 0 deletions drivers/net/ethernet/mellanox/mlxsw/minimal.c
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,20 @@ mlxsw_m_get_module_eeprom_by_page(struct net_device *netdev,
page, extack);
}

static int
mlxsw_m_set_module_eeprom_by_page(struct net_device *netdev,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack)
{
struct mlxsw_m_port *mlxsw_m_port = netdev_priv(netdev);
struct mlxsw_core *core = mlxsw_m_port->mlxsw_m->core;

return mlxsw_env_set_module_eeprom_by_page(core,
mlxsw_m_port->slot_index,
mlxsw_m_port->module,
page, extack);
}

static int mlxsw_m_reset(struct net_device *netdev, u32 *flags)
{
struct mlxsw_m_port *mlxsw_m_port = netdev_priv(netdev);
Expand Down Expand Up @@ -181,6 +195,7 @@ static const struct ethtool_ops mlxsw_m_port_ethtool_ops = {
.get_module_info = mlxsw_m_get_module_info,
.get_module_eeprom = mlxsw_m_get_module_eeprom,
.get_module_eeprom_by_page = mlxsw_m_get_module_eeprom_by_page,
.set_module_eeprom_by_page = mlxsw_m_set_module_eeprom_by_page,
.reset = mlxsw_m_reset,
.get_module_power_mode = mlxsw_m_get_module_power_mode,
.set_module_power_mode = mlxsw_m_set_module_power_mode,
Expand Down
15 changes: 15 additions & 0 deletions drivers/net/ethernet/mellanox/mlxsw/spectrum_ethtool.c
Original file line number Diff line number Diff line change
Expand Up @@ -1067,6 +1067,20 @@ mlxsw_sp_get_module_eeprom_by_page(struct net_device *dev,
module, page, extack);
}

static int
mlxsw_sp_set_module_eeprom_by_page(struct net_device *dev,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack)
{
struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev);
struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
u8 slot_index = mlxsw_sp_port->mapping.slot_index;
u8 module = mlxsw_sp_port->mapping.module;

return mlxsw_env_set_module_eeprom_by_page(mlxsw_sp->core, slot_index,
module, page, extack);
}

static int
mlxsw_sp_get_ts_info(struct net_device *netdev, struct ethtool_ts_info *info)
{
Expand Down Expand Up @@ -1256,6 +1270,7 @@ const struct ethtool_ops mlxsw_sp_port_ethtool_ops = {
.get_module_info = mlxsw_sp_get_module_info,
.get_module_eeprom = mlxsw_sp_get_module_eeprom,
.get_module_eeprom_by_page = mlxsw_sp_get_module_eeprom_by_page,
.set_module_eeprom_by_page = mlxsw_sp_set_module_eeprom_by_page,
.get_ts_info = mlxsw_sp_get_ts_info,
.get_eth_phy_stats = mlxsw_sp_get_eth_phy_stats,
.get_eth_mac_stats = mlxsw_sp_get_eth_mac_stats,
Expand Down
20 changes: 12 additions & 8 deletions include/linux/ethtool.h
Original file line number Diff line number Diff line change
Expand Up @@ -506,17 +506,16 @@ struct ethtool_ts_stats {
#define ETH_MODULE_MAX_I2C_ADDRESS 0x7f

/**
* struct ethtool_module_eeprom - EEPROM dump from specified page
* @offset: Offset within the specified EEPROM page to begin read, in bytes.
* @length: Number of bytes to read.
* @page: Page number to read from.
* @bank: Page bank number to read from, if applicable by EEPROM spec.
* struct ethtool_module_eeprom - plug-in module EEPROM read / write parameters
* @offset: When @offset is 0-127, it is used as an address to the Lower Memory
* (@page must be 0). Otherwise, it is used as an address to the
* Upper Memory.
* @length: Number of bytes to read / write.
* @page: Page number.
* @bank: Bank number, if supported by EEPROM spec.
* @i2c_address: I2C address of a page. Value less than 0x7f expected. Most
* EEPROMs use 0x50 or 0x51.
* @data: Pointer to buffer with EEPROM data of @length size.
*
* This can be used to manage pages during EEPROM dump in ethtool and pass
* required information to the driver.
*/
struct ethtool_module_eeprom {
u32 offset;
Expand Down Expand Up @@ -824,6 +823,8 @@ struct ethtool_rxfh_param {
* @get_module_eeprom_by_page: Get a region of plug-in module EEPROM data from
* specified page. Returns a negative error code or the amount of bytes
* read.
* @set_module_eeprom_by_page: Write to a region of plug-in module EEPROM,
* from kernel space only. Returns a negative error code or zero.
* @get_eth_phy_stats: Query some of the IEEE 802.3 PHY statistics.
* @get_eth_mac_stats: Query some of the IEEE 802.3 MAC statistics.
* @get_eth_ctrl_stats: Query some of the IEEE 802.3 MAC Ctrl statistics.
Expand Down Expand Up @@ -958,6 +959,9 @@ struct ethtool_ops {
int (*get_module_eeprom_by_page)(struct net_device *dev,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack);
int (*set_module_eeprom_by_page)(struct net_device *dev,
const struct ethtool_module_eeprom *page,
struct netlink_ext_ack *extack);
void (*get_eth_phy_stats)(struct net_device *dev,
struct ethtool_eth_phy_stats *phy_stats);
void (*get_eth_mac_stats)(struct net_device *dev,
Expand Down
4 changes: 3 additions & 1 deletion include/linux/netdevice.h
Original file line number Diff line number Diff line change
Expand Up @@ -1990,6 +1990,8 @@ enum netdev_reg_state {
*
* @threaded: napi threaded mode is enabled
*
* @module_fw_flash_in_progress: Module firmware flashing is in progress.
*
* @net_notifier_list: List of per-net netdev notifier block
* that follow this device when it is moved
* to another network namespace.
Expand Down Expand Up @@ -2374,7 +2376,7 @@ struct net_device {
bool proto_down;
bool threaded;
unsigned wol_enabled:1;

unsigned module_fw_flash_in_progress:1;
struct list_head net_notifier_list;

#if IS_ENABLED(CONFIG_MACSEC)
Expand Down
6 changes: 6 additions & 0 deletions include/linux/sfp.h
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,12 @@ enum {
SFF8024_ID_QSFP_8438 = 0x0c,
SFF8024_ID_QSFP_8436_8636 = 0x0d,
SFF8024_ID_QSFP28_8636 = 0x11,
SFF8024_ID_QSFP_DD = 0x18,
SFF8024_ID_OSFP = 0x19,
SFF8024_ID_DSFP = 0x1B,
SFF8024_ID_QSFP_PLUS_CMIS = 0x1E,
SFF8024_ID_SFP_DD_CMIS = 0x1F,
SFF8024_ID_SFP_PLUS_CMIS = 0x20,

SFF8024_ENCODING_UNSPEC = 0x00,
SFF8024_ENCODING_8B10B = 0x01,
Expand Down
Loading

0 comments on commit c977ac4

Please sign in to comment.