S3IP sysfs specification and S3IP sysfs framework HLD #1068

tianshangfei · 2022-08-22T15:04:46Z

SONiC is designed to be portable to a variety of network devices. Many devices share the same ASIC platform and only differ in device-specific hardware components, such as PSUs, fan modules, and environment sensors. Currently, ODM vendor provides drivers to expose the device-specific hardware through sysfs, allowing SONiC to communicate with them, as described in the Porting Guide. However, many inefficient porting works are still required for SONiC developers due to different drivers from different devices.

The S3IP sysfs specification defines a unified interface to access peripheral hardware on devices from different vendors, making it easier for SONiC to support different devices and platforms.

Repo	PR title	State
sonic-buildimage	Two platforms supporting S3IP SYSFS (TCS8400, TCS9400)
sonic-buildimage	S3IP sysfs framework - The PSU driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The FAN driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The Transceiver driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The sensor driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The CPLD and FPGA driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The LED and watchdog driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The slot and switch_rootsysfs driver framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The user framework module complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The demo driver complies with s3ip sysfs specification
sonic-buildimage	S3IP sysfs framework - The build project of s3ip framework

linux-foundation-easycla · 2022-08-22T15:04:53Z

The committers listed above are authorized under a signed CLA.

✅ login: tianshangfei (40fb633, 2f402fd, 48a2c4f, 9a0e85c, 6b24d2d, ba1d192)
✅ login: lihuay / name: Lihua Yuan (d8486e2)

zhangyanzhao · 2022-08-23T15:03:19Z

Can you please include the full name of S3IP in the HLD? Thanks.

zhangyanzhao · 2022-08-23T15:48:06Z

@Pettershao want to be the reviewer of this HLD.

chrispsommers · 2022-08-23T17:24:21Z

doc/s3ip_sysfs/s3ip_sysfs_specification.md

+/sys_switch/curr_sensor/curr[n]/max|R/W|int|Alarm threshold, unit: mA
+/sys_switch/curr_sensor/curr[n]/min|R/W|int|Alarm recovery threshold, unit: mA
+/sys_switch/curr_sensor/curr[n]/value|RO|int|current unit: mA
+


I suggest adding a Power Sensor sysfs section. This would provide the power in milliwatts from various power-sensing devices throughout the platform. Most on-board DC-DC converters (also called PWM or pulse-width modulator chips) provide voltage, current and calculated power. They often provide power In and Power out, allowing easy measurement of efficiency. It might be more reliable to read this directly rather than compute it from current * voltage, which might obtain readings at different times and result in invalid computation. It could also be more accurate because multiplying current * power in the controller results in compound loss of precision (rounded value * rounded value).

Prior to this, the requirement to directly collect the Power sensor was not collected (because it can be obtained normally via Current *voltage), so it was not covered in the first phase.

This is a good point, we can probably add this sysfs definition in the next stage.

chrispsommers · 2022-08-23T17:24:25Z

doc/s3ip_sysfs/s3ip_sysfs_specification.md

+|/sys_switch/fan/fan[n]/status |RO| enum|Fan states are defined as follows:<br>0: not present<br>1: present and normal<br>2: present and abnormal
+|/sys_switch/fan/fan[n]/led_status |R/W| enum| The fan status lights are defined as follows:<br>See the definition of enumeration value of LED status light for details
+
+### 7. Power information sysfs


I think a better title would be PSU Information sysfs, since the data comprise many individual readings including power, current, voltage, inventory info, etc. all related to system-level PSUs. Changing the title would also more clearly differentiate between the proposed Power Information sysfs in the comment above.

Yes, PSU Information sysfs would be more suitable

chrispsommers

Great proposal, please see comments.

chrispsommers · 2022-08-24T15:28:28Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+The S3IP sysfs specification is represented as an organized sysfs directory structure on white-box devices. Device management software and debugging tools need to access hardware through this interface.
+
+Both vendors and users should comply with the S3IP sysfs specification. Device vendors focus on the specification implementation, while users verify the usability of the device against this specification.


I wonder if stronger language should be used to set expectations about how thoroughly vendors implement this specification. For example, a switch main board might have 10-20 sensors; language in this document might specify that "Vendors SHOULD provide software access to every sensor capable of being read. For any available sensor, vendors SHALL provide S3IP sysfs access."

Ok, accept, I'll add it

FuzailBrcm · 2022-08-24T16:08:30Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+#### 2.2 Design of the S3ip sysfs specification
+Design Principles
+ - Unified, specificationized behavior, consistent experience among all SONiC devices, regardless of underlying platform; 


I feel the 2.0 platform APIs does the same thing, isn't it? These APIs are platform independent and platform vendors can implement them as they need i.e.

Either read them directly from the SysFS created by device drivers, or

Read them raw and then apply some filters/masks/pre-processing steps to get the final value

What is the reason/motivation behind the S3IP SysFS?
With S3IP, I see you are able to manage 1 but what about point#2?

We have already discussed the motivations of S3IP in separate meetings.

The S3IP SYSFS specification was proposed. because

For ODM users, hardware data conforming to S3IP SYSFS specification is conducive to in-depth processing and intelligent fault identification and prediction.

For ODM vendors, S3IP SYSFS is easy to understand and implement

The S3IP SYSFS framework is light and has low learning and maintenance costs

I have added a section explaining the differences between S3IP SYSfs and PDDF. Some contents are marked TBD, please help to complete them. thanks

FuzailBrcm · 2022-08-24T16:11:45Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+```
+
+Sample code exposes watchdog register to sysfs, This code should be implemented by the vendor


Vendors can have drivers which are common to various NOS. Isn't the S3IP framework mandating them to create SONiC specific drivers?

We consider the case of the common driver, so by the configuration file, the path of the common driver, soft link to S3IP sysfs directory, do not need to create SONiC specific drivers

jeff-yin · 2022-08-24T22:48:09Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+SONiC is designed to be portable to a variety of network devices. Many devices share the same ASIC platform and only differ in device-specific hardware components, such as PSUs, fan modules, and environment sensors. Currently, ODM vendor provides drivers to expose the device-specific hardware through sysfs, allowing SONiC to communicate with them, as described in the [Porting Guide](https://github.com/sonic-net/SONiC/wiki/Porting-Guide). However, many inefficient porting works are still required for SONiC developers due to different drivers from different devices. 
+
+The S3IP sysfs specification defines a unified interface to access peripheral hardware on devices from different vendors, making it easier for SONiC to support different devices and platforms.


By my count, this is the third proposal to unify vendor device implementations for SONiC. Two others are:

PDDF: https://github.com/sonic-net/SONiC/blob/master/doc/platform/brcm_pdk_pddf.md

PIT: https://github.com/sonic-net/SONiC/pull/1014/files

How is this differentiated from the other two, and can this feature leverage anything from the other two?

I have added a section explaining the differences between S3IP SYSfs and PDDF. Please see this update.

jeff-yin · 2022-08-24T22:48:36Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+3. Porting S3IP sysfs framework to platform project
+ - Create the S3IP sysfs service configuration file. 
+     - Modify /s3ip/s3ip_sysfs_service/s3ip_sysfs_conf.json to create a configuration file
+     - This file should be installed to /etc/s3ip/s3ip_sysfs_conf.json on the device


Like I mentioned in the review for PIT:
https://github.com/sonic-net/SONiC/pull/1014/files#r932428427

I am concerned that this will be yet another configuration file that platform vendors will have to generate and maintain when bringing platforms in to SONiC. Can the platform configuration be derived from already-existing platform configs?

We plan to be two-in-one with the PDDF framework. but in two steps. The first step was to integrate the S3IP SYSFS framework into the Sonic community.
At present, there are many vendors in china that comply with S3IP framework. The S3IP SYSFS Framework is integrated into the SONiC community so that Chinese ODM vendors can more easily contribute their existing platforms to the community, and other vendors can also have the opportunity to adapt their platforms according to S3IP specifications and expand their business opportunities in China. This is good for SONiC community ecology;

chrispsommers · 2022-09-28T17:20:38Z

doc/s3ip_sysfs/s3ip_sysfs_specification.md

+|/sys_switch/fan/fan[n]/led_status |R/W| enum| The fan status lights are defined as follows:<br>See the definition of enumeration value of LED status light for details
+
+### 7. PSU information sysfs
+Power Information The Sysfs path must be /sys_switch/psu/


Recommend "Power Supply Unit" information for clarity, especially if you add a "Power Sensor" section in the future per #1068 (comment)

Regarding #1068 (comment) how do intend to track this? Could you add a placeholder in the spec?

ok，I will update this

already updated

chrispsommers · 2022-09-28T17:26:42Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+| Item  | S3IP sysfs|PDDF|
+|-|-|-|
+|Requirements| The requirements are put forward from the user's perspective, and the sysfs node is summarized from the actual operation experience. The goal is to unify the interface for device management | The requirements are proposed from the technical point of view.TBD
+|Ecological | Devices compliant with S3IP SYSFS specification have been widely used in data centers.<br> The [S3IP](http://www.s3ip.org/) project involved both vendors and users[S3IP] (including Tencent, Alibaba, Baidu, Kuaishou, Meituan, Jingdong and more than a dozen ODM vendors). Vendors and users complete a closed-loop of requirements. standards and debugging tools that have the ability to iterate continuously. | TBD


Ecological -> Ecosystem is probably what you meant.

ok，accept

EcoSystem: (PDDF) PDDF is a new framework and it is developed in the SONiC context. Some ODM platforms are already using PDDF. PDDF is an underlying framework which ODMs can use for faster development but it does not exposes any fixed SysFS nodes to the user.

Please add this comment in the PDDF column for the row "EcoSystem"

already updated

chrispsommers · 2022-09-28T17:27:36Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+#### 2.2 Design of the S3ip sysfs specification
+Design Principles
+ - Unified, specificationized behavior, consistent experience among all SONiC devices, regardless of underlying platform; 


"specificationized" is not a real word, I suggest "well-specified"

ok，accept

already updated

chrispsommers

Minor grammar suggestions, otherwise looks good.

FuzailBrcm · 2022-09-30T10:07:18Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+| Item  | S3IP sysfs|PDDF|
+|-|-|-|
+|Requirements| The requirements are put forward from the user's perspective, and the sysfs node is summarized from the actual operation experience. The goal is to unify the interface for device management | The requirements are proposed from the technical point of view.TBD


*** Requirements: (PDDF) The requirements are proposed from technical point of view aimed at platform driver and APIs development in SONiC. Only those SysFS which are required by SONiC platform APIs are exposed.

Please add this comment in the PDDF column for the row "Requirements"

already updated

FuzailBrcm · 2022-09-30T10:15:11Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+|-|-|-|
+|Requirements| The requirements are put forward from the user's perspective, and the sysfs node is summarized from the actual operation experience. The goal is to unify the interface for device management | The requirements are proposed from the technical point of view.TBD
+|Ecological | Devices compliant with S3IP SYSFS specification have been widely used in data centers.<br> The [S3IP](http://www.s3ip.org/) project involved both vendors and users[S3IP] (including Tencent, Alibaba, Baidu, Kuaishou, Meituan, Jingdong and more than a dozen ODM vendors). Vendors and users complete a closed-loop of requirements. standards and debugging tools that have the ability to iterate continuously. | TBD
+|Development Mode | Regular development model,<br>Programming is required to implement the requirements. <br>ODM venders need to provide professional driver support for customers，Customers validating device with sysfs| Data driven development model.TBD


Development Mode: (PDDF) ODM vendors can use common PDDF kernel drivers and user space common platform APIs. Only some platform specific device data needs to be provided by the ODMs in the form of JSON files. Validation is via usual SONiC CLIs.

Please add this comment in the PDDF column for the row "Development Mode"

already updated

FuzailBrcm · 2022-09-30T10:17:54Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+|Requirements| The requirements are put forward from the user's perspective, and the sysfs node is summarized from the actual operation experience. The goal is to unify the interface for device management | The requirements are proposed from the technical point of view.TBD
+|Ecological | Devices compliant with S3IP SYSFS specification have been widely used in data centers.<br> The [S3IP](http://www.s3ip.org/) project involved both vendors and users[S3IP] (including Tencent, Alibaba, Baidu, Kuaishou, Meituan, Jingdong and more than a dozen ODM vendors). Vendors and users complete a closed-loop of requirements. standards and debugging tools that have the ability to iterate continuously. | TBD
+|Development Mode | Regular development model,<br>Programming is required to implement the requirements. <br>ODM venders need to provide professional driver support for customers，Customers validating device with sysfs| Data driven development model.TBD
+|Flexible | 1.Bus independent, The hardware support:<br>Fan<br>PSU<br>System EEPROM<br>Transceivers<br>CPLD<br>FPGA<br>System LED<br>Temperature sensors<br>Current sensors<br>Voltage sensors<br>Slot<br>Watchdog<br><br>2.Support scenarios with many customization requirements, such as FPGA-Polling, BMC management hardware and firmware upgrades <br><br>3.Normalized SYSFS is easy for hardware fault identification and prediction <br><br>4.Easy to debug for ODM users, and they need not care about the bus topology |Support I2C bus, other bus support requires development work <br>Fan<br>PSU<br>System EEPROM<br>CPLD<br>Optic Transceivers<br>System LED  control via CPLD<br>System Status Registers in CPLD<br>Temp Sensors <br><br> TBD


Flexibility: (PDDF) PDDF can be used on the platforms which use I2C bus to communicate with the peripheral devices. Platforms which use BMC can also be brought up using PDDF. In future PDDF would be supported on platform using PCIE FPGA devices.

Please add this comment in the PDDF column for the row "Flexibility"

already updated

chrispsommers · 2022-10-26T23:47:21Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+ - This specification defines the path name, permissions, data type, and data unit for each hardware
+ - The sysfs path defined in this specification must exist, and the file content should be "NA" if no hardware is available.
+
+S3IP sysfs specifiction : [specifiction](/doc/s3ip_sysfs/s3ip_sysfs_specification.md)


spelling error, should be:
S3IP sysfs specification : specification

chrispsommers · 2022-10-26T23:50:51Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+| Item  | S3IP sysfs|PDDF|
+|-|-|-|
+|Requirements| The requirements are put forward from the user's perspective, and the sysfs node is summarized from the actual operation experience. The goal is to unify the interface for device management | The requirements are proposed from technical point of view aimed at platform driver and APIs development in SONiC. Only those SysFS which are required by SONiC platform APIs are exposed.
+|Ecosystem | Devices compliant with S3IP SYSFS specification have been widely used in data centers.<br> The [S3IP](http://www.s3ip.org/) project involved both vendors and users[S3IP] (including Tencent, Alibaba, Baidu, Kuaishou, Meituan, Jingdong and more than a dozen ODM vendors). Vendors and users complete a closed-loop of requirements. standards and debugging tools that have the ability to iterate continuously. | PDDF is a new framework and it is developed in the SONiC context. Some ODM platforms are already using PDDF. PDDF is an underlying framework which ODMs can use for faster development but it does not exposes any fixed SysFS nodes to the user.


complete a closed-loop of requirements. standards

Should the period be a comma?

chrispsommers · 2022-10-26T23:52:09Z

doc/s3ip_sysfs/s3ip_sysfs_hld.md

+
+PDDF is not incompatible with S3IP SYSFS, we will combine the two parts into two Phases:
+
+Phase 1: The S3IP SYSFS Framework is integrated into the SONiC community so that Chinese ODM vendors can more easily contribute their existing platforms to the community, and other vendors can also have the opportunity to adapt their platforms according to S3IP specifications and expand their business opportunities in China. This is good for SONiC community ecology;


ecology -> ecosystem

chrispsommers · 2022-10-27T15:46:43Z

I modified it once before, and found that some alarms were not introduced by our modification, and some alarms were considered to be false positives by tool inspection.
I will modify it again.
Does LGTM have to be cleared before PR can be merged?

I'm not sure I fully understand your question. LGTM = "Looks good to me," sorry if my use of this acronym was ambiguous. I am OK with the PR.

GitHub is saying someone with write access needs to approve, in other words a repo maintainer.

tianshangfei · 2022-10-28T06:16:02Z

I modified it once before, and found that some alarms were not introduced by our modification, and some alarms were considered to be false positives by tool inspection.
I will modify it again.
Does LGTM have to be cleared before PR can be merged?

I'm not sure I fully understand your question. LGTM = "Looks good to me," sorry if my use of this acronym was ambiguous. I am OK with the PR.

GitHub is saying someone with write access needs to approve, in other words a repo maintainer.

OK，github's code review tool is also called LGTM （refer to LGTM.com）, So there was a misunderstanding.
How can I speed up the merge process？ thanks.

tianshangfei · 2022-10-31T02:36:40Z

@lguohan I have submitted the code PR for some time, and no further comments were received. I hope the 202211 version supports this feature, Please help to review it. Thanks

zhangyanzhao · 2022-11-12T00:46:08Z

@lguohan can you please check if this one can be merged? Thanks.

FuzailBrcm

Looks okay to me

zhangyanzhao · 2022-12-07T23:10:59Z

@lguohan can you please help to check if this PR can be merged? Reviewers have approved and build passed. Thanks.

tianshangfei · 2022-12-21T07:26:05Z

@lguohan please help to review this HLD

zhangyanzhao · 2023-01-03T01:47:40Z

@lguohan would you please help to review and merge this PR? Thanks.

zhangyanzhao · 2023-02-28T04:55:47Z

@lguohan , all the code PRs are merged, can you please check if this PR can be merged as well? Thanks.

zhangyanzhao · 2023-02-28T04:56:47Z

@tianshangfei can you please rebase this PR? Seems like the info is out of sync with master branch

tianshangfei · 2023-03-09T13:27:24Z

@tianshangfei can you please rebase this PR? Seems like the info is out of sync with master branch
Ok, PR is synchronized with master branch

zhangyanzhao · 2023-04-08T06:44:37Z

@lguohan can you please help to review and merge this HLD PR? The code PRs have been merged.

FuzailBrcm · 2023-04-18T05:29:44Z

doc/s3ip_sysfs/s3ip_sysfs_specification.md

+|/sys_switch/fan/fan[n]/motor_number |RO| int| Number of fan motors
+|/sys_switch/fan/fan[n]/direction  |RO| enum| The duct types are defined as follows:<br>0: F2B, forward air duct <br>1: B2F, rear duct
+|/sys_switch/fan/fan[n]/ratio  |R/W| int| Motor speed percentage, value range 0-100
+|/sys_switch/fan/fan[n]/motor[n]/speed |RO| int| Speed value,unit: RPM


@tianshangfei
Are you considering a Fantray as one fan unit? Or Fan[n] refers to individual fans inside each Fantray?

I have some confusion about the "Number of Fan Motors" field. If a Fantray is one fan unit, then "Front" and "Rear" fans can be marked as two motors which have different speed, max & min speeds, spped tolerance etc.

The definitions of fans and motors are as follows: a device contains n fan units, and a fan unit contains 1-2 motors. Each motor may have a fault, so the speed, maximum and minimum speed, speed tolerance, etc. are collected by the motor. "F2B " and "B2F " are used to identify the direction of the fan unit. F2B is the abbreviation for front-to-back airflow.

Signed-off-by: tianshangfei <[email protected]>

chrispsommers reviewed Aug 23, 2022

View reviewed changes

chrispsommers suggested changes Aug 23, 2022

View reviewed changes

chrispsommers reviewed Aug 24, 2022

View reviewed changes

FuzailBrcm reviewed Aug 24, 2022

View reviewed changes

jeff-yin reviewed Aug 24, 2022

View reviewed changes

tianshangfei requested review from FuzailBrcm, chrispsommers and jeff-yin and removed request for FuzailBrcm and chrispsommers September 28, 2022 13:34

chrispsommers reviewed Sep 28, 2022

View reviewed changes

chrispsommers suggested changes Sep 28, 2022

View reviewed changes

FuzailBrcm reviewed Sep 30, 2022

View reviewed changes

zhangyanzhao requested review from adyeung and removed request for jeff-yin October 3, 2022 04:46

tianshangfei mentioned this pull request Oct 13, 2022

Implementation of S3IP SYSFS framework and two platforms supporting S3IP SYSFS sonic-net/sonic-buildimage#12204

Closed

7 tasks

tianshangfei requested review from chrispsommers and removed request for adyeung October 26, 2022 01:42

chrispsommers reviewed Oct 26, 2022

View reviewed changes

zhangyanzhao requested a review from adyeung November 22, 2022 00:55

FuzailBrcm previously approved these changes Nov 23, 2022

View reviewed changes

jeff-yin previously approved these changes Dec 8, 2022

View reviewed changes

zhangyanzhao requested a review from lguohan January 3, 2023 01:46

zhangyanzhao added the Request for 202211 Branch label Apr 8, 2023

FuzailBrcm reviewed Apr 18, 2023

View reviewed changes

adyeung approved these changes May 1, 2023

View reviewed changes

zhangyanzhao dismissed stale reviews from jeff-yin, FuzailBrcm, and chrispsommers via 272e86a May 19, 2023 15:55

zhangyanzhao force-pushed the master branch from f352ebf to 272e86a Compare May 19, 2023 15:55

tianshangfei added 5 commits May 26, 2023 14:21

s3ip sysfs hld

1e4f8fa

add section compare with PDDF

64dae78

update PDDF related section and Correcting grammar errors

1323a06

update s3ip sysfs project part

9e26d1d

fix spelling /punctuation errors

5fb3bc8

Signed-off-by: tianshangfei <[email protected]>

zhangyanzhao force-pushed the master branch from 272e86a to 5fb3bc8 Compare May 26, 2023 21:21

Merge branch 'master' into master

f9113cc

lguohan merged commit dbc5627 into sonic-net:master May 26, 2023


		The S3IP sysfs specification is represented as an organized sysfs directory structure on white-box devices. Device management software and debugging tools need to access hardware through this interface.

		Both vendors and users should comply with the S3IP sysfs specification. Device vendors focus on the specification implementation, while users verify the usability of the device against this specification.


		```

		Sample code exposes watchdog register to sysfs, This code should be implemented by the vendor


		PDDF is not incompatible with S3IP SYSFS, we will combine the two parts into two Phases:

		Phase 1: The S3IP SYSFS Framework is integrated into the SONiC community so that Chinese ODM vendors can more easily contribute their existing platforms to the community, and other vendors can also have the opportunity to adapt their platforms according to S3IP specifications and expand their business opportunities in China. This is good for SONiC community ecology;

S3IP sysfs specification and S3IP sysfs framework HLD #1068

S3IP sysfs specification and S3IP sysfs framework HLD #1068

Conversation

tianshangfei commented Aug 22, 2022 • edited Loading

linux-foundation-easycla bot commented Aug 22, 2022 • edited Loading

zhangyanzhao commented Aug 23, 2022

zhangyanzhao commented Aug 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrispsommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FuzailBrcm Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

tianshangfei Sep 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrispsommers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrispsommers commented Oct 27, 2022

tianshangfei commented Oct 28, 2022

tianshangfei commented Oct 31, 2022

zhangyanzhao commented Nov 12, 2022

FuzailBrcm left a comment

Choose a reason for hiding this comment

zhangyanzhao commented Dec 7, 2022

tianshangfei commented Dec 21, 2022

zhangyanzhao commented Jan 3, 2023

zhangyanzhao commented Feb 28, 2023

zhangyanzhao commented Feb 28, 2023

tianshangfei commented Mar 9, 2023

zhangyanzhao commented Apr 8, 2023

Choose a reason for hiding this comment

tianshangfei Apr 23, 2023 • edited Loading

Choose a reason for hiding this comment

tianshangfei commented Aug 22, 2022 •

edited

Loading

linux-foundation-easycla bot commented Aug 22, 2022 •

edited

Loading

FuzailBrcm Aug 24, 2022 •

edited

Loading

tianshangfei Sep 13, 2022 •

edited

Loading

tianshangfei Apr 23, 2023 •

edited

Loading