Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Accton-AS7726-32X Fans running on full speed / No fan throttling #13047

Open
tobzsc opened this issue Dec 14, 2022 · 4 comments
Open

Accton-AS7726-32X Fans running on full speed / No fan throttling #13047

tobzsc opened this issue Dec 14, 2022 · 4 comments
Labels
Triaged this issue has been triaged

Comments

@tobzsc
Copy link

tobzsc commented Dec 14, 2022 •
edited
Loading

Description

We have some newly installed Accton-AS7726-32X switches and the fans seem to be running on full speed all the time. There is no throttling. There seems to be a missmatch between show platform fan and show environment

Steps to reproduce the issue:

  1. show platform fan
  2. show environment

Describe the results you received:

admin@sonic:~$ show platform fan
  Drawer    LED         FAN    Speed    Direction    Presence    Status          Timestamp
--------  -----  ----------  -------  -----------  ----------  --------  -----------------
Fantray1  green  Fantray1_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray1  green  Fantray1_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray2  green  Fantray2_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray2  green  Fantray2_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray3  green  Fantray3_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray3  green  Fantray3_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray4  green  Fantray4_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray4  green  Fantray4_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray5  green  Fantray5_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray5  green  Fantray5_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray6  green  Fantray6_1   101RPM      EXHAUST     Present        OK  20220809 07:46:25
Fantray6  green  Fantray6_2   101RPM      EXHAUST     Present        OK  20220809 07:46:25
     N/A  green   PSU1_FAN1      24%      EXHAUST     Present        OK  20220809 07:46:25
     N/A  green   PSU2_FAN1      24%      EXHAUST     Present        OK  20220809 07:46:25
admin@sonic:~$

admin@sonic:~$ show environment
lm75-i2c-55-4b
Adapter: i2c-2-mux (chan_id 6)
CB_temp:      +25.5 C  (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-55-48
Adapter: i2c-2-mux (chan_id 6)
MB_RearMAC_temp:  +27.5 C  (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-54-4c
Adapter: i2c-2-mux (chan_id 5)
FB_temp:      +25.0 C  (high = +80.0 C, hyst = +75.0 C)

fan_ctrl-i2c-54-66
Adapter: i2c-2-mux (chan_id 5)
Fantray1 Front: 14400 RPM
Fantray1 Rear:  12700 RPM
Fantray2 Front: 14500 RPM
Fantray2 Rear:  12800 RPM
Fantray3 Front: 14100 RPM
Fantray3 Rear:  12700 RPM
Fantray4 Front: 13700 RPM
Fantray4 Rear:  12500 RPM
Fantray5 Front: 14100 RPM
Fantray5 Rear:  12800 RPM
Fantray6 Front: 14600 RPM
Fantray6 Rear:  12600 RPM

psu_pmbus-i2c-50-5b
Adapter: i2c-2-mux (chan_id 1)
PSU 1 Voltage:      12.12 V
PSU 1 Fan:         6272 RPM
PSU 1 Temperature:  +27.0 C
PSU 1 Power:        81.00 W
PSU 1 Current:       6.50 A

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +33.0 C  (high = +82.0 C, crit = +104.0 C)
Core 0:        +33.0 C  (high = +82.0 C, crit = +104.0 C)
Core 1:        +33.0 C  (high = +82.0 C, crit = +104.0 C)
Core 2:        +33.0 C  (high = +82.0 C, crit = +104.0 C)
Core 3:        +33.0 C  (high = +82.0 C, crit = +104.0 C)

psu_pmbus-i2c-49-58
Adapter: i2c-2-mux (chan_id 0)
PSU 2 Voltage:      12.06 V
PSU 2 Fan:         6016 RPM
PSU 2 Temperature:  +27.0 C
PSU 2 Power:        76.00 W
PSU 2 Current:       7.00 A

lm75-i2c-55-49
Adapter: i2c-2-mux (chan_id 6)
MB_FrontMAC_temp:  +26.0 C  (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-55-4a
Adapter: i2c-2-mux (chan_id 6)
MB_LeftCenter_temp:  +28.0 C  (high = +80.0 C, hyst = +75.0 C)

admin@sonic:~$

Describe the results you expected:

I would expect the fans to throttle. The switches have no load and the temperatures are contantly on normal levels.

Output of show version:

admin@sonic:~$ show version

SONiC Software Version: SONiC.master.188480-4ff15a64c
Distribution: Debian 11.5
Kernel: 5.10.0-18-2-amd64
Build commit: 4ff15a64c
Build date: Tue Dec 13 19:59:32 UTC 2022
Built by: AzDevOps@vmss-soni0004RU

Platform: x86_64-accton_as7726_32x-r0
HwSKU: Accton-AS7726-32X
ASIC: broadcom
ASIC Count: 1
Serial Number: 772632X2239341
Model Number: FP3ZZ7632074A
Hardware Revision: N/A
Uptime: 07:48:24 up 19 min,  1 user,  load average: 0.87, 0.74, 0.52
Date: Tue 09 Aug 2022 07:48:24

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-gbsyncd-broncos        latest                    b77e9f774a5d   404MB
docker-gbsyncd-broncos        master.188480-4ff15a64c   b77e9f774a5d   404MB
docker-gbsyncd-credo          latest                    992ad33c2749   374MB
docker-gbsyncd-credo          master.188480-4ff15a64c   992ad33c2749   374MB
docker-syncd-brcm             latest                    d695f4a4b51c   699MB
docker-syncd-brcm             master.188480-4ff15a64c   d695f4a4b51c   699MB
docker-orchagent              latest                    9acf9d5e9cd5   384MB
docker-orchagent              master.188480-4ff15a64c   9acf9d5e9cd5   384MB
docker-fpm-frr                latest                    5cb8d4dbf683   402MB
docker-fpm-frr                master.188480-4ff15a64c   5cb8d4dbf683   402MB
docker-teamd                  latest                    a4680d33271f   373MB
docker-teamd                  master.188480-4ff15a64c   a4680d33271f   373MB
docker-macsec                 latest                    ad187061cd96   375MB
docker-dhcp-relay             latest                    9f43a42c28a3   366MB
docker-eventd                 latest                    1b552db31b75   356MB
docker-eventd                 master.188480-4ff15a64c   1b552db31b75   356MB
docker-sonic-p4rt             latest                    0f56cfc95a6d   927MB
docker-sonic-p4rt             master.188480-4ff15a64c   0f56cfc95a6d   927MB
docker-snmp                   latest                    d264b996b403   396MB
docker-snmp                   master.188480-4ff15a64c   d264b996b403   396MB
docker-platform-monitor       latest                    b791a654f320   476MB
docker-platform-monitor       master.188480-4ff15a64c   b791a654f320   476MB
docker-database               latest                    cc1de31c06ae   356MB
docker-database               master.188480-4ff15a64c   cc1de31c06ae   356MB
docker-sonic-telemetry        latest                    35314309eb38   655MB
docker-sonic-telemetry        master.188480-4ff15a64c   35314309eb38   655MB
docker-router-advertiser      latest                    17c0aeb6c3d9   356MB
docker-router-advertiser      master.188480-4ff15a64c   17c0aeb6c3d9   356MB
docker-mux                    latest                    c9bddcfd12a4   405MB
docker-mux                    master.188480-4ff15a64c   c9bddcfd12a4   405MB
docker-lldp                   latest                    15b846cca661   399MB
docker-lldp                   master.188480-4ff15a64c   15b846cca661   399MB
docker-nat                    latest                    fa9ba56c38b1   350MB
docker-nat                    master.188480-4ff15a64c   fa9ba56c38b1   350MB
docker-sflow                  latest                    8e2149eb0107   348MB
docker-sflow                  master.188480-4ff15a64c   8e2149eb0107   348MB
docker-sonic-mgmt-framework   latest                    7bf4def41b69   476MB
docker-sonic-mgmt-framework   master.188480-4ff15a64c   7bf4def41b69   476MB

admin@sonic:~$

Additional information you deem important (e.g. issue happens only occasionally):

I also tried different images from other branches. All branches like 202205 or 202111 have this problem. On a Accton-AS7326-56X we do not have this problem.
@tobzsc
Copy link
Author

tobzsc commented Jan 2, 2023 •
edited
Loading

We have spoken with the EdgeCore support and were finally presented with a solution to get the fans back to normal speed.

admin@sonic:~$ sudo i2cget -y -f 54 0x66 0x33
0x01
admin@sonic:~$ sudo i2cset -y -f 54 0x66 0x33 0
admin@sonic:~$ sudo i2cget -y -f 54 0x66 0x33 
0x00
admin@sonic:~$ show platform fan
Drawer LED FAN Speed Direction Presence Status Timestamp
-------- ----- ---------- ------- ----------- ---------- -------- -----------------
Fantray1 green Fantray1_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray1 green Fantray1_2 38% EXHAUST Present OK 20220810 09:46:02
Fantray2 green Fantray2_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray2 green Fantray2_2 38% EXHAUST Present OK 20220810 09:46:02
Fantray3 green Fantray3_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray3 green Fantray3_2 38% EXHAUST Present OK 20220810 09:46:02
Fantray4 green Fantray4_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray4 green Fantray4_2 38% EXHAUST Present OK 20220810 09:46:02
Fantray5 green Fantray5_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray5 green Fantray5_2 38% EXHAUST Present OK 20220810 09:46:02
Fantray6 green Fantray6_1 38% EXHAUST Present OK 20220810 09:46:02
Fantray6 green Fantray6_2 38% EXHAUST Present OK 20220810 09:46:03
N/A green PSU1_FAN1 24% EXHAUST Present OK 20220810 09:46:03
N/A green PSU2_FAN1 24% EXHAUST Present OK 20220810 09:46:03
admin@sonic:~$ sudo sensors
lm75-i2c-55-4b
Adapter: i2c-2-mux (chan_id 6)
CB_temp: +25.0 C (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-55-48
Adapter: i2c-2-mux (chan_id 6)
MB_RearMAC_temp: +28.0 C (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-54-4c
Adapter: i2c-2-mux (chan_id 5)
FB_temp: +25.0 C (high = +80.0 C, hyst = +75.0 C)

fan_ctrl-i2c-54-66
Adapter: i2c-2-mux (chan_id 5)
Fantray1 Front: 9200 RPM
Fantray1 Rear: 8500 RPM
Fantray2 Front: 9500 RPM
Fantray2 Rear: 8700 RPM
Fantray3 Front: 9500 RPM
Fantray3 Rear: 8700 RPM
Fantray4 Front: 9300 RPM
Fantray4 Rear: 8700 RPM
Fantray5 Front: 9500 RPM
Fantray5 Rear: 8700 RPM
Fantray6 Front: 9300 RPM
Fantray6 Rear: 8500 RPM

psu_pmbus-i2c-50-5b
Adapter: i2c-2-mux (chan_id 1)
PSU 1 Voltage: 12.12 V 
PSU 1 Fan: 6144 RPM
PSU 1 Temperature: +28.0 C 
PSU 1 Power: 66.00 W 
PSU 1 Current: 5.00 A

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0 C (high = +82.0 C, crit = +104.0 C)
Core 0: +35.0 C (high = +82.0 C, crit = +104.0 C)
Core 1: +35.0 C (high = +82.0 C, crit = +104.0 C)
Core 2: +35.0 C (high = +82.0 C, crit = +104.0 C)
Core 3: +35.0 C (high = +82.0 C, crit = +104.0 C)

psu_pmbus-i2c-49-58
Adapter: i2c-2-mux (chan_id 0)
PSU 2 Voltage: 12.06 V 
PSU 2 Fan: 6016 RPM
PSU 2 Temperature: +28.0 C 
PSU 2 Power: 66.00 W 
PSU 2 Current: 5.50 A

lm75-i2c-55-4a
Adapter: i2c-2-mux (chan_id 6)
MB_LeftCenter_temp: +27.5 C (high = +80.0 C, hyst = +75.0 C)

lm75-i2c-55-49
Adapter: i2c-2-mux (chan_id 6)
MB_FrontMAC_temp: +26.0 C (high = +80.0 C, hyst = +75.0 C)

admin@sonic:~$

The following image shows that we now have disabled the watchdog.
image

We are not yet deep enough envolved in this project to create a PR to solve this issue. Maybe someone from Accton/EdgeCore can take care of this?!

@gechiang
Copy link
Collaborator

gechiang commented Jan 4, 2023

@jostar-yang CAn you help take a look?

@gechiang gechiang added the Triaged this issue has been triaged label Jan 4, 2023
@jostar-yang
Copy link
Contributor

We are checking the issue internally.

@jostar-yang
Copy link
Contributor

Please update code from PR:
#12993

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tobzsc @jostar-yang @gechiang