Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove over-kill output processing routine #16519

Merged
merged 1 commit into from
Mar 23, 2022
Merged

Remove over-kill output processing routine #16519

merged 1 commit into from
Mar 23, 2022

Conversation

erjiaqing
Copy link
Contributor

Problem

  • The queue in the log processing routing is an overkill, remove it does not affect the correctness of the output.
  • FindBinaryPath is too slow on darwin.

Change overview

  • Remove the use of FindBinaryPath
  • Update doc

Testing

  • CI should green
  • Tested on darwin, and the speed is fast enough.

@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation github scripts workflows labels Mar 22, 2022
@erjiaqing erjiaqing changed the title Remove over-killed output processing Remove over-kill output processing routine Mar 22, 2022
@github-actions
Copy link

github-actions bot commented Mar 22, 2022 •
edited
Loading

PR #16519: Size comparison from 2e658fb to e5515d6

Full report (13 builds for cyw30739, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section 2e658fb e5515d6 change % change
cyw30739 light cyw930739m2evb_01 (read/write) 603870 603870 0 0.0
.app_xip_area 510968 510968 0 0.0
.bss 75656 75656 0 0.0
.data 596 596 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
lock cyw930739m2evb_01 (read/write) 561658 561658 0 0.0
.app_xip_area 470284 470284 0 0.0
.bss 74160 74160 0 0.0
.data 560 560 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 571494 571494 0 0.0
.app_xip_area 470472 470472 0 0.0
.bss 83488 83488 0 0.0
.data 500 500 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
k32w light k32w061+release (read/write) 701092 701092 0 0.0
.bss 77656 77656 0 0.0
.data 1868 1868 0 0.0
.text 615768 615768 0 0.0
lock k32w061+release (read/write) 700964 700964 0 0.0
.bss 77624 77624 0 0.0
.data 1908 1908 0 0.0
.text 615632 615632 0 0.0
linux chip-tool-ipv6only arm64 (read only) 9769404 9769404 0 0.0
(read/write) 472689 472689 0 0.0
.bss 40609 40609 0 0.0
.data 1128 1128 0 0.0
.data.rel.ro 371824 371824 0 0.0
.dynamic 560 560 0 0.0
.got 55328 55328 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 493604 493604 0 0.0
.text 8224436 8224436 0 0.0
thermostat-no-ble arm64 (read only) 2262500 2262500 0 0.0
(read/write) 148273 148273 0 0.0
.bss 62753 62753 0 0.0
.data 1024 1024 0 0.0
.data.rel.ro 77000 77000 0 0.0
.dynamic 560 560 0 0.0
.got 4480 4480 0 0.0
.init 24 24 0 0.0
.init_array 360 360 0 0.0
.rodata 140036 140036 0 0.0
.text 1899152 1899152 0 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2353452 2353452 0 0.0
.bss 184652 184652 0 0.0
.data 5752 5752 0 0.0
.text 1316052 1316052 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1155987 1155987 0 0.0
bss 146716 146716 0 0.0
rodata 145956 145956 0 0.0
text 788148 788148 0 0.0
p6 all-clusters-app default (read/write) 2492976 2492976 0 0.0
.bss 118072 118072 0 0.0
.data 2632 2632 0 0.0
.text 1451240 1451240 0 0.0
light-app default (read/write) 2396376 2396376 0 0.0
.bss 111544 111544 0 0.0
.data 2488 2488 0 0.0
.text 1354640 1354640 0 0.0
lock-app default (read/write) 2359928 2359928 0 0.0
.bss 111288 111288 0 0.0
.data 2448 2448 0 0.0
.text 1318192 1318192 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 895830 895830 0 0.0
bss 87424 87424 0 0.0
noinit 37160 37160 0 0.0
text 633404 633404 0 0.0

@erjiaqing erjiaqing force-pushed the python/remove-wrapper-code branch from e5515d6 to c68b28e Compare March 22, 2022 10:34
@github-actions
Copy link

github-actions bot commented Mar 22, 2022 •
edited
Loading

PR #16519: Size comparison from 2e658fb to c68b28e

Full report (27 builds for cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section 2e658fb c68b28e change % change
cyw30739 light cyw930739m2evb_01 (read/write) 603870 603870 0 0.0
.app_xip_area 510968 510968 0 0.0
.bss 75656 75656 0 0.0
.data 596 596 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
lock cyw930739m2evb_01 (read/write) 561658 561658 0 0.0
.app_xip_area 470284 470284 0 0.0
.bss 74160 74160 0 0.0
.data 560 560 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 571494 571494 0 0.0
.app_xip_area 470472 470472 0 0.0
.bss 83488 83488 0 0.0
.data 500 500 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A (read only) 922008 922008 0 0.0
(read/write) 128760 128760 0 0.0
.bss 126768 126768 0 0.0
.data 1988 1988 0 0.0
.text 922000 922000 0 0.0
BRD4161A+rpc (read only) 950820 950820 0 0.0
(read/write) 144712 144712 0 0.0
.bss 142544 142544 0 0.0
.data 2168 2168 0 0.0
.text 950812 950812 0 0.0
window-app BRD4161A (read only) 852384 852384 0 0.0
(read/write) 126720 126720 0 0.0
.bss 124856 124856 0 0.0
.data 1864 1864 0 0.0
.text 852376 852376 0 0.0
esp32 all-clusters-app c3devkit (read only) 963776 963776 0 0.0
(read/write) 1393690 1393690 0 0.0
.dram0.bss 62072 62072 0 0.0
.dram0.data 14188 14188 0 0.0
.flash.rodata 198512 198512 0 0.0
.flash.text 963776 963776 0 0.0
.iram0.text 62016 62016 0 0.0
m5stack (read only) 1020319 1020319 0 0.0
(read/write) 461460 461460 0 0.0
.dram0.bss 67592 67592 0 0.0
.dram0.data 34016 34016 0 0.0
.flash.rodata 228016 228016 0 0.0
.flash.text 1014935 1014935 0 0.0
.iram0.text 123107 123107 0 0.0
k32w light k32w061+release (read/write) 701092 701092 0 0.0
.bss 77656 77656 0 0.0
.data 1868 1868 0 0.0
.text 615768 615768 0 0.0
lock k32w061+release (read/write) 700964 700964 0 0.0
.bss 77624 77624 0 0.0
.data 1908 1908 0 0.0
.text 615632 615632 0 0.0
linux all-clusters-app debug (read only) 2477089 2477089 0 0.0
(read/write) 143152 143152 0 0.0
.bss 57312 57312 0 0.0
.data 1264 1264 0 0.0
.data.rel.ro 78840 78840 0 0.0
.dynamic 592 592 0 0.0
.got 4168 4168 0 0.0
.init 27 27 0 0.0
.init_array 936 936 0 0.0
.rodata 212005 212005 0 0.0
.text 2101634 2101634 0 0.0
bridge-app debug+rpc (read only) 1752637 1752637 0 0.0
(read/write) 89424 89424 0 0.0
.bss 44456 44456 0 0.0
.data 1952 1952 0 0.0
.data.rel.ro 37944 37944 0 0.0
.dynamic 592 592 0 0.0
.got 3920 3920 0 0.0
.init 27 27 0 0.0
.init_array 544 544 0 0.0
.rodata 144684 144684 0 0.0
.text 1492741 1492741 0 0.0
chip-tool debug (read only) 10145893 10145893 0 0.0
(read/write) 354848 354848 0 0.0
.bss 22304 22304 0 0.0
.data 1072 1072 0 0.0
.data.rel.ro 325408 325408 0 0.0
.dynamic 608 608 0 0.0
.got 4784 4784 0 0.0
.init 27 27 0 0.0
.init_array 632 632 0 0.0
.rodata 517813 517813 0 0.0
.text 8845477 8845477 0 0.0
chip-tool-ipv6only arm64 (read only) 9769404 9769404 0 0.0
(read/write) 472689 472689 0 0.0
.bss 40609 40609 0 0.0
.data 1128 1128 0 0.0
.data.rel.ro 371824 371824 0 0.0
.dynamic 560 560 0 0.0
.got 55328 55328 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 493604 493604 0 0.0
.text 8224436 8224436 0 0.0
door-lock-app debug (read only) 2004473 2004473 0 0.0
(read/write) 116832 116832 0 0.0
.bss 47584 47584 0 0.0
.data 992 992 0 0.0
.data.rel.ro 62856 62856 0 0.0
.dynamic 592 592 0 0.0
.got 4120 4120 0 0.0
.init 27 27 0 0.0
.init_array 664 664 0 0.0
.rodata 181180 181180 0 0.0
.text 1673634 1673634 0 0.0
lighting-app debug+rpc (read only) 2178137 2178137 0 0.0
(read/write) 123792 123792 0 0.0
.bss 48864 48864 0 0.0
.data 1472 1472 0 0.0
.data.rel.ro 67928 67928 0 0.0
.dynamic 608 608 0 0.0
.got 4168 4168 0 0.0
.init 27 27 0 0.0
.init_array 744 744 0 0.0
.rodata 175484 175484 0 0.0
.text 1843634 1843634 0 0.0
ota-provider-app debug (read only) 1938833 1938833 0 0.0
(read/write) 112624 112624 0 0.0
.bss 47488 47488 0 0.0
.data 1288 1288 0 0.0
.data.rel.ro 58200 58200 0 0.0
.dynamic 608 608 0 0.0
.got 4376 4376 0 0.0
.init 27 27 0 0.0
.init_array 616 616 0 0.0
.rodata 166571 166571 0 0.0
.text 1620498 1620498 0 0.0
ota-requestor-app debug (read only) 1969001 1969001 0 0.0
(read/write) 115592 115592 0 0.0
.bss 48480 48480 0 0.0
.data 1416 1416 0 0.0
.data.rel.ro 60264 60264 0 0.0
.dynamic 592 592 0 0.0
.got 4168 4168 0 0.0
.init 27 27 0 0.0
.init_array 640 640 0 0.0
.rodata 162980 162980 0 0.0
.text 1654034 1654034 0 0.0
shell debug (read only) 2426433 2426433 0 0.0
(read/write) 147208 147208 0 0.0
.bss 67240 67240 0 0.0
.data 784 784 0 0.0
.data.rel.ro 73472 73472 0 0.0
.dynamic 592 592 0 0.0
.got 4152 4152 0 0.0
.init 27 27 0 0.0
.init_array 920 920 0 0.0
.rodata 209426 209426 0 0.0
.text 2060306 2060306 0 0.0
thermostat-no-ble arm64 (read only) 2262500 2262500 0 0.0
(read/write) 148273 148273 0 0.0
.bss 62753 62753 0 0.0
.data 1024 1024 0 0.0
.data.rel.ro 77000 77000 0 0.0
.dynamic 560 560 0 0.0
.got 4480 4480 0 0.0
.init 24 24 0 0.0
.init_array 360 360 0 0.0
.rodata 140036 140036 0 0.0
.text 1899152 1899152 0 0.0
tv-app debug (read only) 2677657 2677657 0 0.0
(read/write) 247712 247712 0 0.0
.bss 164384 164384 0 0.0
.data 3104 3104 0 0.0
.data.rel.ro 74168 74168 0 0.0
.dynamic 592 592 0 0.0
.got 4552 4552 0 0.0
.init 27 27 0 0.0
.init_array 888 888 0 0.0
.rodata 207413 207413 0 0.0
.text 2295394 2295394 0 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2353452 2353452 0 0.0
.bss 184652 184652 0 0.0
.data 5752 5752 0 0.0
.text 1316052 1316052 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1155987 1155987 0 0.0
bss 146716 146716 0 0.0
rodata 145956 145956 0 0.0
text 788148 788148 0 0.0
p6 all-clusters-app default (read/write) 2492976 2492976 0 0.0
.bss 118072 118072 0 0.0
.data 2632 2632 0 0.0
.text 1451240 1451240 0 0.0
light-app default (read/write) 2396376 2396376 0 0.0
.bss 111544 111544 0 0.0
.data 2488 2488 0 0.0
.text 1354640 1354640 0 0.0
lock-app default (read/write) 2359928 2359928 0 0.0
.bss 111288 111288 0 0.0
.data 2448 2448 0 0.0
.text 1318192 1318192 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 895830 895830 0 0.0
bss 87424 87424 0 0.0
noinit 37160 37160 0 0.0
text 633404 633404 0 0.0

@mrjerryjohns
Copy link
Contributor

I see this (the error itself is because of a mis-matched all-clusters-app), where the STDERR and STDOUT log lines are mixed up:

Screen Shot 2022-03-22 at 10 13 50 AM

@mrjerryjohns
Copy link
Contributor

Ran this branch on my Mac, and confirmed that this now runs a heck of a lot faster,

@erjiaqing
Copy link
Contributor Author

I see this (the error itself is because of a mis-matched all-clusters-app), where the STDERR and STDOUT log lines are mixed up:

Screen Shot 2022-03-22 at 10 13 50 AM

This is a common issue when mixing the output of many output streams. This change should be done on the test script side. Probably by holding some exceptions and write them to stdout after potential large outputs.

Also, we can solve this case by case, since this single issue can be expressed in one line infact. And probably some other exceptions.

For now, is it possible to use some naive filters (e.g. grep '[TEST][STDERR]') to reorder the outputs.

@bzbarsky-apple bzbarsky-apple merged commit dd2f7e0 into project-chip:master Mar 23, 2022
andrei-menzopol pushed a commit to andrei-menzopol/connectedhomeip that referenced this pull request Apr 14, 2022
@erjiaqing @mrjerryjohns @andrei-menzopol
Remove over-killed output processing (project-chip#16519)
878d0f4 
Co-authored-by: Jerry Johns <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bzbarsky-apple bzbarsky-apple bzbarsky-apple approved these changes

@andy31415 andy31415 andy31415 approved these changes

@anush-apple anush-apple Awaiting requested review from anush-apple

@austinh0 austinh0 Awaiting requested review from austinh0

@Byungjoo-Lee Byungjoo-Lee Awaiting requested review from Byungjoo-Lee

@carol-apple carol-apple Awaiting requested review from carol-apple

@cecille cecille Awaiting requested review from cecille

@chrisdecenzo chrisdecenzo Awaiting requested review from chrisdecenzo

@chshu chshu Awaiting requested review from chshu

@chulspro chulspro Awaiting requested review from chulspro

@Damian-Nordic Damian-Nordic Awaiting requested review from Damian-Nordic

@dhrishi dhrishi Awaiting requested review from dhrishi

@electrocucaracha electrocucaracha Awaiting requested review from electrocucaracha

@franck-apple franck-apple Awaiting requested review from franck-apple

@gjc13 gjc13 Awaiting requested review from gjc13

@harimau-qirex harimau-qirex Awaiting requested review from harimau-qirex

@hawk248 hawk248 Awaiting requested review from hawk248

@holbrookt holbrookt Awaiting requested review from holbrookt

@harsha-rajendran harsha-rajendran Awaiting requested review from harsha-rajendran

@isiu-apple isiu-apple Awaiting requested review from isiu-apple

@jelderton jelderton Awaiting requested review from jelderton

@jepenven-silabs jepenven-silabs Awaiting requested review from jepenven-silabs

@jmartinez-silabs jmartinez-silabs Awaiting requested review from jmartinez-silabs

@lazarkov lazarkov Awaiting requested review from lazarkov

@LuDuda LuDuda Awaiting requested review from LuDuda

@mlepage-google mlepage-google Awaiting requested review from mlepage-google

@mrjerryjohns mrjerryjohns Awaiting requested review from mrjerryjohns

@msandstedt msandstedt Awaiting requested review from msandstedt

@mspang mspang Awaiting requested review from mspang

@sagar-apple sagar-apple Awaiting requested review from sagar-apple

@saurabhst saurabhst Awaiting requested review from saurabhst

@selissia selissia Awaiting requested review from selissia

@tecimovic tecimovic Awaiting requested review from tecimovic

@turon turon Awaiting requested review from turon

@vijs vijs Awaiting requested review from vijs

@vivien-apple vivien-apple Awaiting requested review from vivien-apple

@wbschiller wbschiller Awaiting requested review from wbschiller

@woody-apple woody-apple Awaiting requested review from woody-apple

@xylophone21 xylophone21 Awaiting requested review from xylophone21

@yufengwangca yufengwangca Awaiting requested review from yufengwangca

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation github review - approved scripts workflows
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@erjiaqing @mrjerryjohns @andy31415 @bzbarsky-apple