Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it possible for apps into the placeholder directory to have cust… #11332

Conversation

vivien-apple
Copy link
Contributor

…om tests

Problem

Applications from examples/placeholder, used as simulated devices, needs a way to specify custom tests to be generated.

Change overview

  • Update examples/chip-tool/templates/partials/test_cluster.zapt to support wait commands
  • Expose a getTests method to placeholder apps so they can specify tests individually
  • Add test generation

Testing

The simulated device has some generated tests. I have manually validate that those expectations fails or succeeds depending on the commands sent to the device.

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

PR #11332: Size comparison from e788c90 to e8211de

Increases (6 builds for esp32, linux)
platform target config section e788c906 e8211de change % change
esp32 all-clusters-app m5stack (read only) 911495 911571 76 0.0
.flash.text 911495 911571 76 0.0
linux all-clusters-app debug (read only) 1690385 1690537 152 0.0
.rodata 139637 139733 96 0.1
.text 1417378 1417410 32 0.0
lighting-app debug+rpc (read only) 1542617 1542785 168 0.0
.rodata 129297 129393 96 0.1
.text 1280626 1280674 48 0.0
ota-provider-app debug (read only) 1243145 1243313 168 0.0
.rodata 112136 112232 96 0.1
.text 1034930 1034978 48 0.0
ota-requestor-app debug (read only) 1310441 1310593 152 0.0
.rodata 123104 123200 96 0.1
.text 1089618 1089650 32 0.0
tv-app debug (read only) 1790633 1790801 168 0.0
(read/write) 289240 289272 32 0.0
.bss 222736 222768 32 0.0
.rodata 155032 155128 96 0.1
.text 1493410 1493458 48 0.0
Full report (37 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
platform target config section e788c906 e8211de change % change
efr32 lighting-app BRD4161A (read only) 736844 736844 0 0.0
(read/write) 114612 114612 0 0.0
.bss 112852 112852 0 0.0
.data 1760 1760 0 0.0
.text 736836 736836 0 0.0
BRD4161A+rpc (read only) 724268 724268 0 0.0
(read/write) 131216 131216 0 0.0
.bss 129356 129356 0 0.0
.data 1860 1860 0 0.0
.text 724260 724260 0 0.0
lock-app BRD4161A (read only) 716124 716124 0 0.0
(read/write) 112428 112428 0 0.0
.bss 110708 110708 0 0.0
.data 1720 1720 0 0.0
.text 716116 716116 0 0.0
window-app BRD4161A (read only) 717024 717024 0 0.0
(read/write) 112756 112756 0 0.0
.bss 111028 111028 0 0.0
.data 1724 1724 0 0.0
.text 717016 717016 0 0.0
esp32 all-clusters-app c3devkit (read only) 880466 880466 0 0.0
(read/write) 1307824 1307824 0 0.0
.dram0.bss 58592 58592 0 0.0
.dram0.data 16464 16464 0 0.0
.flash.rodata 199536 199536 0 0.0
.flash.text 880466 880466 0 0.0
.iram0.text 57554 57554 0 0.0
m5stack (read only) 911495 911571 76 0.0
(read/write) 427592 427592 0 0.0
.dram0.bss 61088 61088 0 0.0
.dram0.data 32100 32100 0 0.0
.flash.rodata 208240 208240 0 0.0
.flash.text 911495 911571 76 0.0
.iram0.text 125115 125115 0 0.0
k32w lighting-app k32w061+se05x+release (read/write) 699104 699104 0 0.0
.bss 77848 77848 0 0.0
.data 1908 1908 0 0.0
.text 613548 613548 0 0.0
lock-app k32w061+debug (read/write) 591412 591412 0 0.0
.bss 68364 68364 0 0.0
.data 1872 1872 0 0.0
.text 515376 515376 0 0.0
shell k32w061+debug (read/write) 424800 424800 0 0.0
.bss 63288 63288 0 0.0
.data 672 672 0 0.0
.text 359136 359136 0 0.0
linux all-clusters-app debug (read only) 1690385 1690537 152 0.0
(read/write) 119664 119664 0 0.0
.bss 51120 51120 0 0.0
.data 1042 1042 0 0.0
.data.rel.ro 62240 62240 0 0.0
.dynamic 592 592 0 0.0
.got 4088 4088 0 0.0
.init 27 27 0 0.0
.init_array 536 536 0 0.0
.rodata 139637 139733 96 0.1
.text 1417378 1417410 32 0.0
bridge-app debug+rpc (read only) 1283293 1283293 0 0.0
(read/write) 68488 68488 0 0.0
.bss 35024 35024 0 0.0
.data 992 992 0 0.0
.data.rel.ro 27496 27496 0 0.0
.dynamic 592 592 0 0.0
.got 3952 3952 0 0.0
.init 27 27 0 0.0
.init_array 400 400 0 0.0
.rodata 111204 111204 0 0.0
.text 1076533 1076533 0 0.0
chip-tool debug (read only) 4325869 4325869 0 0.0
(read/write) 123936 123936 0 0.0
.bss 18192 18192 0 0.0
.data 1584 1584 0 0.0
.data.rel.ro 98744 98744 0 0.0
.dynamic 592 592 0 0.0
.got 4368 4368 0 0.0
.init 27 27 0 0.0
.init_array 424 424 0 0.0
.rodata 218832 218832 0 0.0
.text 3841621 3841621 0 0.0
lighting-app debug+rpc (read only) 1542617 1542785 168 0.0
(read/write) 101680 101680 0 0.0
.bss 40696 40696 0 0.0
.data 1138 1138 0 0.0
.data.rel.ro 54560 54560 0 0.0
.dynamic 608 608 0 0.0
.got 4112 4112 0 0.0
.init 27 27 0 0.0
.init_array 528 528 0 0.0
.rodata 129297 129393 96 0.1
.text 1280626 1280674 48 0.0
ota-provider-app debug (read only) 1243145 1243313 168 0.0
(read/write) 67520 67520 0 0.0
.bss 37120 37120 0 0.0
.data 752 752 0 0.0
.data.rel.ro 24568 24568 0 0.0
.dynamic 592 592 0 0.0
.got 4016 4016 0 0.0
.init 27 27 0 0.0
.init_array 440 440 0 0.0
.rodata 112136 112232 96 0.1
.text 1034930 1034978 48 0.0
ota-requestor-app debug (read only) 1310441 1310593 152 0.0
(read/write) 76840 76840 0 0.0
.bss 45376 45376 0 0.0
.data 816 816 0 0.0
.data.rel.ro 25512 25512 0 0.0
.dynamic 592 592 0 0.0
.got 3984 3984 0 0.0
.init 27 27 0 0.0
.init_array 512 512 0 0.0
.rodata 123104 123200 96 0.1
.text 1089618 1089650 32 0.0
shell debug (read only) 786529 786529 0 0.0
(read/write) 57664 57664 0 0.0
.bss 16104 16104 0 0.0
.data 242 242 0 0.0
.data.rel.ro 36816 36816 0 0.0
.dynamic 592 592 0 0.0
.got 3528 3528 0 0.0
.init 27 27 0 0.0
.init_array 336 336 0 0.0
.rodata 77839 77839 0 0.0
.text 607202 607202 0 0.0
tv-app debug (read only) 1790633 1790801 168 0.0
(read/write) 289240 289272 32 0.0
.bss 222736 222768 32 0.0
.data 2064 2064 0 0.0
.data.rel.ro 58784 58784 0 0.0
.dynamic 592 592 0 0.0
.got 4408 4408 0 0.0
.init 27 27 0 0.0
.init_array 608 608 0 0.0
.rodata 155032 155128 96 0.1
.text 1493410 1493458 48 0.0
mbed all-clusters-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2292704 2292704 0 0.0
.bss 179956 179956 0 0.0
.data 5224 5224 0 0.0
.heap 851264 851264 0 0.0
.text 1255304 1255304 0 0.0
lighting-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2271648 2271648 0 0.0
.bss 172004 172004 0 0.0
.data 5576 5576 0 0.0
.heap 858864 858864 0 0.0
.text 1234248 1234248 0 0.0
lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2249368 2249368 0 0.0
.bss 170900 170900 0 0.0
.data 5560 5560 0 0.0
.heap 859984 859984 0 0.0
.text 1211968 1211968 0 0.0
pigweed-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 1139744 1139744 0 0.0
.bss 11752 11752 0 0.0
.data 4368 4368 0 0.0
.heap 1020328 1020328 0 0.0
.text 103128 103128 0 0.0
shell CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2048432 2048432 0 0.0
.bss 156248 156248 0 0.0
.data 4968 4968 0 0.0
.heap 875232 875232 0 0.0
.text 1011032 1011032 0 0.0
nrfconnect lighting-app nrf52840dk_nrf52840 (read/write) 861027 861027 0 0.0
bss 111312 111312 0 0.0
rodata 96548 96548 0 0.0
text 577548 577548 0 0.0
nrf52840dk_nrf52840+rpc (read/write) 821851 821851 0 0.0
bss 107552 107552 0 0.0
rodata 87324 87324 0 0.0
text 550720 550720 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 786050 786050 0 0.0
bss 112688 112688 0 0.0
rodata 91788 91788 0 0.0
text 507012 507012 0 0.0
lock-app nrf52840dk_nrf52840 (read/write) 837731 837731 0 0.0
bss 110348 110348 0 0.0
rodata 92900 92900 0 0.0
text 559028 559028 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 763030 763030 0 0.0
bss 111756 111756 0 0.0
rodata 88208 88208 0 0.0
text 488584 488584 0 0.0
pigweed-app nrf52840dk_nrf52840 (read/write) 497323 497323 0 0.0
bss 51824 51824 0 0.0
rodata 45776 45776 0 0.0
text 339436 339436 0 0.0
pump-app nrf52840dk_nrf52840 (read/write) 843843 843843 0 0.0
bss 110484 110484 0 0.0
rodata 94612 94612 0 0.0
text 563208 563208 0 0.0
pump-controller-app nrf52840dk_nrf52840 (read/write) 837587 837587 0 0.0
bss 110384 110384 0 0.0
rodata 92900 92900 0 0.0
text 558764 558764 0 0.0
shell nrf52840dk_nrf52840 (read/write) 775911 775911 0 0.0
bss 109104 109104 0 0.0
rodata 72412 72412 0 0.0
text 519804 519804 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 690962 690962 0 0.0
bss 110084 110084 0 0.0
rodata 67052 67052 0 0.0
text 440404 440404 0 0.0
p6 lock-app default (read/write) 2167680 2167680 0 0.0
.bss 66352 66352 0 0.0
.data 2424 2424 0 0.0
.heap 964568 964568 0 0.0
.text 1125944 1125944 0 0.0
qpg lighting-app qpg6100+debug (read only) 490384 490384 0 0.0
(read/write) 114144 114144 0 0.0
.bss 50408 50408 0 0.0
.data 1008 1008 0 0.0
.text 485064 485064 0 0.0
lock-app qpg6100+debug (read only) 466720 466720 0 0.0
(read/write) 114140 114140 0 0.0
.bss 49352 49352 0 0.0
.data 964 964 0 0.0
.text 461400 461400 0 0.0
persistent-storage-app qpg6100+debug (read only) 155796 155796 0 0.0
(read/write) 114140 114140 0 0.0
.bss 27760 27760 0 0.0
.data 372 372 0 0.0
.text 150476 150476 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 662422 662422 0 0.0
bss 69116 69116 0 0.0
noinit 33216 33216 0 0.0
text 457800 457800 0 0.0

@vivien-apple
Copy link
Contributor Author

/rebase

@woody-apple woody-apple force-pushed the YAML_Generate_Simulated_Device_Tests branch from e8211de to 2531dc4 Compare November 3, 2021 00:33
@github-actions
Copy link

github-actions bot commented Nov 3, 2021

PR #11332: Size comparison from c9db3dc to 2531dc4

Increases (5 builds for linux)
platform target config section c9db3dc 2531dc4 change % change
linux all-clusters-app debug (read only) 1690913 1691065 152 0.0
.rodata 139861 139957 96 0.1
.text 1417682 1417714 32 0.0
lighting-app debug+rpc (read only) 1543193 1543345 152 0.0
.rodata 129553 129649 96 0.1
.text 1280946 1280978 32 0.0
ota-provider-app debug (read only) 1243689 1243841 152 0.0
.rodata 112360 112456 96 0.1
.text 1035250 1035282 32 0.0
ota-requestor-app debug (read only) 1311049 1311201 152 0.0
.rodata 123328 123424 96 0.1
.text 1089954 1089986 32 0.0
tv-app debug (read only) 1810545 1810697 152 0.0
(read/write) 289944 289976 32 0.0
.bss 222768 222800 32 0.0
.rodata 155640 155736 96 0.1
.text 1512642 1512674 32 0.0
Full report (37 builds for efr32, esp32, k32w, linux, mbed, nrfconnect, p6, qpg, telink)
platform target config section c9db3dc 2531dc4 change % change
efr32 lighting-app BRD4161A (read only) 737248 737248 0 0.0
(read/write) 114620 114620 0 0.0
.bss 112860 112860 0 0.0
.data 1756 1756 0 0.0
.text 737240 737240 0 0.0
BRD4161A+rpc (read only) 724672 724672 0 0.0
(read/write) 131220 131220 0 0.0
.bss 129364 129364 0 0.0
.data 1856 1856 0 0.0
.text 724664 724664 0 0.0
lock-app BRD4161A (read only) 716528 716528 0 0.0
(read/write) 112436 112436 0 0.0
.bss 110716 110716 0 0.0
.data 1716 1716 0 0.0
.text 716520 716520 0 0.0
window-app BRD4161A (read only) 717428 717428 0 0.0
(read/write) 112756 112756 0 0.0
.bss 111036 111036 0 0.0
.data 1720 1720 0 0.0
.text 717420 717420 0 0.0
esp32 all-clusters-app c3devkit (read only) 880718 880718 0 0.0
(read/write) 1308080 1308080 0 0.0
.dram0.bss 58608 58608 0 0.0
.dram0.data 16464 16464 0 0.0
.flash.rodata 199776 199776 0 0.0
.flash.text 880718 880718 0 0.0
.iram0.text 57554 57554 0 0.0
m5stack (read only) 911895 911895 0 0.0
(read/write) 427852 427852 0 0.0
.dram0.bss 61112 61112 0 0.0
.dram0.data 32100 32100 0 0.0
.flash.rodata 208476 208476 0 0.0
.flash.text 911895 911895 0 0.0
.iram0.text 125115 125115 0 0.0
k32w lighting-app k32w061+se05x+release (read/write) 699528 699528 0 0.0
.bss 77856 77856 0 0.0
.data 1904 1904 0 0.0
.text 613968 613968 0 0.0
lock-app k32w061+debug (read/write) 591836 591836 0 0.0
.bss 68372 68372 0 0.0
.data 1868 1868 0 0.0
.text 515796 515796 0 0.0
shell k32w061+debug (read/write) 424916 424916 0 0.0
.bss 63296 63296 0 0.0
.data 668 668 0 0.0
.text 359248 359248 0 0.0
linux all-clusters-app debug (read only) 1690913 1691065 152 0.0
(read/write) 119696 119696 0 0.0
.bss 51152 51152 0 0.0
.data 1042 1042 0 0.0
.data.rel.ro 62240 62240 0 0.0
.dynamic 592 592 0 0.0
.got 4088 4088 0 0.0
.init 27 27 0 0.0
.init_array 536 536 0 0.0
.rodata 139861 139957 96 0.1
.text 1417682 1417714 32 0.0
bridge-app debug+rpc (read only) 1283869 1283869 0 0.0
(read/write) 68520 68520 0 0.0
.bss 35056 35056 0 0.0
.data 992 992 0 0.0
.data.rel.ro 27496 27496 0 0.0
.dynamic 592 592 0 0.0
.got 3952 3952 0 0.0
.init 27 27 0 0.0
.init_array 400 400 0 0.0
.rodata 111460 111460 0 0.0
.text 1076853 1076853 0 0.0
chip-tool debug (read only) 4349141 4349141 0 0.0
(read/write) 124768 124768 0 0.0
.bss 18224 18224 0 0.0
.data 2224 2224 0 0.0
.data.rel.ro 98896 98896 0 0.0
.dynamic 592 592 0 0.0
.got 4368 4368 0 0.0
.init 27 27 0 0.0
.init_array 424 424 0 0.0
.rodata 219528 219528 0 0.0
.text 3863909 3863909 0 0.0
lighting-app debug+rpc (read only) 1543193 1543345 152 0.0
(read/write) 101712 101712 0 0.0
.bss 40728 40728 0 0.0
.data 1138 1138 0 0.0
.data.rel.ro 54560 54560 0 0.0
.dynamic 608 608 0 0.0
.got 4112 4112 0 0.0
.init 27 27 0 0.0
.init_array 528 528 0 0.0
.rodata 129553 129649 96 0.1
.text 1280946 1280978 32 0.0
ota-provider-app debug (read only) 1243689 1243841 152 0.0
(read/write) 67552 67552 0 0.0
.bss 37152 37152 0 0.0
.data 752 752 0 0.0
.data.rel.ro 24568 24568 0 0.0
.dynamic 592 592 0 0.0
.got 4016 4016 0 0.0
.init 27 27 0 0.0
.init_array 440 440 0 0.0
.rodata 112360 112456 96 0.1
.text 1035250 1035282 32 0.0
ota-requestor-app debug (read only) 1311049 1311201 152 0.0
(read/write) 76872 76872 0 0.0
.bss 45408 45408 0 0.0
.data 816 816 0 0.0
.data.rel.ro 25512 25512 0 0.0
.dynamic 592 592 0 0.0
.got 3984 3984 0 0.0
.init 27 27 0 0.0
.init_array 512 512 0 0.0
.rodata 123328 123424 96 0.1
.text 1089954 1089986 32 0.0
shell debug (read only) 786593 786593 0 0.0
(read/write) 57664 57664 0 0.0
.bss 16104 16104 0 0.0
.data 242 242 0 0.0
.data.rel.ro 36816 36816 0 0.0
.dynamic 592 592 0 0.0
.got 3528 3528 0 0.0
.init 27 27 0 0.0
.init_array 336 336 0 0.0
.rodata 77839 77839 0 0.0
.text 607266 607266 0 0.0
tv-app debug (read only) 1810545 1810697 152 0.0
(read/write) 289944 289976 32 0.0
.bss 222768 222800 32 0.0
.data 2704 2704 0 0.0
.data.rel.ro 58824 58824 0 0.0
.dynamic 592 592 0 0.0
.got 4408 4408 0 0.0
.init 27 27 0 0.0
.init_array 608 608 0 0.0
.rodata 155640 155736 96 0.1
.text 1512642 1512674 32 0.0
mbed all-clusters-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2293216 2293216 0 0.0
.bss 179964 179964 0 0.0
.data 5216 5216 0 0.0
.heap 851264 851264 0 0.0
.text 1255816 1255816 0 0.0
lighting-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2272216 2272216 0 0.0
.bss 172012 172012 0 0.0
.data 5568 5568 0 0.0
.heap 858864 858864 0 0.0
.text 1234816 1234816 0 0.0
lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2249872 2249872 0 0.0
.bss 170908 170908 0 0.0
.data 5560 5560 0 0.0
.heap 859976 859976 0 0.0
.text 1212472 1212472 0 0.0
pigweed-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 1139744 1139744 0 0.0
.bss 11752 11752 0 0.0
.data 4368 4368 0 0.0
.heap 1020328 1020328 0 0.0
.text 103128 103128 0 0.0
shell CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2048496 2048496 0 0.0
.bss 156256 156256 0 0.0
.data 4960 4960 0 0.0
.heap 875232 875232 0 0.0
.text 1011096 1011096 0 0.0
nrfconnect lighting-app nrf52840dk_nrf52840 (read/write) 861395 861395 0 0.0
bss 111320 111320 0 0.0
rodata 96772 96772 0 0.0
text 577692 577692 0 0.0
nrf52840dk_nrf52840+rpc (read/write) 822267 822267 0 0.0
bss 107560 107560 0 0.0
rodata 87564 87564 0 0.0
text 550864 550864 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 786434 786434 0 0.0
bss 112696 112696 0 0.0
rodata 92028 92028 0 0.0
text 507152 507152 0 0.0
lock-app nrf52840dk_nrf52840 (read/write) 838099 838099 0 0.0
bss 110356 110356 0 0.0
rodata 93140 93140 0 0.0
text 559168 559168 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 763398 763398 0 0.0
bss 111764 111764 0 0.0
rodata 88448 88448 0 0.0
text 488720 488720 0 0.0
pigweed-app nrf52840dk_nrf52840 (read/write) 497323 497323 0 0.0
bss 51824 51824 0 0.0
rodata 45776 45776 0 0.0
text 339436 339436 0 0.0
pump-app nrf52840dk_nrf52840 (read/write) 844211 844211 0 0.0
bss 110492 110492 0 0.0
rodata 94852 94852 0 0.0
text 563348 563348 0 0.0
pump-controller-app nrf52840dk_nrf52840 (read/write) 837955 837955 0 0.0
bss 110392 110392 0 0.0
rodata 93140 93140 0 0.0
text 558904 558904 0 0.0
shell nrf52840dk_nrf52840 (read/write) 775991 775991 0 0.0
bss 109112 109112 0 0.0
rodata 72412 72412 0 0.0
text 519884 519884 0 0.0
nrf5340dk_nrf5340_cpuapp (read/write) 691026 691026 0 0.0
bss 110092 110092 0 0.0
rodata 67052 67052 0 0.0
text 440480 440480 0 0.0
p6 lock-app default (read/write) 2168104 2168104 0 0.0
.bss 66360 66360 0 0.0
.data 2416 2416 0 0.0
.heap 964568 964568 0 0.0
.text 1126368 1126368 0 0.0
qpg lighting-app qpg6100+debug (read only) 490784 490784 0 0.0
(read/write) 114140 114140 0 0.0
.bss 50416 50416 0 0.0
.data 1004 1004 0 0.0
.text 485464 485464 0 0.0
lock-app qpg6100+debug (read only) 467120 467120 0 0.0
(read/write) 114144 114144 0 0.0
.bss 49360 49360 0 0.0
.data 960 960 0 0.0
.text 461800 461800 0 0.0
persistent-storage-app qpg6100+debug (read only) 153212 153212 0 0.0
(read/write) 114140 114140 0 0.0
.bss 18336 18336 0 0.0
.data 356 356 0 0.0
.text 147892 147892 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 662894 662894 0 0.0
bss 69132 69132 0 0.0
noinit 33216 33216 0 0.0
text 458028 458028 0 0.0

@woody-apple
Copy link
Contributor

Fast tracking this, as it's adding tests.

@woody-apple woody-apple merged commit 1093d25 into project-chip:master Nov 3, 2021
PSONALl pushed a commit to PSONALl/connectedhomeip that referenced this pull request Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants