Added method in python runner to get eventNumber for TC-SMOKECO tests #28704
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericzijian1994
changed the title
pullapprove
bot
requested review from
andyg-apple,
anush-apple,
arkq,
bzbarsky-apple,
carol-apple,
cecille,
chrisdecenzo,
chshu,
chulspro,
cliffamzn,
Damian-Nordic,
dhrishi,
electrocucaracha,
gjc13,
harsha-rajendran,
hawk248,
hicklin,
jepenven-silabs,
jmartinez-silabs,
jmeg-sfy,
joonhaengHeo,
jtung-apple and
kkasperczyk-no
ericzijian1994
changed the title
|
PR #28704: Size comparison from 8060222 to 65d4f29 Increases (1 build for bl702l)
Decreases (1 build for efr32)
Full report (62 builds for bl602, bl702, bl702l, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
ericzijian1994
changed the title
andy31415
requested changes
Aug 23, 2023
|
PR #28704: Size comparison from 8060222 to 7874d39 Increases above 0.2%:
Increases (3 builds for linux, nrfconnect)
Decreases (1 build for cc32xx)
Full report (12 builds for cc32xx, k32w, linux, mbed, nrfconnect, qpg)
|
|
PR #28704: Size comparison from 8060222 to e1bbbc3 Increases above 0.2%:
Increases (9 builds for bl702l, linux, nrfconnect, psoc6, telink)
Decreases (14 builds for bl702, bl702l, cc32xx, efr32, esp32, linux, psoc6)
Full report (62 builds for bl602, bl702, bl702l, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
andy31415
reviewed
Aug 24, 2023
andy31415
reviewed
Aug 24, 2023
|
PR #28704: Size comparison from 8060222 to 923ea3c Increases above 0.2%:
Increases (10 builds for esp32, linux, nrfconnect, psoc6, telink)
Decreases (32 builds for bl702, bl702l, cc32xx, efr32, esp32, linux, psoc6, qpg, telink)
Full report (62 builds for bl602, bl702, bl702l, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, telink)
|
yunhanw-google
approved these changes
Aug 24, 2023
andy31415
approved these changes
Aug 24, 2023
This was referenced Aug 25, 2023
abpoth
pushed a commit
to abpoth/connectedhomeip
that referenced
this pull request
Aug 25, 2023
…project-chip#28704) * Add method to get last event number in python * Update TC-SMOKECO tests * Retain event number when reading events * Improve the method of get event number * Restyled by autopep8 * Add variable checking * Update name --------- Co-authored-by: Hare <[email protected]> Co-authored-by: Restyled.io <[email protected]>
HunsupJung
pushed a commit
to HunsupJung/connectedhomeip
that referenced
this pull request
Oct 23, 2023
…project-chip#28704) * Add method to get last event number in python * Update TC-SMOKECO tests * Retain event number when reading events * Improve the method of get event number * Restyled by autopep8 * Add variable checking * Update name --------- Co-authored-by: Hare <[email protected]> Co-authored-by: Restyled.io <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposing the method to fixes #27539
Adding a method in Python runner to get eventNumber in order to get the TC-SMOKECO test to work, I understand this could not be a good implementation and may need improvement/use another alternative.
The
readEventcommand from eventTest returns a list of events. But in yaml, the value I expect needs to match the value actually returned. And the eventNumber parameter in yaml acts as a filter that allows me to accomplish this.If I know the event number before a readEvent step, I can get the latest one located and read the event generated after this eventNumber to pass the test plan, which also makes it easier for automation.
In the testing of the smoke and CO Test plan, we will verify a series of new events that are received. The EventNumber is used to distinguish whether an event is the most recent one. However, the issue lies in the fact that the EventNumber provided in the testEvents currently relies on an initial eventNumber in a configuration file. (I have set it to 0 in the config, which results in retrieving all records.)
Since I didn't find certain commands only returned the latest event sequence number. I can't do the effective update to flag the new starting point with the EventNumber to search event.
The readEvent returns a list of all corresponding events found. Due to the fact that the test plan will test multiple times for different scenarios triggering the same event, this leads to the continuous growth of the returned list... Consequently, if testing persists, the returned records will become unpredictable and lead to test failure.
Example test Ymal:
Until then, I've tried to modify EVENT_NUMBER at config and letting testers enter it manually.
But there are some issues with using the eventNumber parameter in TC-SMOKECO:
So this would need a new method to get the latest event automatically.
My logic for the change is to
last_event_numbervariable to the python runner to record the latest event number of the readEvent.readEventwith no response to let the last_event_number have a value.isLastEventNumberparameter is true, the eventNumber parameter is set to last_event_number.This will make a method for tracking the last event number. By running tests, this method worked with the PR proposed here.