-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #60 from padovan/experimental-results-fetching
Experimental command to fetch results from the Dashboard API
- Loading branch information
Showing
9 changed files
with
445 additions
and
214 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| +++ | ||
| title = 'maestro-results' | ||
| date = 2024-01-14T07:07:07+01:00 | ||
| description = 'Command for show Maestro test results.' | ||
| +++ | ||
|
|
||
| This command is Maestro-specific and will show the test result by node id. | ||
|
|
||
| Do not use it, unless you are requesting test on Maestro through `kci-dev`. | ||
|
|
||
| Example: | ||
| ```sh | ||
| kci-dev maestro-results --nodeid <str: testnodeid> | ||
| ``` | ||
|
|
||
| This command will show the results of tests by page nodes limit and page offset. | ||
|
|
||
| Example: | ||
| ```sh | ||
| kci-dev maestro-results --nodes --limit <int: page nodes limit> --offset <int: page nodes offset> | ||
| ``` | ||
|
|
||
| Result sample: | ||
| ```yaml | ||
| {'artifacts': None, | ||
| 'created': '2024-10-04T00:49:15.691000', | ||
| 'data': {'arch': 'x86_64', | ||
| 'compiler': 'gcc-12', | ||
| 'config_full': 'x86_64_defconfig+lab-setup+x86-board', | ||
| 'defconfig': 'x86_64_defconfig', | ||
| 'device': None, | ||
| 'error_code': None, | ||
| 'error_msg': None, | ||
| 'job_context': None, | ||
| 'job_id': None, | ||
| 'kernel_revision': {'branch': 'staging-mainline', | ||
| 'commit': '232edaea0fd9b4b7feb7b43508834bba7e820584', | ||
| 'commit_message': 'staging-mainline-20241004.0', | ||
| 'commit_tags': ['staging-mainline-20241004.0'], | ||
| 'describe': 'staging-mainline-20241004.0', | ||
| 'patchset': None, | ||
| 'tip_of_branch': False, | ||
| 'tree': 'kernelci', | ||
| 'url': 'https://github.com/kernelci/linux.git', | ||
| 'version': {'extra': '-rc1-115-g232edaea0fd9b', | ||
| 'name': None, | ||
| 'patchlevel': 12, | ||
| 'sublevel': None, | ||
| 'version': 6}}, | ||
| 'kernel_type': 'bzimage', | ||
| 'platform': 'hp-14b-na0052xx-zork', | ||
| 'regression': None, | ||
| 'runtime': 'lava-collabora', | ||
| 'test_revision': None, | ||
| 'test_source': None}, | ||
| 'debug': None, | ||
| 'group': 'kselftest-exec', | ||
| 'holdoff': None, | ||
| 'id': '66ff3b8c0abcc4c8343d1c71', | ||
| 'jobfilter': None, | ||
| 'kind': 'test', | ||
| 'name': 'exec_execveat_Check_success_of_execveat_20_4096', | ||
| 'owner': 'staging.kernelci.org', | ||
| 'parent': '66ff3b8b0abcc4c8343d1b8a', | ||
| 'path': ['checkout', | ||
| 'kbuild-gcc-12-x86', | ||
| 'kselftest-exec', | ||
| 'kselftest-exec', | ||
| 'exec_execveat_Check_success_of_execveat_20_4096'], | ||
| 'result': 'pass', | ||
| 'state': 'done', | ||
| 'submitter': 'service:pipeline', | ||
| 'timeout': '2024-10-04T06:49:15.691000', | ||
| 'treeid': 'a44035dadc31327a5c30db4013b0e7e90acbb6a8fc45f94a6d91671e76cdfd8a', | ||
| 'updated': '2024-10-04T00:49:15.691000', | ||
| 'user_groups': []} | ||
| ``` | ||
| testnodeid is the node id of the test job, which you can get from the KernelCI dashboard. | ||
| Usually it is hexadecimal string. | ||
| Additionally, for --nodes you can provide optional parameters --filter to filter the results by the given key and value. | ||
| For example: | ||
| ```sh | ||
| ./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test | ||
| ``` | ||
| This command will show the nodes of tests in particular tree checkout. | ||
| But as you might see, there is a lot of fields you might be not interested in. | ||
|
|
||
| For this we have additional option --field, that will restrict output only to specified fields. | ||
| For example: | ||
| ```sh | ||
| ./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test --field name --field result | ||
| ``` | ||
| Example: | ||
|
|
||
| ```json | ||
| {'name': 'kver', 'result': 'pass'} | ||
| {'name': 'config', 'result': 'pass'} | ||
| {'name': 'build', 'result': 'pass'} | ||
| {'name': 'example_init_test', 'result': 'pass'} | ||
| {'name': 'time64_to_tm_test_date_range', 'result': 'pass'} | ||
| {'name': 'test_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_many_cpus', 'result': 'skip'} | ||
| {'name': 'test_one_task_on_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_one_task_on_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_one_task_mixed', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_one_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_task_on_all_and_one_cpu', 'result': 'skip'} | ||
| {'name': 'resource_test_union', 'result': 'pass'} | ||
| {'name': 'resource_test_intersection', 'result': 'pass'} | ||
| .... | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,113 +1,66 @@ | ||
| +++ | ||
| title = 'results' | ||
| date = 2024-01-14T07:07:07+01:00 | ||
| description = 'Command for show test results.' | ||
| date = 2024-12-10T07:07:07+01:00 | ||
| description = 'Fetch results from the KernelCI ecosystem.' | ||
| +++ | ||
|
|
||
| This command will show the test result by node id. | ||
| `kci-dev` pulls from our Dashboard API. As of now, it is an EXPERIMENTAL tooling under development with close collaboration from Linux kernel maintainers. | ||
|
|
||
| > KNOWN ISSUE: The Dashboard endpoint we are using returns a file of a few megabytes in size, so download may take | ||
| a few seconds, slowing down your usage of `kci-dev results`. We are working on [it](https://github.com/kernelci/dashboard/issues/661). | ||
|
|
||
| ## Base parameters | ||
|
|
||
| ### --origin | ||
|
|
||
| Set the KCIDB origin desired. 'maestro' is the default. | ||
|
|
||
| ### --giturl | ||
|
|
||
| The url of the tree to fetch results | ||
|
|
||
| ### --branch | ||
|
|
||
| The branch to get results for | ||
|
|
||
| ### --commit | ||
|
|
||
| The tip of tree commit being tested. It needs to be the full commit hash. | ||
|
|
||
| Unfortunately the Dashboard API doesn't support git tags as parameters yet. | ||
|
|
||
| ## Results actions | ||
|
|
||
| ### --action=summary | ||
|
|
||
| Shows a numeric summary of the build, boot and test results. | ||
| If `--action` is omitted, it will show the summary by default. | ||
|
|
||
| Example: | ||
|
|
||
| ```sh | ||
| kci-dev results --nodeid <str: testnodeid> | ||
| kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action=summary | ||
| ``` | ||
|
|
||
| This command will show the results of tests by page nodes limit and page offset. | ||
| ### --action=failed-builds | ||
|
|
||
| List failed builds. | ||
|
|
||
| Example: | ||
|
|
||
| ```sh | ||
| kci-dev results --nodes --limit <int: page nodes limit> --offset <int: page nodes offset> | ||
| kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action failed-builds | ||
| ``` | ||
|
|
||
| Result sample: | ||
| ```yaml | ||
| {'artifacts': None, | ||
| 'created': '2024-10-04T00:49:15.691000', | ||
| 'data': {'arch': 'x86_64', | ||
| 'compiler': 'gcc-12', | ||
| 'config_full': 'x86_64_defconfig+lab-setup+x86-board', | ||
| 'defconfig': 'x86_64_defconfig', | ||
| 'device': None, | ||
| 'error_code': None, | ||
| 'error_msg': None, | ||
| 'job_context': None, | ||
| 'job_id': None, | ||
| 'kernel_revision': {'branch': 'staging-mainline', | ||
| 'commit': '232edaea0fd9b4b7feb7b43508834bba7e820584', | ||
| 'commit_message': 'staging-mainline-20241004.0', | ||
| 'commit_tags': ['staging-mainline-20241004.0'], | ||
| 'describe': 'staging-mainline-20241004.0', | ||
| 'patchset': None, | ||
| 'tip_of_branch': False, | ||
| 'tree': 'kernelci', | ||
| 'url': 'https://github.com/kernelci/linux.git', | ||
| 'version': {'extra': '-rc1-115-g232edaea0fd9b', | ||
| 'name': None, | ||
| 'patchlevel': 12, | ||
| 'sublevel': None, | ||
| 'version': 6}}, | ||
| 'kernel_type': 'bzimage', | ||
| 'platform': 'hp-14b-na0052xx-zork', | ||
| 'regression': None, | ||
| 'runtime': 'lava-collabora', | ||
| 'test_revision': None, | ||
| 'test_source': None}, | ||
| 'debug': None, | ||
| 'group': 'kselftest-exec', | ||
| 'holdoff': None, | ||
| 'id': '66ff3b8c0abcc4c8343d1c71', | ||
| 'jobfilter': None, | ||
| 'kind': 'test', | ||
| 'name': 'exec_execveat_Check_success_of_execveat_20_4096', | ||
| 'owner': 'staging.kernelci.org', | ||
| 'parent': '66ff3b8b0abcc4c8343d1b8a', | ||
| 'path': ['checkout', | ||
| 'kbuild-gcc-12-x86', | ||
| 'kselftest-exec', | ||
| 'kselftest-exec', | ||
| 'exec_execveat_Check_success_of_execveat_20_4096'], | ||
| 'result': 'pass', | ||
| 'state': 'done', | ||
| 'submitter': 'service:pipeline', | ||
| 'timeout': '2024-10-04T06:49:15.691000', | ||
| 'treeid': 'a44035dadc31327a5c30db4013b0e7e90acbb6a8fc45f94a6d91671e76cdfd8a', | ||
| 'updated': '2024-10-04T00:49:15.691000', | ||
| 'user_groups': []} | ||
| ``` | ||
| ## Downloading logs | ||
|
|
||
| testnodeid is the node id of the test job, which you can get from the KernelCI dashboard. | ||
| Usually it is hexadecimal string. | ||
| `--download-logs` Download failed logs when used with `--action=failed-*` commands. | ||
|
|
||
| Additionally, for --nodes you can provide optional parameters --filter to filter the results by the given key and value. | ||
| For example: | ||
| Example: | ||
| ```sh | ||
| ./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test | ||
| kci-dev results --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --commit d1486dca38afd08ca279ae94eb3a397f10737824 --action failed-builds --download-logs | ||
| ``` | ||
| This command will show the nodes of tests in particular tree checkout. | ||
| But as you might see, there is a lot of fields you might be not interested in. | ||
|
|
||
| For this we have additional option --field, that will restrict output only to specified fields. | ||
| For example: | ||
| ```sh | ||
| ./kci-dev.py results --nodes --filter treeid=e25266f77837de335edba3c1b8d2a04edc2bfb195b77c44711d81ebea4494140 --filter kind=test --field name --field result | ||
| ``` | ||
| Example: | ||
|
|
||
| ```json | ||
| {'name': 'kver', 'result': 'pass'} | ||
| {'name': 'config', 'result': 'pass'} | ||
| {'name': 'build', 'result': 'pass'} | ||
| {'name': 'example_init_test', 'result': 'pass'} | ||
| {'name': 'time64_to_tm_test_date_range', 'result': 'pass'} | ||
| {'name': 'test_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_many_cpus', 'result': 'skip'} | ||
| {'name': 'test_one_task_on_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_one_task_on_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_one_task_mixed', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_one_cpu', 'result': 'skip'} | ||
| {'name': 'test_two_tasks_on_one_all_cpus', 'result': 'skip'} | ||
| {'name': 'test_task_on_all_and_one_cpu', 'result': 'skip'} | ||
| {'name': 'resource_test_union', 'result': 'pass'} | ||
| {'name': 'resource_test_intersection', 'result': 'pass'} | ||
| .... | ||
| ``` | ||
|
|
||
|
|
Oops, something went wrong.