Add feature kernelci-try #2594

aliceinwire · 2024-07-01T08:54:30Z

Add a feature similar to buildbot try where Kernel developers provided of a API key can push request for builds to the KernelCI current running environment from the command line.
Also add the possibility of pushing local kernel source changes.

aliceinwire · 2024-07-03T02:58:49Z

Following this issue
I propose to create a tool called kci-dev

$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes

This file will also accept a .kci-dev.yaml configuration file for define password and api keys
The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.

nuclearcat · 2024-07-03T07:28:13Z

Hi! Thats exactly what we are working on this milestone.
We implemented already resubmission for different kernel branch commit, i will implement also lab test retry logic and then we will move on discussing how to implement this tool. I will update issue with follow up on design, before implementing it.

aliceinwire · 2024-07-03T08:36:23Z

@nuclearcat which milestone are you talking about, there is any way I can help out?

pawiecz · 2024-07-03T09:58:10Z

@aliceinwire The milestone @nuclearcat mentioned is a public roadmap/development plan, e.g. M3 document

Kernel developers provided of a API key can push request for builds to the KernelCI current running environment from the command line.

This should be already possible for monitored trees/branches by submitting custom Node to the API

Also add the possibility of pushing local kernel source changes.

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

aliceinwire · 2024-07-03T10:06:40Z

what about the publish feature that permit to decide to publish or don't publish results

	--publish           define if the test results will be published

aliceinwire · 2024-07-03T10:08:47Z

This should be already possible for monitored trees/branches by submitting custom Node to the API

do you mean by using the KernelCI-api ? I tried doing it with the help of @nuclearcat but looks like it was not a feature that KernelCI-api can provide.

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

?

nuclearcat · 2024-07-03T10:32:25Z

@nuclearcat which milestone are you talking about, there is any way I can help out?
We're currently planning our work for the coming weeks and prioritizing new tasks, including:

We may add more issues based on logical and demanded features (including this one). Additional ideas will be gathered during our upcoming weekly meeting.

Once planning is complete, we'll publish a Milestone document outlining the defined tasks. We'll strive to complete these tasks within the milestone, but some may extend beyond due to external dependencies or underestimated complexity.

pawiecz · 2024-07-03T17:58:41Z

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

?

API supports Nodes with patches stored as artifacts. Kernel sources that given Node references together with these patches are later packaged into tarball for further processing.

By CLI I meant kci tool which should be extended to cover patchset-related use cases.

nuclearcat · 2024-07-11T09:11:38Z

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way.
All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)

So request can be as following:
POST /api/checkout

{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}

Will return custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can
answer to bisect good/bad.

P2)Custom test run retry

POST /api/retry

{
    "node": "<failing_test_node_id>"
}

Not sure yet, we cannot return node id in this case, but i can return kbuild node id that will have new "child" node with test run results.

Purpose: Sometimes hardware or tests are "flakey" (unreliable), so it is useful to retry test several times.

P3) Test fix(patch)

POST /api/testfix

{
    "node": "<failing_test_node_id>",
    "patch": "<patch_content_base64?>"
}

It will fetch same sources as test failed, but apply patch before build. Then after building patched kernel it will run test.
It will return as in p1 custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can test patches on kernelci infrastructure, if they fix failing test/regression.

P4) Testing new patches

POST /api/patch

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}

Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.

It will fetch sources from treeurl, checkout to branch, then to commit, apply patch.
Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.

Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

nuclearcat · 2024-07-11T09:30:38Z

I updated draft

aliceinwire · 2024-07-12T04:57:49Z

@nuclearcat thanks for the draft of the API, as discussed on the weekly web meeting this is a good start.
my further suggestion was:

Get the results as json file or at least get the build id number for getting the results from other sources (like KCIDB)
Add a optional email option for send the results by email
Add a privacy option for don't publish the results to KCIDB or KernelCI dashboard

padovan · 2024-07-25T11:39:11Z

Following this issue I propose to create a tool called kci-dev
$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes
This file will also accept a .kci-dev.yaml configuration file for define password and api keys The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.

I believe it should be mandatory to the repository enabled in KernelCI already. I think that is safer.

I also wonder why we would not publish the results up to KCIDB. If you are sending to KernelCI, it is public code.

aliceinwire · 2024-07-25T12:37:11Z

Following this issue I propose to create a tool called kci-dev
$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes
This file will also accept a .kci-dev.yaml configuration file for define password and api keys The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.
I believe it should be mandatory to the repository enabled in KernelCI already. I think that is safer.

currently yes also because we are sending only a diff, not all the code and we need to match the diff with a full repository tree
the --repository option is for know to which repository tree we are matching against the diff patches

I also wonder why we would not publish the results up to KCIDB. If you are sending to KernelCI, it is public code.

As we discussed on the weekly meeting we thought of some possibility:

In the case of embargoed security related code (but after talking with Greg K-H looks like there is no need for this case)
for development testing
for preventing to show fake failure on stable tree

aliceinwire · 2024-08-01T02:21:57Z

P4) Testing new patches

POST /api/patch
{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}
Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.

It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.

Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.

aliceinwire · 2024-08-01T02:25:33Z

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way. All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)

So request can be as following: POST /api/checkout
{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}
Will return custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can answer to bisect good/bad.

P2)Custom test run retry

POST /api/retry
{
    "node": "<failing_test_node_id>"
}
Not sure yet, we cannot return node id in this case, but i can return kbuild node id that will have new "child" node with test run results.

Purpose: Sometimes hardware or tests are "flakey" (unreliable), so it is useful to retry test several times.

P3) Test fix(patch)

POST /api/testfix
{
    "node": "<failing_test_node_id>",
    "patch": "<patch_content_base64?>"
}
It will fetch same sources as test failed, but apply patch before build. Then after building patched kernel it will run test. It will return as in p1 custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can test patches on kernelci infrastructure, if they fix failing test/regression.

where do you get the failing test node id? is this some public information and easy to retrieve?
also retry the same test I feel like is more something for KCI developers than kernel developers. retry the same tests on the same kernel build with same code is supposed to give same results.

aliceinwire · 2024-08-01T02:30:16Z

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way. All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)
So request can be as following: POST /api/checkout
{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}
Will return custom checkout node id that can be followed by traditional API calls or on dashboard.
Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can answer to bisect good/bad.

Kernel developers are not suppose to ssh into KernelCI for doing bisection on KernelCI infrastructure

nuclearcat · 2024-08-01T06:47:06Z

where do you get the failing test node id? is this some public information and easy to retrieve? also retry the same test I feel like is more something for KCI developers than kernel developers. retry the same tests on the same kernel build with same code is supposed to give same results.

Yes, it is available in Grafana (which is available now) and in public dashboard. It is something like failed test ID.
Additionally, i just added recently function to specify parameters manually (or retrieve them semi-automatically if you point to git repo) kernelci/kernelci-pipeline#739, such as github repo URL, branch, commit id, but then developer will have to specify in cli options kernel build and test he wants to run.
Today or tomorrow i will add to PoC tool support of that too.

nuclearcat · 2024-08-01T06:48:10Z

P4) Testing new patches

POST /api/patch
{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}
Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.
It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.
Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.
would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.

Yes, thats planned. Right now i have filters for job (to limit test to specific build and test name), but need to add also lab AND also possible filter to limit test to specific device type as well.

aliceinwire · 2024-08-05T07:15:09Z

P4) Testing new patches

POST /api/patch
{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}
Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.
It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.
Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.
would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.
Yes, thats planned. Right now i have filters for job (to limit test to specific build and test name), but need to add also lab AND also possible filter to limit test to specific device type as well.

We don't need the commit id when the change is only local,
commit_id should be optional too

nuclearcat · 2024-08-05T07:23:41Z

I can make it optional, but that means it will just fetch current "tip of tree" commit id and fill it "automagically" (and create some uncertainity).
If we specify tree, it is not necessary that your local repository is updated and have same tip of tree, this is why i need commit id.
For example developer started to develop on Aug 1, and tip of tree at that moment was commit XXXX, but when he finished patch and he want to test it, tip of tree moved forward and it is commit YYYY and patch might not apply. So there is always required commit id to know at which place of tree we apply patch.
Also if we want to review results later, to reproduce build/tests we will need to know commit id, as after few days again tip of tree will change.

aliceinwire · 2024-08-06T05:17:10Z

I can make it optional, but that means it will just fetch current "tip of tree" commit id and fill it "automagically" (and create some uncertainity). If we specify tree, it is not necessary that your local repository is updated and have same tip of tree, this is why i need commit id. For example developer started to develop on Aug 1, and tip of tree at that moment was commit XXXX, but when he finished patch and he want to test it, tip of tree moved forward and it is commit YYYY and patch might not apply. So there is always required commit id to know at which place of tree we apply patch. Also if we want to review results later, to reproduce build/tests we will need to know commit id, as after few days again tip of tree will change.

I see, that works for me.
I will try to add the commit id from the commit, if there is one, in case there isn't a commit id drop to add it automatically on the server side works for me. so please leave it as optional.

aliceinwire · 2024-08-06T05:20:53Z

two other parameters that I would like to see are:

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
   "laboratory": <laboratory names>
    "tests": <tests name>
}

laboratories is for decide in which laboratory or laboratories the tests will be executed, default to all laboraries
tests is for decide which tests will be executed, default to all tests available for such build

nuclearcat · 2024-08-06T07:19:12Z

kbuild and testname is already in jobfilter (i will need to explain how they works)
i will add also labfilter and dutfilter (filter by lab and device type)
this is also within plans

aliceinwire · 2024-08-06T14:10:31Z

the current PoC of the kci-dev tool
https://github.com/aliceinwire/kci-dev

aliceinwire · 2024-08-07T07:09:42Z

poetry run kci-dev --help
Usage: kci-dev [OPTIONS] COMMAND [ARGS]...

  Stand alone tool for Linux Kernel developers and maintainers that can test
  local Linux Kernel changes on a enabled KernelCI server

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  commit  Test commits from a local Kernel repository
  patch   Test a patch or a mbox file

aliceinwire · 2024-08-07T07:10:22Z

poetry run kci-dev commit --help
Usage: kci-dev commit [OPTIONS]

  Test commits from a local Kernel repository

Options:
  --repository TEXT  define the kernel upstream repository where to test local
                     changes
  --branch TEXT      define the repository branch
  --private          define if the test results will be published
  --path TEXT        define the directory of the local tree with local changes
  --settings TEXT    path of toml setting file
  --help             Show this message and exit.

aliceinwire · 2024-08-07T07:10:53Z

poetry run kci-dev patch --help
Usage: kci-dev patch [OPTIONS]

  Test a patch or a mbox file

Options:
  --repository TEXT  define the kernel upstream repository where to test local
                     changes
  --branch TEXT      define the repository branch
  --private          define if the test results will be published
  --patch TEXT       mbox or patch file path  [required]
  --settings TEXT    path of toml setting file
  --help             Show this message and exit.

aliceinwire · 2024-08-07T07:12:21Z

the JWT token and api host address are managed by the .kci-dev.toml.example setting file

nuclearcat · 2024-08-07T07:21:17Z

I will finish implementing today initial patchset features (after discovering how this subsystem works, and will publish updates), and will propose API for it.
Then i will take a look at tool. I guess we might need some repository where can publish such tools, or even individual repository for kci-dev. Probably this is good topic to discuss during upcoming weekly.

aliceinwire · 2024-08-07T07:34:02Z

My intention is to move the tool to kernelci organization, can you give me the admin on the repo after I moved it?
Also the tool can be published to the pip repository so that it can be used also by downloading it from pip

nuclearcat · 2024-08-07T07:36:55Z

Definitely, great idea, personally i am all for it and it should be possible.
I think to create repo and give admin privileges we need to vote at TSC maillist (we can do it before meeting).

aliceinwire · 2024-08-07T07:38:18Z

Is not creating the repo is a transfer
I already checked that I can transfer the repo even now, I just want to be sure to don't loose my admin rights after the transfer

nuclearcat · 2024-08-07T08:27:57Z

Yes, but i think according the rules we need to discuss with TSC adding or transferring repo to kernelci organization, but i am not sure about that.
We voted also for creating new repo, but TSC rules doesn't require that.
Also we need to update https://docs.kernelci.org/org/maintainers/#software-maintainers if we move repo and we need to set you as maintainer, and describe project.

aliceinwire · 2024-08-07T08:43:29Z

sure I was just specifying that is not a creation but a transfer

nuclearcat · 2024-09-03T08:16:07Z

Adding as example usage of semi-reference tool i am using in development, for future documentation:

./kci-maintainer --checkout -u https://github.com/kernelci/linux.git -b staging-stable -f kbuild-gcc-12-x86 -f watchdog-reset-x86-intel -c 9c23c1e2e8b9e3eee88a6f3928c085d501137933

This will run custom commit on particular tree for specific build and test.

/kci-maintainer --checkout -n 66d67809bc920e0e668f9110 -c 9c23c1e2e8b9e3eee88a6f3928c085d501137933

If you are not happy about some existing test result, you can try it on another commit in same tree (-c commit, -n test node id)

nuclearcat · 2024-09-05T07:41:24Z

I added also to each tree (custom checkout tree, or normal automated tip-of-tree) unique treeid field, which will make easier identification and tracking of custom submissions (custom checkout, checkout with patch/patchset).
Relevant PR/issues:
#2655
kernelci/kernelci-project#443
kernelci/kernelci-pipeline#783

aliceinwire assigned nuclearcat Jul 3, 2024

nuclearcat assigned pawiecz Jul 3, 2024

aliceinwire mentioned this issue Jul 5, 2024

Private builds and tests #2605

Open

pawiecz mentioned this issue Jul 22, 2024

Add support for running bisection jobs in the pipeline kernelci/kernelci-pipeline#711

Open

aliceinwire mentioned this issue Sep 6, 2024

API : Add feature needed for kci-dev for sending request with the KernelCI API/pipeline kernelci/kci-dev#2

Closed

Add feature kernelci-try #2594

Add feature kernelci-try #2594

Comments

aliceinwire commented Jul 1, 2024

aliceinwire commented Jul 3, 2024

nuclearcat commented Jul 3, 2024 • edited Loading

aliceinwire commented Jul 3, 2024

pawiecz commented Jul 3, 2024

aliceinwire commented Jul 3, 2024

aliceinwire commented Jul 3, 2024 • edited Loading

nuclearcat commented Jul 3, 2024

pawiecz commented Jul 3, 2024

nuclearcat commented Jul 11, 2024 • edited Loading

Design of tree maintainers API (KISS principle):

P1)Custom checkout for bisection

P2)Custom test run retry

P3) Test fix(patch)

P4) Testing new patches

nuclearcat commented Jul 11, 2024

aliceinwire commented Jul 12, 2024

padovan commented Jul 25, 2024

aliceinwire commented Jul 25, 2024 • edited Loading

aliceinwire commented Aug 1, 2024

P4) Testing new patches

aliceinwire commented Aug 1, 2024

Design of tree maintainers API (KISS principle):

P1)Custom checkout for bisection

P2)Custom test run retry

P3) Test fix(patch)

aliceinwire commented Aug 1, 2024

Design of tree maintainers API (KISS principle):

P1)Custom checkout for bisection

nuclearcat commented Aug 1, 2024

nuclearcat commented Aug 1, 2024

P4) Testing new patches

aliceinwire commented Aug 5, 2024

P4) Testing new patches

nuclearcat commented Aug 5, 2024 • edited Loading

aliceinwire commented Aug 6, 2024

aliceinwire commented Aug 6, 2024

nuclearcat commented Aug 6, 2024

aliceinwire commented Aug 6, 2024

aliceinwire commented Aug 7, 2024

aliceinwire commented Aug 7, 2024

aliceinwire commented Aug 7, 2024

aliceinwire commented Aug 7, 2024

nuclearcat commented Aug 7, 2024

aliceinwire commented Aug 7, 2024

nuclearcat commented Aug 7, 2024

aliceinwire commented Aug 7, 2024 • edited Loading

nuclearcat commented Aug 7, 2024

aliceinwire commented Aug 7, 2024

nuclearcat commented Sep 3, 2024

nuclearcat commented Sep 5, 2024

nuclearcat commented Jul 3, 2024 •

edited

Loading

aliceinwire commented Jul 3, 2024 •

edited

Loading

nuclearcat commented Jul 11, 2024 •

edited

Loading

aliceinwire commented Jul 25, 2024 •

edited

Loading

nuclearcat commented Aug 5, 2024 •

edited

Loading

aliceinwire commented Aug 7, 2024 •

edited

Loading