[EngAut] CommandBot: Create GHA version

[crystalin]

Background:

• current command bot is fully dependent on GitLab, because it was designed to manage tasks queue via GitLab CI pipelines.
• if any parachain team would want to re-use this bot for their benchmarks and other tasks - the GHA+self-hosted runner is much simpler to set up than deploying a runtime NodeJS app with DB + self-hosted runners

Stakeholders and Goals:

• [Parity Internal]: unblock gitlab deprecation, as current implementation is locked-in for GitLab
• [Ecosystem] setup and run benchmarks easier

Original request:
We've been using the benchbot and are thinking to move to the command-bot, however we don't rely on gitlab for our CI.
Our current setup is using Github Action but with self-hosted instances (dedicated to building/testing).

I like the idea of the command-bot allowing commands from different sources (matrix/github/...) but it seems to be quite customized to how parity work (using gitlab, helm,...).

Do you have any plan on making it more generic to allow non-gitlab service ?

---

GitHub action milestone

Tasks

Preview Give feedback

1. Remove GitLab code cmd-action#4
   action +1
   
2. Command schema cmd-action#7
   action +1
   
3. Status check - schema cmd-action#10
   action +1
   
4. Add json schema for yaml files cmd-action#14
   action +1
   
5. Generate commands documentation cmd-action#12
   action +1
   
6. Make app get information from a comment cmd-action#2
   action +1
   
7. Report error on failure cmd-action#20
   action +1
   
8. Make app work as a one time executable cmd-action#1
   action +1
   
9. Rewrite bench to be generic for any kind of substrate-based projects #274
   +0
10. Use file name instead of name field cmd-action#22
    action +1
    
11. Add a small action that reacts to bot xyz commands and sends a link to the forum post

[mutantcornholio]

Could you elaborate on what currently doesn't work, or how command-bot's way of spawning jobs contradicts your CI configuration?

[crystalin]

I didn't go as far a trying to setup everything yet. The documentation contains many element but it is a bit hard to know how all of the elements are orchestrated overall.
But in the setup part it mentions:

Create a repository in GitLab and set up GITLAB_PUSH_NAMESPACE along with GITLAB_ACCESS_TOKEN to run the commands for the GitHub repository (Step 3).

That is why I supposed it required to do so, I'll try to set the bot up and see where I get stuck

[crystalin]

Also, the first line of readme also mention:

command-bot executes arbitrary commands on GitLab CI

[mutantcornholio]

You're going through deployment/development setup. Are you trying to host your own instance?

[crystalin]

Ideally yes, but if you offer a service where we can plug our repo and our CI servers, I'm happy to try it (I don't think that would work in term of security however so we would end up hosting it)

[mutantcornholio]

Okay, I think I've misread the situation. Let us think on it.
Currently, it'll basically mean replacing at least half of the code, and that raises a big question on the vision of the project.

[mordamax]

@crystalin Currently, we defer the tasks to run and queue for the CI, in our case Gitlab. We are going to make it less opinionated and instead of creating a Gitlab CI job - create Kubernetes jobs.

What would be a desired infrastructure set up so you could deploy and use it on your side?

[crystalin]

Thank you, I think plugin it with github and using their GHA (self-hosted runner) is preferred but we could probably combine it with our kubernetes cluster too

[mordamax]

Current plan is to move away from gitlab completely, so will be porting the exiting command bot to separate repo, the idea is to set this up as Github action, with decoupled repo for command scripts and interface.
https://github.com/paritytech/command-bot-scripts/tree/main/commands
This allows to avoid re-deploying/releasing the bot business logic from the commands implementation.

[Bullrich]

I will be taking up this task.

I have created a milestone to track this:

GitHub action milestone

[Bullrich]

The repository cmd-action has been created. The action will be developed there.

I ran two experiments to see if this is feasible, both of them were succesfull:

• added prototype ci script cmd-action#6
• added cmd-action prototype paritytech-stg/polkadot-sdk#26

[Bullrich]

Currently working on paritytech/cmd-action#1.

I'm making the system parse content from the comments and validating it with the commands.

[Bullrich]

Preparing schema handling.

I've been in contact with Oliver, our main stakeholder.

• We have decided that the repository will hold the required scripts (unless that a project evolves too much)
• Also, a similar action was commented by Basti: Exclude pallet_example_* from bench-all-substrate command-bot-scripts#69 (comment)

The logic of configuration handling will be similar to review-bot's: Easy to read and hosted in the target repository.

cmd-action will be mainly a tool to "parse" commands and pass it to a github action with a repo script.

[Bullrich]

Currently working on paritytech/cmd-action#7

Configuration files will be moved to be in yaml (can also be toml) so it's easier to read.

I'm preparing this ticket in combination with paritytech/cmd-action#10 so we have a schema validator out of the box

[Bullrich]

Completed the following tasks

• Command schema cmd-action#7
• Status check - schema cmd-action#10
  • Added schema validation cmd-action#11
• Generate commands documentation cmd-action#12
  • added generation of command documentation cmd-action#15
• Add json schema for yaml files cmd-action#14
  • Added json schema validation cmd-action#17

[Bullrich]

22-26 of April

• Generate commands documentation cmd-action#12
  • added generation of command documentation cmd-action#15
• Currently working on Make app get information from a comment cmd-action#2
  • Extract parameters from a command

[Bullrich]

Currently working on the finishing touches for paritytech/cmd-action#18

This will close:

• Make app work as a one time executable cmd-action#1
• Make app get information from a comment cmd-action#2

[Bullrich]

Closed:

• Report error on failure cmd-action#20
  • Report run and failure with comment cmd-action#21

[Bullrich]

Completed last week:

• Use file name instead of name field cmd-action#22
  • Use filename instead of name cmd-action#24
• Report error on failure cmd-action#20
  • Report run and failure with comment cmd-action#21
• Added ability to make push trigger actions cmd-action#25

[Bullrich]

WIP: #8

• ✍️ Added ability to parse arguments cmd-action#26

Most of the task is done. Only some things remaining:

• Update the JSON schema
• Add more test with failing cases
• Update the generated documentation

[Bullrich]

Work in progress to migrate the command action to work as a github workflow:

• Migrated commands to github actions polkadot-sdk#4701

Missing commands

• bench-all
• bench-overhead
• bench
• fmt
• sample
• sync
• update-ui

[mordamax]

The fellows/runtimes integration is on pause due to couple of reasons

• upgrade to polkadot-sdk to v1.13 which supports omni-bencher is not stable yet and blocking
• switched for now to parity.io rework

[Bullrich]

Finished preparing paritytech/polkadot-sdk#4701

[Bullrich]

Completed migration of most commands:

• Migrated commands to github actions polkadot-sdk#4701

Currently working on adding the last command (sync): paritytech/polkadot-sdk#4818
Blocked by https://github.com/paritytech/ci_cd/issues/995

[rzadp]

Note: I added a task to the list to:

• Add a small action that reacts to bot xyz commands and sends a link to the forum post.

[Bullrich]

Note: I added a task to the list to:

• Add a small action that reacts to bot xyz commands and sends a link to the forum post.

paritytech/polkadot-sdk#5000 should provide the fix for this.

[mordamax]

Closing this as done in V1 and V2 versions ->

• polkadot-sdk https://github.com/paritytech/product-engineering/issues/93
• runtimes cmd actions bench & fmt polkadot-fellows/runtimes#406

@crystalin we've gone with simple python script (could be short bash too, like Astar does)
we didn't go with too complex generic (or runtime) solution as it's simple and could be adjusted per project needs