TSC Meeting (12 Jul 2022): pants, Puppet-ST2 metrics, CI/CD and AWS discussions, v3.8 release planning

[rush-skills]

July 2022 @StackStorm/tsc 1 hour planned meeting: Tuesday, 12 July 2022, 09:30 AM US Pacific.

See #33 for more info on how to join.
(TLDR; Zoom link: https://us02web.zoom.us/j/81082101702?pwd=N1V4TWdYRVQ4SXBsaFh1TFYvVDA0UT09)

Meeting host: @rush-skills

Meeting Agenda

Demo: pants, a dependency management, test/lint and packaging framework (@cognifloyd)

Demo: Puppet-st2 based StackStorm metrics (@bishopbm1)

• Based on the added metrics in the PR: RabbitMQ and Metrics puppet-st2#358

Discussion: De-rooting the applications and containers (@rk4n3)

• Builds up on issue: De root the applications and containers st2-docker#187

CI/CD Cost Reduction discussions

Follow-up on AWS migration

Follow-up on Linux and Python support plan

v3.8 release discussions

• Pending Tasks? Target Date?
• Release Manager: @nzlosh
• Release Assistant: @rush-skills

[rush-skills]

Let me know if anything else needs to be added

[rush-skills]

Attendees

@amanda11 @armab @arms11 @bishopbm @cognifloyd @rk4n3 @rush-skills Benjy@pantsbuild

Meeting Minutes

Demo: pants, a dependency management, test/lint and packaging framework (@cognifloyd)

• @cognifloyd demo'ed pants as a replacement for Makefile based build system, as well for running testing and linting, and then also for building python wheels for packaging against specified python versions
• Opinion was asked about whether it seemed like a feasible replacement for st2 build systems, positive feedback was gathered.
• The python-version targeting was also taken as a new approach to move from supporting OS+Python pairs to supporting specified Python versions packaged as wheels (which pants enables)
• When questioned about the dev effort required to create a PoC PR, @cognifloyd indicated that most effort has already been carried out by him, and thus should not be a big hassle for community if we choose to adopt it
• Benjy from the pantsbuild community joined for the demo and cleared extra questions regarding pants
• Follow-Up:
  • RFC for changing StackStorm build system to pants + Dropping (OS+Python version) support in favor of only Python version.
  • PR running pants against st2 for linting, testing and packaging

Demo: Puppet-st2 based StackStorm metrics (@bishopbm1)

• Based on the added metrics in the PR: RabbitMQ and Metrics puppet-st2#358
• Using Telegraf to ingest the data into Prometheus and plotting the data on Grafana, @bishopbm1 was able to demo the attendees dashboards for various StackStorm components and dependencies (Mongo, Rabbit)
• Bradley asked if such dashboards and configuration for metrics is something that we want to include in the StackStorm repo, for which
• The feedback gathered was mostly positive, with people interested in the dashboards and the metric aggregation scripts
• The proper place/format to distribute the dashboard and configuration of various elements was discussed, and eventually was agreed that
  • Sending metric to Prometheus with statsd/Telegraf can be integrated to puppet-st2 module as it is already implemented in puppet by the authors
  • Dashboards can be put in the repo and documented with the main project, maybe with Metrics and Instrumentation, which can then be used later by the community in different setups as desired
  • Setup for Prometheus, Grafana etc. for the dashboards was deemed something that is very org-specific, and thus might not belong to any st2 code-base in general
    • However, adding basic setup code to puppet or k8s repo as a basic setup to get people up-to speed is also acceptable
  • Updates to Puppet, Logging and Metrics documentation will be needed to keep things up-to date

Discussion: De-rooting the applications and containers (@rk4n3)

• Builds up on issue: De root the applications and containers st2-docker#187
• @rk4b3 started a discussion about de-rooting the application and containers such that the loopholes in permission and security models of StackStorm can be leveled.
• He then further went and proposed integration of a new type of runner, running another layer of (dockerized) actionrunner instances, in parallel to the current approach, which if integrated with current StackStorm, allows actions to run without root.
• He also proposed some changes to st2 core for HA, in order to make StackStorm more scalable and reliable
• Community was asked if such changes would be welcome from their org as contributions
  • The idea was accepted and generally welcomed
  • Both security and HA are a current pain point to the community, so any effort in that direction is welcomed
• Follow-up:
  • @rk4n3 to propose a RFC for the changes

CI/CD Cost Reduction discussions

• @amanda confirmed the migration of build machine from always-running VM to short-running machines and was given a green light.

Follow-up on AWS migration

• @bishopbm1 confirmed that AWS account has been migrated to Encore's (parent) account and billing credentials have been changed.

Follow-up on Linux and Python support plan

• As demonstrated with pants demo, it will be possible to build individual wheels for each StackStorm component and target specific Python versions without being concerned about the underlying OS, it was discussed that Linux support will now be decoupled from Python version support in future releases, subject to RFC approval by TSC.

v3.8 release discussions

• Planned for end of Aug/Sept'22
• Release Manager: @nzlosh
• Release Assistant: @rush-skills

[rush-skills]

Future meetings: #107
#108

Closing the discussion for this TSC Meeting.