Pants: add `python_distribution` BUILD metadata for st2* components #5909

cognifloyd · 2023-02-20T20:07:24Z

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This PR adds the st2_component_python_distribution() macro to the BUILD metadata for all of our components (st2common, st2client, st2auth, etc).

Thanks to the macro, we only have to define 3 (or 4) parameters for each runner: component_name, scripts, and entry_points. I copied the scripts and entry_points from the setup.py file, which will become obsolete once pants can generate the setup.py file for us.

Under entry_points, I have "tagged" the st2auth.sso.backends, the st2common.metrics.driver and the st2common.rbac.backend categories as stevedore_namespaces. Once we switch to 2.16 and enable the pants.backend.experimental.python.framework.stevedore plugin, the stevedore plugin will, for tests, use this "tag" to look up which wheels implement a plugin in that stevedore namespace.

To facilitate comparison, I will add a comment to each of the files that shows the current setup.py entry_points and scripts.

Next steps

Follow-up PRs will adjust the dependencies on the python_distributions to ensure all files get included.

cognifloyd · 2023-02-20T20:09:20Z

st2actions/BUILD

+    scripts=[
+        "bin/st2actionrunner",
+        "bin/st2notifier",
+        "bin/st2workflowengine",
+        "bin/st2scheduler",
+        "bin/runners.sh:shell",  # used by service files installed by st2-packaging
+    ],


st2/st2actions/setup.py

Lines 49 to 54 in f115934

scripts=[

"bin/st2actionrunner",

"bin/st2notifier",

"bin/st2workflowengine",

"bin/st2scheduler",

],

Plus I noticed that st2-packaging needs runners.sh, so I included that as well.

cognifloyd · 2023-02-20T20:09:41Z

st2api/BUILD

@@ -0,0 +1,4 @@
+st2_component_python_distribution(
+    component_name="st2api",
+    scripts=["bin/st2api"],


st2/st2api/setup.py

Line 48 in f115934

scripts=["bin/st2api"],

cognifloyd · 2023-02-20T20:10:10Z

st2auth/BUILD

+    scripts=["bin/st2auth"],
+    entry_points={
+        stevedore_namespace("st2auth.sso.backends"): {
+            "noop": "st2auth.sso.noop:NoOpSingleSignOnBackend",
+        },
+    },


st2/st2auth/setup.py

Lines 49 to 52 in f115934

scripts=["bin/st2auth"],

entry_points={

"st2auth.sso.backends": ["noop = st2auth.sso.noop:NoOpSingleSignOnBackend"]

},

cognifloyd · 2023-02-20T20:10:59Z

st2client/BUILD

+    entry_points={
+        "console_scripts": {
+            "st2": "st2client.shell:main",
+        },
+    },


st2/st2client/setup.py

Line 69 in f115934

entry_points={"console_scripts": ["st2 = st2client.shell:main"]},

cognifloyd · 2023-02-20T20:12:05Z

st2common/BUILD

+    scripts=[
+        # some scripts in bin are only for development and should not be included.
+        "bin/st2-bootstrap-rmq",
+        "bin/st2-cleanup-db",
+        "bin/st2-register-content",
+        "bin/st2-purge-executions",
+        "bin/st2-purge-workflows",
+        "bin/st2-purge-task-executions",
+        "bin/st2-purge-tokens",
+        "bin/st2-purge-trigger-instances",
+        "bin/st2-purge-traces",
+        "bin/st2-purge-rule-enforcements",
+        "bin/st2-generate-symmetric-crypto-key",
+        "bin/st2-track-result",
+        "bin/st2-validate-pack",
+        "bin/st2-validate-pack-config",
+        "bin/st2-pack-install",
+        "bin/st2-pack-download",
+        "bin/st2-pack-setup-virtualenv",
+        "bin/migrations/v3.5/st2-migrate-db-dict-field-values",
+        "bin/st2-run-pack-tests:shell",
+        "bin/st2ctl:shell",
+        "bin/st2-self-check:shell",
+        # dev scripts we might want to include
+        # "bin/st2-generate-schemas",
+    ],
+    entry_points={
+        stevedore_namespace("st2common.metrics.driver"): {
+            "statsd": "st2common.metrics.drivers.statsd_driver:StatsdDriver",
+            "noop": "st2common.metrics.drivers.noop_driver:NoopDriver",
+            "echo": "st2common.metrics.drivers.echo_driver:EchoDriver",
+        },
+        stevedore_namespace("st2common.rbac.backend"): {
+            "noop": "st2common.rbac.backends.noop:NoOpRBACBackend",
+        },
+    },


st2/st2common/setup.py

Lines 50 to 82 in f115934

scripts=[

"bin/st2-bootstrap-rmq",

"bin/st2-cleanup-db",

"bin/st2-register-content",

"bin/st2-purge-executions",

"bin/st2-purge-workflows",

"bin/st2-purge-task-executions",

"bin/st2-purge-trigger-instances",

"bin/st2-purge-traces",

"bin/st2-purge-rule-enforcements",

"bin/st2-purge-tokens",

"bin/st2-run-pack-tests",

"bin/st2ctl",

"bin/st2-generate-symmetric-crypto-key",

"bin/st2-self-check",

"bin/st2-track-result",

"bin/st2-validate-pack",

"bin/st2-validate-pack-config",

"bin/st2-pack-install",

"bin/st2-pack-download",

"bin/st2-pack-setup-virtualenv",

"bin/migrations/v3.5/st2-migrate-db-dict-field-values",

],

entry_points={

"st2common.metrics.driver": [

"statsd = st2common.metrics.drivers.statsd_driver:StatsdDriver",

"noop = st2common.metrics.drivers.noop_driver:NoopDriver",

"echo = st2common.metrics.drivers.echo_driver:EchoDriver",

],

"st2common.rbac.backend": [

"noop = st2common.rbac.backends.noop:NoOpRBACBackend"

],

},

A few of the scripts were missing in setup.py, so I included them here. Also, a couple of them were missing the execute bit, so I added that as well.,

cognifloyd · 2023-02-20T20:13:53Z

st2reactor/BUILD

+    scripts=[
+        "bin/st2-rule-tester",
+        "bin/st2-trigger-refire",
+        "bin/st2rulesengine",
+        "bin/st2sensorcontainer",
+        "bin/st2garbagecollector",
+        "bin/st2timersengine",
+    ],


st2/st2reactor/setup.py

Lines 49 to 56 in f115934

scripts=[

"bin/st2-rule-tester",

"bin/st2-trigger-refire",

"bin/st2rulesengine",

"bin/st2sensorcontainer",

"bin/st2garbagecollector",

"bin/st2timersengine",

],

cognifloyd · 2023-02-20T20:14:15Z

st2stream/BUILD

@@ -0,0 +1,4 @@
+st2_component_python_distribution(
+    component_name="st2stream",
+    scripts=["bin/st2stream"],


st2/st2stream/setup.py

Line 48 in f115934

scripts=["bin/st2stream"],

cognifloyd · 2023-02-20T20:17:52Z

#5906 should be merged first

amanda11

Why are st2common/bin/st2-purge-task-executions and st2common/bin/st2-purge-workflows now empty files?
Is that github just showing it odd in PR, and all you've done is change the permissions on the file?

cognifloyd · 2023-02-27T16:03:37Z

Yeah. That's just GitHub being weird as I only changed the execute bit, not the file contents. If you do a "view file" on those, you'll see the file contents.

…ants

tag with stevedore_namespace so that once we enable the pants-plugin for stevedore, pants can install things appropriately for tests to access the setuptools metadata.

cognifloyd added the pantsbuild label Feb 20, 2023

cognifloyd added this to the pants milestone Feb 20, 2023

cognifloyd self-assigned this Feb 20, 2023

pull-request-size bot added the size/M PR that changes 30-99 lines. Good size to review. label Feb 20, 2023

cognifloyd commented Feb 20, 2023

View reviewed changes

cognifloyd marked this pull request as ready for review February 20, 2023 20:20

cognifloyd requested review from rush-skills, arm4b, amanda11, nzlosh, winem and a team February 20, 2023 20:20

cognifloyd force-pushed the pants-python_distributions-components branch from 7016ce7 to 4dd0184 Compare February 24, 2023 17:44

amanda11 reviewed Feb 27, 2023

View reviewed changes

cognifloyd force-pushed the pants-python_distributions-components branch from 4dd0184 to c3615b5 Compare February 27, 2023 17:51

cognifloyd enabled auto-merge February 27, 2023 17:52

cognifloyd requested review from amanda11 and a team February 27, 2023 17:53

amanda11 approved these changes Feb 27, 2023

View reviewed changes

cognifloyd requested a review from a team February 27, 2023 23:44

arm4b approved these changes Mar 2, 2023

View reviewed changes

cognifloyd added 5 commits March 2, 2023 14:52

add python_distribution for building st2 component wheel/sdist with p…

e564418

…ants

Add scripts to BUILD python_distribution for st2 components

796a08c

fix execute bit on some scripts

d72061f

record entry_points for wheels

575bd57

tag with stevedore_namespace so that once we enable the pants-plugin for stevedore, pants can install things appropriately for tests to access the setuptools metadata.

update changelog entry

083fd00

cognifloyd force-pushed the pants-python_distributions-components branch from c3615b5 to 083fd00 Compare March 2, 2023 20:52

cognifloyd merged commit 7fe538a into master Mar 2, 2023

cognifloyd deleted the pants-python_distributions-components branch March 2, 2023 21:22

cognifloyd mentioned this pull request Mar 24, 2023

Testing Wheels generated by Pantsbuild StackStorm/community#119

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pants: add `python_distribution` BUILD metadata for st2* components #5909

Pants: add `python_distribution` BUILD metadata for st2* components #5909

cognifloyd commented Feb 20, 2023 •

edited

Loading

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd Feb 20, 2023

cognifloyd commented Feb 20, 2023

amanda11 left a comment

cognifloyd commented Feb 27, 2023

	scripts=[
	"bin/st2actionrunner",
	"bin/st2notifier",
	"bin/st2workflowengine",
	"bin/st2scheduler",
	],

	scripts=["bin/st2auth"],
	entry_points={
	"st2auth.sso.backends": ["noop = st2auth.sso.noop:NoOpSingleSignOnBackend"]
	},

	scripts=[
	"bin/st2-bootstrap-rmq",
	"bin/st2-cleanup-db",
	"bin/st2-register-content",
	"bin/st2-purge-executions",
	"bin/st2-purge-workflows",
	"bin/st2-purge-task-executions",
	"bin/st2-purge-trigger-instances",
	"bin/st2-purge-traces",
	"bin/st2-purge-rule-enforcements",
	"bin/st2-purge-tokens",
	"bin/st2-run-pack-tests",
	"bin/st2ctl",
	"bin/st2-generate-symmetric-crypto-key",
	"bin/st2-self-check",
	"bin/st2-track-result",
	"bin/st2-validate-pack",
	"bin/st2-validate-pack-config",
	"bin/st2-pack-install",
	"bin/st2-pack-download",
	"bin/st2-pack-setup-virtualenv",
	"bin/migrations/v3.5/st2-migrate-db-dict-field-values",
	],
	entry_points={
	"st2common.metrics.driver": [
	"statsd = st2common.metrics.drivers.statsd_driver:StatsdDriver",
	"noop = st2common.metrics.drivers.noop_driver:NoopDriver",
	"echo = st2common.metrics.drivers.echo_driver:EchoDriver",
	],
	"st2common.rbac.backend": [
	"noop = st2common.rbac.backends.noop:NoOpRBACBackend"
	],
	},

	scripts=[
	"bin/st2-rule-tester",
	"bin/st2-trigger-refire",
	"bin/st2rulesengine",
	"bin/st2sensorcontainer",
	"bin/st2garbagecollector",
	"bin/st2timersengine",
	],

Pants: add python_distribution BUILD metadata for st2* components #5909

Pants: add python_distribution BUILD metadata for st2* components #5909

Conversation

cognifloyd commented Feb 20, 2023 • edited Loading

Background

Overview of this PR

Next steps

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd Feb 20, 2023

Choose a reason for hiding this comment

cognifloyd commented Feb 20, 2023

amanda11 left a comment

Choose a reason for hiding this comment

cognifloyd commented Feb 27, 2023

Pants: add `python_distribution` BUILD metadata for st2* components #5909

Pants: add `python_distribution` BUILD metadata for st2* components #5909

cognifloyd commented Feb 20, 2023 •

edited

Loading