Declarative dependencies #85

noise64 · 2024-09-24T11:14:39Z

Implemented:

commands:
- wasm-rpc declarative pre-build [-c xxx.yaml -c yyy.yaml]: executes "stub build" for are dependency and "add stub deps" for all components
- wasm-rpc declarative post-build [-c xxx.yaml -c yyy.yaml]: executes "compose" for all components (uses copy if no deps were )
manifest resolution:
- if they are passed directly to the command, then that explicit set of manifests are used
- if no manifests are specified as command line arg, then we search for golem.yaml in the current and parent folders, and use the one closest to the root, then we "pre-parse" this golem.yaml, which can define the include property (see below in examples), which will be used as a glob expression for finding more manifests
manifest model:
- a generic open application model
- a specific one extracted from the above with out custom component properties and traits:
  - component types
    - wasm: we can extended this in golem-cli with e.g. durable and other deployment relevant properties
    - wasm-build: for customizing generic build properties, currently only contains the build dir and the include glob
    - wasm-rpc-stub-build: for customizing the stub generation and build, can be specific for a stub by component name, when no component name is specified, then it used as a common preset
- both models should be reusable and extendable in golem-cli
- path properties are resolved relative to the yaml source which contains them
validations:
- the manifest are validated in a mostly non-short-circuiting manner, and the validation result can contain warns and errors, also the warns and errors contain context (e.g. source manifest file and component name)
- some examples for validations:
  - warns for extra and unknown properties in our custom properties and traits
  - uniqueness checks, e.g.: errors for duplicated component names
  - cross checks, e.g.: errors for unknown referenced component names
  - warns for duplicated dependencies
added some coloring to build messages (similar in colors to cargo, but no padding on actions)
changed files copies to always change modtime
component build commands with optional inputs and outputs
up-to-date checks (WIP)
replaced wasm-compose with wac-graph plugs, and now non-used stubs are skipped

Example specification:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: App
spec:
  # All of these can be in different yaml sources
  components:
    - name: common-wasm-build
      type: wasm-build
      properties:
        buildDir: out
        include: src/components/*/golem.yaml

    - name: common-wasm-rpc-stub-build
      type: wasm-rpc-stub-build
      properties:
        alwaysInlineTypes: true

    - name: component-one-stub-build
      type: wasm-rpc-stub-build
      properties:
        componentName: component-one
        buildDir: custom-build-dir
        alwaysInlineTypes: false

    - name: component-one
      type: wasm
      properties:
        build:
          - command: npm run componentize
            inputs:
              - src
              - wit
            outputs:
              - out/comp_a.wasm
        inputWasm: out/build/component-one/component.wasm
        outputWasm: out/components/component-one.wasm
        wit: src/components/component-one/wit
      traits:
        - type: wasm-rpc
          properties:
            componentName: component-two
        - type: wasm-rpc
          properties:
            componentName: component-three
      
    - name: component-two
      type: wasm
      properties:
        inputWasm: out/build/component-two/component.wasm
        outputWasm: out/components/component-two.wasm
        wit: src/components/component-one/wit
      traits:
        - type: wasm-rpc
          properties:
            componentName: component-three

Some screenshots about coloring and validation:

TODO:

recheck "add stub dep" up-to-date-check

TODO In follow up PR:

more validation tests
build tests

jdegoes · 2024-09-24T14:16:37Z

wasm-rpc-stubgen/src/model/component.rs

+        wasm: out/component.wasm
+        composedWasm: out/component-composed.wasm
+      traits:
+        - type: worker-rpc


Suggested change

- type: worker-rpc

- type: wasm-rpc

jdegoes · 2024-09-24T14:16:49Z

wasm-rpc-stubgen/src/model/component.rs

+        - type: worker-rpc
+          properties:
+            componentName: component-two
+        - type: worker-rpc


Suggested change

- type: worker-rpc

- type: wasm-rpc

jdegoes · 2024-09-24T14:20:27Z

wasm-rpc-stubgen/src/model/component.rs

+spec:
+  components:
+    - name: component-one
+      type: component-durable


Suggested change

type: component-durable

type: durable

I'm assuming this would be durable or ephemeral (no need, I think, to replicate the component part since it's already in the components part of the spec.

since we won't really support anything else (at least for now) we can make them shorter, but if we imagine that we might have different kind of components (e.g. not workers), then we might want to think more about this. note that here the OAM component and our component naming kinda works out now, but the OAM component is something more general. but for now probably we should go with your recommendation.

now that i'm thinking about it, in our model durable and ephemeral are the property of a component version, and not a component, so maybe we use simply wasm?

or is it okay that in these files it always references the latest one?

renamed, and kept as a component type for now

jdegoes · 2024-09-24T14:42:22Z

wasm-rpc-stubgen/src/model/component.rs

+      properties:
+        wit: wit
+        wasm: out/component.wasm
+        composedWasm: out/component-composed.wasm


I think we should stay away from 'composed' terminology as much as possible. A user may think in terms of 'output' or 'target', but probably not 'composed'.

Suggested change

composedWasm: out/component-composed.wasm

output-wasm: out/component-final.wasm

renamed, but with using camelCase

jdegoes · 2024-09-24T14:43:12Z

wasm-rpc-stubgen/src/model/component.rs

+      type: component-durable
+      properties:
+        wit: wit
+        wasm: out/component.wasm


Could this legitimately be called input-wasm???

Suggested change

wasm: out/component.wasm

input-wasm: out/component.wasm

Other parts of the schema follow camelCase.

I do agree on wasm and composedWasm not being good names, i'm just afraid that using input and output can be also confusing, given the input is the output of the build step, and e.g. in golem-cli, if we reach the point of not having to compose on the client side, we would have an input wasm alone, or if we would have some other tooling integration, this would be confusing. Given currently this is the property of the component, and not the dependency trait itself - where the input would totally make sense - i'm a bit unsure on this naming. (And we probably do not want to move these to the deps trait, because then it would be redundant when having multiple dependencies)

but happy to change it for now, and let's how it feels

renamed, but with using camelCase

…al validation model

…hout overrides

vigoo · 2024-10-01T14:33:34Z

One question, based on the description only:
Why do we have a type: durable | ephemeral property? It does not affect the stub generator at all - can we only add this property in golem(-cli) where it has an actual meaning and use?

vigoo · 2024-10-01T14:34:35Z

wasm-rpc-stubgen/src/lib.rs

@@ -40,6 +41,12 @@ pub enum Command {
    /// Initializes a Golem-specific cargo-make configuration in a Cargo workspace for automatically
    /// generating stubs and composing results.
    InitializeWorkspace(InitializeWorkspaceArgs),
+    /// TODO
+    #[cfg(feature = "unstable-dec-dep")]
+    Declarative {


I hope this won't be the final name of the command :)

no, this is just the WIP name while it is still behind a feature

vigoo · 2024-10-01T14:37:36Z

wasm-rpc-stubgen/src/commands/declarative.rs

+            commands::dependencies::add_stub_dependency(
+                &app.stub_wit(dep_component_name),
+                &app.component_wit(component_name),
+                true,  // TODO: is overwrite ever not needed?


I added it to protect the potentially user-written files from getting overwritten. I think in the declarative mode, especially if we do the next planned step which moves the user-defined part "inside", we can assume that all wit directories are managed by this tool so it can freely modify everything except the original user defined wit specs.

Added a note about always using true for this in declarative mode

noise64 · 2024-10-01T15:17:11Z

@vigoo for the ephemeral / durable: if it's okay to have incompatible yaml formats, the we can make it different here (as component-type is mandatory in this model)

Another option could be using "component" or "golem" as component type, and only add durability as a property in golem-cli. That way it would compatible and probably even easier to extend in golem-cli.

… and can be an optional property there) and use "wasm" naming

noise64 · 2024-10-02T08:01:51Z

After discussing with @vigoo:

removed durable and ephemeral (can be added as an extra prop in golem-cli)
unified naming on traits and oam components to use wasm, wasm-build, wasm-rpc, wasm-rpc-stub-build

…em sequentially to avoid parallel cargo build problems

noise64 added 3 commits September 24, 2024 13:14

add generic oam model and golem specific extensions

c4f1a57

fix component name in trait and skip unknown traits instead of failing

b0156a1

add declarative command stubs

c0b6f9e

jdegoes reviewed Sep 24, 2024

View reviewed changes

noise64 added 19 commits September 24, 2024 17:29

use test names for oam model tests

1debfa0

component type checks + naming

279fa67

rename model

1e71ffc

rework oam modal to support generic typed extractions and added initi…

da7ff0f

…al validation model

cargo fmt

49bf44d

checkpoint: validated models and component deps

172a2fd

cleanups

7730bc2

use git_version for WASM_RPC_VERSION, so local builds can be used wit…

b3c44c2

…hout overrides

minimal declarative pre-build impl

8efa05c

extract unknown property validation

cddd83a

comment

a1df774

full pre-build

37b2e66

format logs

e9d1b75

post-build

223b15a

common build properties and include glob pattern

56c08b8

update tests

532734b

format

69f9043

fix missing context pop

8e586a4

fmt

fbc675d

vigoo reviewed Oct 1, 2024

View reviewed changes

noise64 added 5 commits October 1, 2024 18:32

pre-build up-to-date checks

395877d

post-build up-to-date check + TODO for add stub

f252f9f

add UpdateCargoToml enum to support automatic updating

87e8f5c

drop durable and ephemeral (as it will be only relevant in golem-cli,…

9f0cbd3

… and can be an optional property there) and use "wasm" naming

fmt

d9402c9

noise64 added 10 commits October 2, 2024 10:57

validation / logging cleanups

3a94249

Adding optional build steps for components

c70472b

rename Declarative to App and add build (all) command

5ef99f0

cleanups

bf0cede

use wac plug instead of wasm-compose

677d6a9

command naming / messages

63240e4

add and use custom copy with modtime change

a785c74

command naming / doc

9d5067a

build step up-to-date checks + some more validation

86e91b1

fix test

f556df3

noise64 marked this pull request as ready for review October 3, 2024 12:41

noise64 added 7 commits October 4, 2024 11:19

force build flag + TODOS

3e18675

fix command process error handling

2d3ffef

fix unused plug tracking

7902821

log

abe3468

log / naming

e9ffd34

extract stubgen tests that use cargo to tests-integration, and run th…

ae6a479

…em sequentially to avoid parallel cargo build problems

fix integ test exclude

91d6d50

vigoo approved these changes Oct 7, 2024

View reviewed changes

afsalthaj approved these changes Oct 8, 2024

View reviewed changes

noise64 merged commit 9785eae into main Oct 8, 2024
2 checks passed

noise64 deleted the dec-dep branch October 8, 2024 06:16

jorgehermo9 mentioned this pull request Oct 18, 2024

Initial File System - Bounty-to-Hire golemcloud/golem#1004

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Declarative dependencies #85

Declarative dependencies #85

noise64 commented Sep 24, 2024 •

edited

Loading

jdegoes Sep 24, 2024

noise64 Sep 24, 2024

jdegoes Sep 24, 2024

noise64 Sep 24, 2024

jdegoes Sep 24, 2024

noise64 Sep 24, 2024

noise64 Sep 24, 2024

noise64 Sep 24, 2024

jdegoes Sep 24, 2024 •

edited

Loading

noise64 Sep 24, 2024

jdegoes Sep 24, 2024

noise64 Sep 24, 2024 •

edited

Loading

noise64 Sep 24, 2024

noise64 Sep 24, 2024

vigoo commented Oct 1, 2024

vigoo Oct 1, 2024

noise64 Oct 1, 2024

vigoo Oct 1, 2024

noise64 Oct 2, 2024

noise64 commented Oct 1, 2024 •

edited

Loading

noise64 commented Oct 2, 2024

	composedWasm: out/component-composed.wasm
	output-wasm: out/component-final.wasm

Declarative dependencies #85

Declarative dependencies #85

Conversation

noise64 commented Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdegoes Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noise64 Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vigoo commented Oct 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noise64 commented Oct 1, 2024 • edited Loading

noise64 commented Oct 2, 2024

noise64 commented Sep 24, 2024 •

edited

Loading

jdegoes Sep 24, 2024 •

edited

Loading

noise64 Sep 24, 2024 •

edited

Loading

noise64 commented Oct 1, 2024 •

edited

Loading