feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference #1142

acl-cqc · 2024-05-30T15:43:39Z

This is probably phase 1 of 4, later steps being

Remove input_extensions and NodeType
Make every operation require it's own extension (Custom operations implicitly require their extension #388)
Infer deltas for parent nodes (Infer extension-requirements for nested DFGs #640) - this may be disruptive, and/or take some subtlety, as currently every FunctionType stores an ExtensionSet not an Option thereof
Finally, remove the feature flag

So, this PR updates validation to ignore the input_extensions, and removes the inference algorithm that would set them. There were a few complications:

A lot of tests create a DFGBuilder with an empty delta, then put things in it that have extension requirements. The inference algorithm figures out that this is all OK if it can just put that extension-requirement on the input_extensions to the DFG node (root) itself. So this might be a call for input_extensions - they do reduce the need for Infer extension-requirements for nested DFGs #640 a bit.
- We can see roughly how painful life is (without delta-inference nor input-extensions) looking at the various extra extension-deltas I've had to specify here
- I think that given we are under the feature flag at the moment, we are OK to continue, however the need for Infer extension-requirements for nested DFGs #640 is now somewhat increased, so discussion there strongly encouraged, please! :)
TailLoop turned out not to have an extension-delta, where it clearly needs one (just as DataflowBlock and others). Also there was an implementation of OpParent for it, whereas in fact we needed DataflowParent as that would also have got us the ValidateOp for free. This all in 08bb5a5.
Another bug in DataflowBlock::inner_signature, and some others.

....That is to say, in some ways this validation scheme is stricter than what we had; there is both the slight flexibility that input_extensions gave us in mis-specifying deltas, and that this scheme's simplicity makes it much harder to accidentally omit cases from validation....

I've also kept Hugr::infer_extensions around as a no-op rather than remove it and later bring it back (in #640). Maybe we should remove it, but that would be a breaking change....OTOH I've removed validate_with_extension_closure and in theory we could have that (a closure containing deltas for nested DFGs) too...

BREAKING CHANGE: TailLoop node and associated builder functions now require specifying an ExtensionSet; extension/validate.rs deleted; some changes to Hugrs validated/rejected when the extension_inference feature flag is turned on

This reverts commit 8d117e8.

…bset(parent)

…taflowParent

codecov · 2024-05-30T15:45:26Z

Codecov Report

Attention: Patch coverage is 94.53925% with 16 lines in your changes missing coverage. Please review.

Project coverage is 86.68%. Comparing base (15c3ea9) to head (546ced4).
Report is 2 commits behind head on main.

Files	Patch %	Lines
hugr-core/src/hugr/validate/test.rs	93.37%	3 Missing and 8 partials ⚠️
hugr-core/src/builder/tail_loop.rs	81.81%	0 Missing and 2 partials ⚠️
hugr-core/src/hugr/validate.rs	91.30%	2 Missing ⚠️
hugr-core/src/ops/constant.rs	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1142      +/-   ##
==========================================
- Coverage   87.09%   86.68%   -0.42%     
==========================================
  Files          97       94       -3     
  Lines       19297    17808    -1489     
  Branches    18432    16942    -1490     
==========================================
- Hits        16807    15437    -1370     
+ Misses       1636     1601      -35     
+ Partials      854      770      -84

Flag	Coverage Δ
python	`89.37% <100.00%> (+0.01%)`	⬆️
rust	`86.54% <94.52%> (-0.45%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

First is subsumed by parent_io_mismatch Second we don't care (delta broader than necessary)

… featflag)

acl-cqc · 2024-06-07T16:30:06Z

hugr-core/src/hugr/validate/test.rs

@@ -865,11 +864,158 @@ fn test_polymorphic_load() -> Result<(), Box<dyn std::error::Error>> {
    Ok(())
 }

+#[test]
+/// Validation errors in a controlflow subgraph.
+fn cfg_children_restrictions() {


I've moved this out of extension_tests as it has nothing to do with extensions

acl-cqc · 2024-06-07T16:30:34Z

hugr-core/src/hugr/validate/test.rs

+//          /->->\
+//          |    |
+// Entry -> Middle -> Exit
+fn cfg_connections() -> Result<(), Box<dyn std::error::Error>> {


This one is kinda new, but also has a kinda connection with one I've removed

acl-cqc · 2024-06-07T16:32:43Z

hugr-core/src/hugr/validate/test.rs

+    #[rstest]
+    #[case(make_bb, |bb: &mut DFGWrapper<_,_>, outs| bb.make_tuple(outs))]
+    #[case(make_tailloop, |tl: &mut DFGWrapper<_,_>, outs| tl.make_break(tl.loop_signature().unwrap().clone(), outs))]
+    fn bb_extension_mismatch<T>(


Note the <T> here is inferred, so we never have to write it, but is different for make_bb and make_tuple, so switching between the two internally is more awkward than one might think. That type inference allows this is a nice side effect of rstest case :)

ss2165

looks great thanks 👍

Following #1142 the `input_extensions` are unused, so remove the storage for them BREAKING CHANGE: * `add_child_op`(`_with_parent`), etc., gone; use `add_child_node`(`_with_parent`) with an (impl Into-)OpType. * `get_nodetype` gone - use `get_optype`. * `NodeType` gone - use `OpType` directly. * Various (Into<)Option<ExtensionSet> params removed from builder methods especially {cfg_,dfg_}builder. * `input_extensions` removed from serialization schema.

Removes the conditional test skip introduced in #1168 (comment), now that #1142 is merged.

## 🤖 New release * `hugr`: 0.5.1 -> 0.6.0 * `hugr-core`: 0.2.0 -> 0.3.0 * `hugr-passes`: 0.2.0 -> 0.3.0 * `hugr-cli`: 0.1.1 -> 0.1.2 <details><summary>Changelog</summary> ## `hugr` <blockquote> ## 0.6.0 (2024-06-28) ### Bug Fixes - SimpleReplacement panic on multiports ([#1191](#1191)) - Add some validation for const nodes ([#1222](#1222)) - Cfg not validating entry/exit types ([#1229](#1229)) - `extract_hugr` not removing root node ports ([#1239](#1239)) ### Documentation - Fix documentation of `ValidationError::ConstTypeError` ([#1227](#1227)) ### Features - CircuitBuilder::add_constant ([#1168](#1168)) - [**breaking**] Make the rewrite errors more useful ([#1174](#1174)) - [**breaking**] Validate Extensions using hierarchy, ignore input_extensions, RIP inference ([#1142](#1142)) - [**breaking**] Infer extension deltas for Case, Cfg, Conditional, DataflowBlock, Dfg, TailLoop ([#1195](#1195)) - Helper functions for requesting inference, use with builder in tests ([#1219](#1219)) ### Refactor - [**breaking**] FunctionBuilder takes impl Into<PolyFuncType> ([#1220](#1220)) - [**breaking**] Remove NodeType and input_extensions ([#1183](#1183)) </blockquote> ## `hugr-core` <blockquote> ## 0.3.0 (2024-06-28) ### Bug Fixes - SimpleReplacement panic on multiports ([#1191](#1191)) - Add some validation for const nodes ([#1222](#1222)) - Cfg not validating entry/exit types ([#1229](#1229)) - `extract_hugr` not removing root node ports ([#1239](#1239)) ### Documentation - Fix documentation of `ValidationError::ConstTypeError` ([#1227](#1227)) ### Features - CircuitBuilder::add_constant ([#1168](#1168)) - [**breaking**] Make the rewrite errors more useful ([#1174](#1174)) - [**breaking**] Validate Extensions using hierarchy, ignore input_extensions, RIP inference ([#1142](#1142)) - [**breaking**] Infer extension deltas for Case, Cfg, Conditional, DataflowBlock, Dfg, TailLoop ([#1195](#1195)) - Helper functions for requesting inference, use with builder in tests ([#1219](#1219)) ### Refactor - [**breaking**] Remove NodeType and input_extensions ([#1183](#1183)) - [**breaking**] FunctionBuilder takes impl Into<PolyFuncType> ([#1220](#1220)) </blockquote> ## `hugr-passes` <blockquote> ## 0.3.0 (2024-06-28) ### Features - [**breaking**] Validate Extensions using hierarchy, ignore input_extensions, RIP inference ([#1142](#1142)) - Helper functions for requesting inference, use with builder in tests ([#1219](#1219)) </blockquote> ## `hugr-cli` <blockquote> ## 0.1.1 (2024-06-07) ### Features - Reexport `clap::Parser` and `clap_verbosity_flag::Level` from hugr_cli ([#1146](#1146)) ### Refactor - Move binary to hugr-cli ([#1134](#1134)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/). --------- Co-authored-by: Douglas Wilson <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.3.0](hugr-py-v0.2.1...hugr-py-v0.3.0) (2024-07-03) ### ⚠ BREAKING CHANGES * * `add_child_op`(`_with_parent`), etc., gone; use `add_child_node`(`_with_parent`) with an (impl Into-)OpType. * `get_nodetype` gone - use `get_optype`. * `NodeType` gone - use `OpType` directly. * Various (Into<)Option<ExtensionSet> params removed from builder methods especially {cfg_,dfg_}builder. * `input_extensions` removed from serialization schema. * the Signature class is gone, but it's not clear how or why you might have been using it... * TailLoop node and associated builder functions now require specifying an ExtensionSet; extension/validate.rs deleted; some changes to Hugrs validated/rejected when the `extension_inference` feature flag is turned on * Type::validate takes extra bool (allow_rowvars); renamed {FunctionType, PolyFuncType}::(validate=>validate_var_len). ### Features * Allow "Row Variables" declared as List<Type> ([#804](#804)) ([3ea4834](3ea4834)) * **hugr-py:** add builders for Conditional and TailLoop ([#1210](#1210)) ([43569a4](43569a4)) * **hugr-py:** add CallIndirect, LoadFunction, Lift, Alias ([#1218](#1218)) ([db09193](db09193)), closes [#1213](#1213) * **hugr-py:** add values and constants ([#1203](#1203)) ([f7ea178](f7ea178)), closes [#1202](#1202) * **hugr-py:** automatically add state order edges for inter-graph edges ([#1165](#1165)) ([5da06e1](5da06e1)) * **hugr-py:** builder for function definition/declaration and call ([#1212](#1212)) ([af062ea](af062ea)) * **hugr-py:** builder ops separate from serialised ops ([#1140](#1140)) ([342eda3](342eda3)) * **hugr-py:** CFG builder ([#1192](#1192)) ([c5ea47f](c5ea47f)), closes [#1188](#1188) * **hugr-py:** define type hierarchy separate from serialized ([#1176](#1176)) ([10f4c42](10f4c42)) * **hugr-py:** only require input type annotations when building ([#1199](#1199)) ([2bb079f](2bb079f)) * **hugr-py:** python hugr builder ([#1098](#1098)) ([23408b5](23408b5)) * **hugr-py:** store children in node weight ([#1160](#1160)) ([1cdaeed](1cdaeed)), closes [#1159](#1159) * **hugr-py:** ToNode interface to treat builders as nodes ([#1193](#1193)) ([1da33e6](1da33e6)) * Validate Extensions using hierarchy, ignore input_extensions, RIP inference ([#1142](#1142)) ([8bec8e9](8bec8e9)) ### Bug Fixes * Add some validation for const nodes ([#1222](#1222)) ([c05edd3](c05edd3)) * **hugr-py:** more ruff lints + fix some typos ([#1246](#1246)) ([f158384](f158384)) * **py:** get rid of pydantic config deprecation warnings ([#1084](#1084)) ([52fcb9d](52fcb9d)) ### Documentation * **hugr-py:** add docs link to README ([#1259](#1259)) ([d2a9148](d2a9148)) * **hugr-py:** build and publish docs ([#1253](#1253)) ([902fc14](902fc14)) * **hugr-py:** docstrings for builder ([#1231](#1231)) ([3e4ac18](3e4ac18)) ### Code Refactoring * Remove "Signature" from hugr-py ([#1186](#1186)) ([65718f7](65718f7)) * Remove NodeType and input_extensions ([#1183](#1183)) ([ea5213d](ea5213d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: hugrbot <[email protected]> Co-authored-by: Seyon Sivarajah <[email protected]>

acl-cqc added 14 commits May 30, 2024 12:47

WIP start removing NodeType

8d117e8

Revert "WIP start removing NodeType"

974236b

This reverts commit 8d117e8.

Drop validate_with_extension_closure...may want to revert eventually

16c72e5

Rm extension/validate.rs (ExtensionValidator), just check child.is_su…

759af86

…bset(parent)

Fix DataflowBlock::inner_signature (fixes 2*cfg tests in infer.rs)

07c09cd

Fix tail_loop tests by adding extension_delta to TailLoop and impl Da…

08bb5a5

…taflowParent

Fix/regenerate schema for TailLoop

d4e92d8

Fix constant.rs test_sum

968cc15

Fix prelude.rs tests

e744527

Fix constant.rs test

7aa06b1

Fix merge_bbs tests

fcd10c1

Calculate delta in const_fold

664112f

const_fold/tests.rs via helper functions

fffacf9

const_fold/test.rs: drop first param to helper

3367490

acl-cqc changed the title ~~Validate Extensions using hierarchy (ignore input_extensions)~~ feat!: Validate Extensions using hierarchy (ignore input_extensions) May 30, 2024

acl-cqc requested review from croyzor and ss2165 May 30, 2024 15:48

acl-cqc marked this pull request as ready for review May 30, 2024 15:48

acl-cqc requested a review from a team as a code owner May 30, 2024 15:48

acl-cqc added 8 commits May 31, 2024 01:02

Generalize parent_io_mismatch to cover Case, DFG and FuncDefn

8a9628f

Drop funcdefn_signature_mismatch(,2) tests

a6e6459

First is subsumed by parent_io_mismatch Second we don't care (delta broader than necessary)

Don't call infer_extensions; drop broken tests

abe27fb

Rm infer.rs, infer/test.rs, infer_extensions, instantiate, etc. (keep…

87b80d3

… featflag)

Bonus (BREAKING): remove io_extensions()

5715ea1

Reinstate ExtensionError but as a struct

d605c5b

And keep infer_extensions around as a No-op

e691dea

Test for CFG, BB

4b61ff0

acl-cqc changed the title ~~feat!: Validate Extensions using hierarchy (ignore input_extensions)~~ feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference May 31, 2024

Be more explicit in ModuleRoot check

19e64a9

acl-cqc commented Jun 7, 2024

View reviewed changes

clippy

1bf1c14

acl-cqc mentioned this pull request Jun 7, 2024

Tracking issue for extension requirement inference #426

Open

22 tasks

Try to fix 1.75 test

546ced4

acl-cqc mentioned this pull request Jun 10, 2024

refactor!: Remove NodeType and input_extensions #1183

Merged

ss2165 approved these changes Jun 11, 2024

View reviewed changes

acl-cqc added this pull request to the merge queue Jun 11, 2024

Merged via the queue into main with commit 8bec8e9 Jun 11, 2024
22 checks passed

acl-cqc deleted the extension_hierarchy branch June 11, 2024 18:02

This was referenced Jun 11, 2024

chore(main): release hugr-py 0.3.0 #1092

Merged

chore: release #1181

Closed

aborgna-q mentioned this pull request Jun 12, 2024

chore: Unskip test now that input_extensions is no more #1187

Merged

This was referenced Jun 13, 2024

feat!: Infer extension deltas for Case, Cfg, Conditional, DataflowBlock, Dfg, TailLoop #1195

Merged

bug: Two call nodes for the same function must have matching resources #695

Closed

Extension Inference for functions relies on builder #702

Closed

github-merge-queue bot pushed a commit that referenced this pull request Jun 14, 2024

chore: Unskip test now that input_extensions is no more (#1187)

cbe31be

Removes the conditional test skip introduced in #1168 (comment), now that #1142 is merged.

acl-cqc mentioned this pull request Jun 24, 2024

feat: Helper functions for requesting inference, use with builder in tests #1219

Merged

hugrbot mentioned this pull request Jun 28, 2024

chore: release #1233

Closed

acl-cqc mentioned this pull request Jun 28, 2024

Infer extension constraints on FuncDefn; remove many ExtensionSets from builder #739

Closed

hugrbot mentioned this pull request Jun 28, 2024

chore: release #1240

Merged

hugrbot mentioned this pull request Jul 1, 2024

chore: release #1248

Merged

acl-cqc mentioned this pull request Jul 3, 2024

When building a TailLoop, calling outputs_arr, you are given an OrderEdge "output wire" #1257

Closed

aborgna-q mentioned this pull request Jul 8, 2024

chore(rs): Update to hugr 0.6 CQCL/tket2#463

Closed

acl-cqc mentioned this pull request Oct 2, 2024

(Maybe?) Extension Inference: infer one layer at a time #918

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference #1142

feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference #1142

acl-cqc commented May 30, 2024 •

edited

Loading

codecov bot commented May 30, 2024 •

edited

Loading

acl-cqc Jun 7, 2024

acl-cqc Jun 7, 2024

acl-cqc Jun 7, 2024

ss2165 left a comment

feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference #1142

feat!: Validate Extensions using hierarchy, ignore input_extensions, RIP inference #1142

Conversation

acl-cqc commented May 30, 2024 • edited Loading

codecov bot commented May 30, 2024 • edited Loading

Codecov Report

acl-cqc Jun 7, 2024

Choose a reason for hiding this comment

acl-cqc Jun 7, 2024

Choose a reason for hiding this comment

acl-cqc Jun 7, 2024

Choose a reason for hiding this comment

ss2165 left a comment

Choose a reason for hiding this comment

acl-cqc commented May 30, 2024 •

edited

Loading

codecov bot commented May 30, 2024 •

edited

Loading