Enhancement: Houdini use instance transient data for `instance_node` #5616

BigRoy · 2023-09-13T19:54:21Z

Changelog Description

Use transient data to store the actual houdini node object in the instance as opposed to the path so hou.node() isn't needed to retrieve the node per plugin.

Also removed:

Instance toggled callback which does nothing with new publisher
Remove legacy collect instances plugin which is not used for new publisher

Additional notes

👇 Please consider this question

I wasn't sure whether I should open this as a Draft PR or not. I'm not sure whether everyone agrees this is a design improvement or whether it's worse. Thoughts?

☝️

Testing notes:

Use houdini to create and publish all families

…tance as opposed to the path

hound · 2023-09-13T19:54:32Z

openpype/hosts/houdini/plugins/publish/collect_usd_layers.py

@@ -56,7 +56,7 @@ def process(self, instance):
            layer_inst.data["subset"] = "__stub__"
            layer_inst.data["label"] = label
            layer_inst.data["asset"] = instance.data["asset"]
-            layer_inst.data["instance_node"] = instance.data["instance_node"]
+            layer_inst.data["instance_node"] = instance.data["transientData"]["instance_node"]


line too long (94 > 79 characters)

instance_node = instance.data["transientData"]["instance_node"] for layer in usdlib.get_configured_save_layers(rop_node): ... layer_inst.data["instance_node"] = instance_node ...

hound · 2023-09-13T19:54:32Z

openpype/hosts/houdini/plugins/publish/validate_no_errors.py

@@ -38,7 +38,7 @@ def process(self, instance):
        validate_nodes = []

        if len(instance) > 0:
-            validate_nodes.append(hou.node(instance.data.get("instance_node")))
+            validate_nodes.append(instance.data["transientData"]["instance_node"])


line too long (82 > 79 characters)

I think this should be:

instance_node = instance.data["transientData"].get("instance_node") if instance_node is not None: validate_nodes.append(instance_node)

hound · 2023-09-13T19:54:32Z

openpype/hosts/houdini/plugins/publish/validate_vdb_output_node.py

@@ -98,7 +98,7 @@ def process(self, instance):
                "Invalid VDB content: {}".format(message),
                formatting_data={
                    "message": message,
-                    "rop_path": instance.data.get("instance_node"),
+                    "rop_path": instance.data["transientData"]["instance_node"].path(),


line too long (87 > 79 characters)

instance_node = instance.data["transientData"]["instance_node"] formatting_data={ "message": message, "rop_path": instance_node.path(), "sop_path": output_path }

MustafaJafar · 2023-09-21T07:54:41Z

Will this PR be still useful if #5490 is merged ?

MustafaJafar

Tested as a User these product types work fine

camer
imsage sequence
ABC point cache
BGEO point cache
VDB cache
Review

These families (that submits to farm) error

Mantra
Karma

After applying this fix, it worked

OpenPype/openpype/modules/deadline/plugins/publish/submit_houdini_render_deadline.py

Line 125 in 23e9f5e

driver = hou.node(instance.data["instance_node"])

        driver = instance.data["transientData"]["instance_node"]

MustafaJafar · 2023-10-05T10:21:37Z

I was Misled and thought that instance.data["instance_node"] is no longer exists after merging Use host's node uniqueness for instance id in new publisher #5490, but I found out that it exists because of the following line:

OpenPype/openpype/hosts/houdini/plugins/publish/collect_instances.py

Line 83 in 2c7fc21

instance.data["instance_node"] = node.path()

So, This PR

Save instnace_node in instance.data["transientData"]["instance_node"] instead of instance.data["instance_node"]
removes collect_instances.py

where transientData are saved ? and how instance_node value is maintained/updated ?

BigRoy · 2023-10-05T10:33:56Z

The idea is that instance.data["instance_node"] should indeed not exist anymore since it's now transient data. That CollectInstances should've been removed some time ago :)

where transientData are saved ? and how instance_node value is maintained/updated ?

How do you mean? The Creators set it. They don't need to be 'maintained/updated' because it always represents the physical hou.Node that the Creator collects.

Save instnace_node in instance.data["transientData"]["instance_node"] instead of instance.data["instance_node"]

Yes, I mean. Not by choice really ;) I'd prefer it instance.data because it's shorter but it's by design for transient data in the new publisher.

MustafaJafar · 2023-10-05T11:11:28Z

Let me rephrase my questions:

what are transient data and how are they stored ?
what will be instance.transient_data["instance_node"] value when the rop is duplicated ? in the light of the recently merged PR Use host's node uniqueness for instance id in new publisher #5490

BigRoy · 2023-10-05T12:00:04Z

Transient Data

Transient Data was once added because all instance data generated by the Creator had to be json serializable - the hou.Node is not and transient data refers to the fact that it's runtime data that is really only valid for the current runtime session. You can't use that object instance after say, closing down houdini and opening it again.

It got added with this PR: #3897 where at first it was referred to as lifetime data but later got renamed to transient data.

It seems that Transient Data unfortunately was never added to the documentation: https://openpype.io/docs/dev_publishing - probably best to create an issue about that.

what will be instance.transient_data["instance_node"] value when the rop is duplicated ? in the light of the recently merged PR #5490

It will still refer to the ROP node, but by hou.Node instead of the node's string path. So it refers to the python hou.Node instance as opposed to a string path value.

It's likely however that some conflicts need to be resolved since that PR - but before anyone would do so I'd rather first know whether this PR is even remotely worth the effort and whether it's a good design choice.

As a reference, Fusion also uses the transientData and that's what it was initially added for so you knew exactly what object you were referring to since just a node name/path there wasn't sufficient since Fusion allows to have multiple workfiles open at the same time in a single session. 🤯

MustafaJafar · 2023-10-05T16:35:30Z

It seems that Transient Data unfortunately was never added to the documentation: https://openpype.io/docs/dev_publishing - probably best to create an issue about that.

yeah, I will add that to the community guide AYON / Openpype Publish process - Development guide

When I started learning about Openpype 4 or 5 months ago, I didn't expect all these details!
and I love that!

MustafaJafar · 2023-10-05T17:49:52Z

That's pretty interesting!
transient data: These data are not stored to scene and will be lost on object deletion.
let, me re-draw this uml diagram from my guide.

I can recognize these three types of data, they are used in the Houdini creator.

So, running this in Houdini's python source editor, can possibly tell us something.
I don't know how it works but it's working, I tried opening old files and run the following script and it seems to catch old instances too.

from openpype.pipeline.create import CreateContext
from openpype.pipeline import registered_host

host = registered_host()
context = CreateContext(host, reset=True)

for instance in context.instances:
    print(instance.subset_name, instance.transient_data)

how Openpype is able to retrieve these transient data ??

BigRoy · 2023-10-05T18:36:29Z

The Creator collects the transientData - exactly as shown in this PR. Based on searching for the nodes relevant to the creator identifier the Creator assumes this is an instance node belonging to this creator - and then stores the hou.Node() in transient data, exactly like how previously the node's path was stored in the instance data.

Personally I see it like this: The only reason for me that hou.Node() is in transient data is because it can't be converted to JSON data and thus by design it has to be put in transientData.

But transient data tells a bit more as well - it says 'do not store these on the node' - which happens to be the case for non-JSON data (because we can't serialize it well) but also for this instance node. We don't need to store it on the node, because it is the node.

Not sure how to better explain it.

I know @iLLiCiTiT has a view of his own on it and likely can explain it better in a different way.

I don't know how it works but it's working, I tried opening old files and run the following script and it seems to catch old instances too.

Exactly - the Creator already knows how to find the nodes, that logic never relied on the node's path being stored on the node, etc. All the Creator does was "find all nodes with attribute creator_identifier - if identifier matches this Creator class then it's an instance of this Creator. That's all how they are found. That hasn't changed.

The fact that instance_node was stored as a parm on the node was - in my view - a design flaw / bug from the beginning so it's good that since instance_node would be the actual node we're querying the parm from haha, so we don't need to store that data.

Similarly we can have frameStart and frameEnd instance data collected by the Creator but when updating the instance not store the data in the JSON data parms but directly into the ROP nodes time parameters so that we frame start and end can both be set from the publisher AND on the Houdini ROP node. (This is sligthly more complex due to the fact that in Houdini the values can also be expression so the UI would have to likely show it as a TextDef, etc.) Anyway, I deviate too much. I hope this helps.

MustafaJafar · 2023-10-05T18:55:25Z

The Creator collects the transientData - exactly as shown in this PR. Based on searching for the nodes relevant to the creator identifier the Creator assumes this is an instance node belonging to this creator - and then stores the hou.Node() in transient data, exactly like how previously the node's path was stored in the instance data.

I think I realized how it works, Thanks!

So, there are two ways to create instances in Houdini

HoudiniCreator(...).create(...) # which creates nodes and instances from user input
HoudiniCreator(...).collect_instances() # which create instances from existing nodes

and this snippet some where inside calls HoudiniCreator(...).collect_instances() which recreate instances from existing openpype/ayon rop nodes

from openpype.pipeline.create import CreateContext
from openpype.pipeline import registered_host

host = registered_host()
context = CreateContext(host)

for instance in context.instances:
    print(instance.subset_name, instance.transient_data)

Eureka !

OpenPype/openpype/pipeline/create/context.py

Line 2141 in 12f4128

creator.collect_instances()

MustafaJafar · 2023-10-06T11:33:12Z

I like the idea introduced by this PR, I'm amazed that same approach is followed in Nuke, Fusion and also Maya.

BigRoy · 2023-11-15T12:47:08Z

Would you just need me to resolve the conflicts, or are there other things to implement?

MustafaJafar · 2023-11-15T13:08:18Z

Nah, It's a casual PR revisiting during PRs cleaning up.
what is the status of this PR :D ? and what's missing ?

BigRoy · 2023-11-15T13:44:22Z

Is it worth our while to refactor like this. Since it doesn't change anything end-user wise it doesn't bring any new features. So, biggest question is - do we want this?

If so, I can resolve conflicts and prepare it for testing.

MustafaJafar · 2023-11-20T09:27:26Z

So, biggest question is - do we want this?

I'm afraid I can't answer this question, I'll leave it to @moonyuet or @antirotor
but, I can tell that this PR is helpful because plugins/publish/collect_instances.py is redundant as it does the same work as HoudiniCreator.collect_instances() .

when dealing with contexts HoudiniCreator.collect_instances() is called as mentioned here #5616 (comment)
and we only need plugins/publish/collect_instances.py when publishing which is an overwork.

So, my opinion this PR is helpful and I like it.

iLLiCiTiT · 2023-12-21T09:33:20Z

Is this PR still relevant? I think most of the changes are, but not sure if are up to date.

mkolar · 2024-02-07T22:46:21Z

Not sure how to deal with this one right now. Lot's of conflicts, but probably a reasonable change. @antirotor I might need your guidance here.

Seems that it should be OP and AYON compatible out of the box though

mkolar · 2024-02-09T23:47:53Z

Because we're splitting OpenPype into ayon-core and individual host addons, this PR would have to be re-created to target one of those.

We're closing it down, but we'll he happy for a new PR to ynput/ayon-core or the host addon repository once it's up.

BigRoy added 3 commits September 13, 2023 21:25

Remove instance toggled callback (not used for new publisher)

e52c648

Remove legacy collect instances

ff3d96e

Use transient data to store the actual houdini node object in the ins…

da23323

…tance as opposed to the path

BigRoy added host: Houdini type: refactor Structural changes not affecting functionality community contribution labels Sep 13, 2023

BigRoy requested review from MustafaJafar, antirotor and iLLiCiTiT September 13, 2023 19:54

hound bot reviewed Sep 13, 2023

View reviewed changes

ynbot added type: enhancement Enhancements to existing functionality size/S Denotes a PR changes 100-499 lines, ignoring general files labels Sep 13, 2023

MustafaJafar suggested changes Sep 21, 2023

View reviewed changes

MustafaJafar mentioned this pull request Sep 29, 2023

Enhancement : Houdini dynamic publish parameters #5666

Closed

MustafaJafar assigned BigRoy Nov 15, 2023

iLLiCiTiT requested a review from MustafaJafar December 21, 2023 09:32

mkolar added target: OpenPype target: AYON port to AYON labels Feb 7, 2024

mkolar closed this Feb 9, 2024

ynbot added this to the next-patch milestone Feb 9, 2024

jakubjezek001 removed this from the next-patch milestone Feb 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement: Houdini use instance transient data for `instance_node` #5616

Enhancement: Houdini use instance transient data for `instance_node` #5616

BigRoy commented Sep 13, 2023 •

edited

Loading

hound bot Sep 13, 2023

iLLiCiTiT Dec 21, 2023

hound bot Sep 13, 2023

iLLiCiTiT Dec 21, 2023

hound bot Sep 13, 2023

iLLiCiTiT Dec 21, 2023 •

edited

Loading

MustafaJafar commented Sep 21, 2023

MustafaJafar left a comment •

edited

Loading

MustafaJafar commented Oct 5, 2023

BigRoy commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

BigRoy commented Oct 5, 2023

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

BigRoy commented Oct 5, 2023

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 6, 2023 •

edited

Loading

BigRoy commented Nov 15, 2023

MustafaJafar commented Nov 15, 2023

BigRoy commented Nov 15, 2023

MustafaJafar commented Nov 20, 2023 •

edited

Loading

iLLiCiTiT commented Dec 21, 2023

mkolar commented Feb 7, 2024

mkolar commented Feb 9, 2024

Enhancement: Houdini use instance transient data for instance_node #5616

Enhancement: Houdini use instance transient data for instance_node #5616

Conversation

BigRoy commented Sep 13, 2023 • edited Loading

Changelog Description

Additional notes

Testing notes:

hound bot Sep 13, 2023

Choose a reason for hiding this comment

iLLiCiTiT Dec 21, 2023

Choose a reason for hiding this comment

hound bot Sep 13, 2023

Choose a reason for hiding this comment

iLLiCiTiT Dec 21, 2023

Choose a reason for hiding this comment

hound bot Sep 13, 2023

Choose a reason for hiding this comment

iLLiCiTiT Dec 21, 2023 • edited Loading

Choose a reason for hiding this comment

MustafaJafar commented Sep 21, 2023

MustafaJafar left a comment • edited Loading

Choose a reason for hiding this comment

MustafaJafar commented Oct 5, 2023

BigRoy commented Oct 5, 2023 • edited Loading

MustafaJafar commented Oct 5, 2023 • edited Loading

BigRoy commented Oct 5, 2023

Transient Data

MustafaJafar commented Oct 5, 2023 • edited Loading

MustafaJafar commented Oct 5, 2023 • edited Loading

BigRoy commented Oct 5, 2023

MustafaJafar commented Oct 5, 2023 • edited Loading

MustafaJafar commented Oct 6, 2023 • edited Loading

BigRoy commented Nov 15, 2023

MustafaJafar commented Nov 15, 2023

BigRoy commented Nov 15, 2023

MustafaJafar commented Nov 20, 2023 • edited Loading

iLLiCiTiT commented Dec 21, 2023

mkolar commented Feb 7, 2024

mkolar commented Feb 9, 2024

Enhancement: Houdini use instance transient data for `instance_node` #5616

Enhancement: Houdini use instance transient data for `instance_node` #5616

BigRoy commented Sep 13, 2023 •

edited

Loading

iLLiCiTiT Dec 21, 2023 •

edited

Loading

MustafaJafar left a comment •

edited

Loading

BigRoy commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 5, 2023 •

edited

Loading

MustafaJafar commented Oct 6, 2023 •

edited

Loading

MustafaJafar commented Nov 20, 2023 •

edited

Loading