Deeper hierarchy in templates and paths

[mkolar]

Issue

Currently it's doesn't seem possible to create more complex hierarchies using avalon than the one in the default config https://getavalon.github.io/2.0/reference/#project-configuration-api

A deeper hierarchy is often needed, especially for bigger projects like features and episodic TV.
{root}/{project}/{silo}/{asset}/work/{task}/{app}
with simple episodes this could still work, having each episode as a silo
{root}/{project}/episodes/{silo}/{asset}/work/{task}/{app}
But there are also more complicated scenarios:

- Development
    - Concept
    - Teaser
        - Animatic
        - Previz
        - Shots
            - sh010
            - sh020

now to get to those shots in a template it might even be hard to specify the member names
{root}/{Folder1}/{Folder2}/{Folder3}/{shot}/work/{task}

Proposal

The system we use outside avalon right now is to save anything above shot/asset to a member called hierarchy. Our templates then look like this
{root}/{hierarchy}/{shot}/work/{task} where hierarchy = "development/teaser/shots

In term of avalon this path member could always contain silo and could prepend any other visual parents or categories. It would allow creating completely arbitrary folder structures.

This whole thing stems from the fact, that we're technically unable to convert our ftrack project structures to avalon, because it's lacking the granularity we need.

[BigRoy]

Visually it's possible to have a deeper hierarchy by hooking it up with a visualParent. For example the project manager can create "nested" items by selecting a previously existing asset and then "adding" an asset. These new nested assets get their parent prefixed in the asset name {parent}_{asset} - this could potentially also be swapped to {parent}/{asset} -> where if the parent would also be prefixed with a potential parent would allow any deep nesting, for example the parent could be s01e01/seq01 and then this asset is shot 010 within that sequence. The full name then becomes s01e01/seq01/010. Yet this could introduce some ambiguity, see below.

I'm not sure if the forward slash is supported in the asset's name, but if it is then even the folders created on disk would allow to "contain" assets. The only problem there would be what if you create an asset called work within a parent asset and your template also defines the work folder as work within an asset. They would end up being the same folder. That's why I had opted to, at least on disk, make it flat, yet make the browseable hierarchy visible in the interfaces.

That's also one of the reason to "prefix" the parent in the asset's name. But the other reason is: that asset names in the database have to be unique, so for example 010 couldn't end up twice in the database - even if nested because in the database it would always remain flat.

So on disk the above example ends up being:

s01e01
s01e01_seq01
s01e01_seq01_010
s01e01_seq01_020
s01e01_seq01_030

Yet in the pipeline it would visualize as:

s01e01
  > seq01
    > 010
    > 020
    > 030

Does that make any sense?

[mkolar]

That on its own makes sense and is a nice feature I guess, but certainly not very intuitive.

I've tried tweaking the project manager to use / instead of _ to see your theory could work, however that crashes the manager when you try creating an asset.

I understand the sentiment to keep it flat on the disk, however, with a long-term project (52 episode tv show...) we're talking well over 5000 shots in one folder. Artist might be able to get past that if everything is connected via avalon (which is a noble target, but we can't count on that), but production would go crazy, I included.

Preventing the work problem you described could be a question of reserved names. These could even be specified in the project config and system then shouldn't allow you to create a silo or asset of the same name.

Unique asset names should certainly be enforced, but I don't share the opinion that prefixing them automatically is the way to go. It creates confusion where you think you're creating asset 101sh010, but ep101_sq01_101sh010 get created. It's also making it somewhat harder to see what is the 'hierarchy' part of the name and what is the actual asset name in cases where you use underscore in the asset name itself.

I see that every objects parent right now points to the project. What is the point of the parent data then? Why not allow to point to its actual hierarchical parent? Wouldn't that inherently allow arbitrary hierarchies out of the box and allow traversing them? I'd say that storing a few more attributes (parent: 'actual parent', parents: 'sorted list of all the parents') on the database entries could effectively solve what @BigRoy has solved in the project manager with a 'hack' of visual parents.

It would be good to differentiate between assets and 'folders' in the database of course but that should be trivial, by simply creating a new object type. silos still makes sense in this approach as they differentiate between the main chunks of work like 'assets', 'episodes', 'promo'....whatever

[mottosso]

Thanks @mkolar, arbitrary directory depth is supported in the design of Avalon, it just hasn't been implemented yet.

The silo member is meant to foreshadow how arbitrary members of the hierarchy is meant to work. Internally, silo is just data. It's special data, because it's data you are able to access through the directory template, like {silo}. But I'd for all data to be accessible this way, such that you can access {sequence}, {episode} and what not, including a {hierarchy}.

The parent you see in the database isn't the parent directory on a filesystem, but rather the "pipeline parent". It's the one you see here.

The "pipeline parent", a.k.a. "object model" is then reflected onto a filesystem in whatever way makes the most sense for that filesystem via the path templates; but could also be drawn differently in a browser, on a web page or a diagram etc. The object model is important when you program towards the pipeline, as you can make assumptions without taking into consideration a hierarchy meant to simplify visual browsing in e.g. Windows Explorer. E.g. the parent of any Asset is always the project, and the parent of any Version is always the Subset etc. It remains fixed regardless of the visual hierarchy such that your tools will work across any hierarchy.

So, finally, to work around the issues you are having at the moment, you could either try treating silo as your hierarchy, and give it a value such as film\sequence1\episode12. Alternatively, you could take a look at how you could expose another member to the path template, and then another, and then ideally get a generic system going. I was initially thinking that the path template could have access to all of .data[] which may still be intuitive. It'd be a matter of passing it to the folder create method, as template = work_template.format(**all_data), for example here.

[mottosso]

with a 'hack' of visual parents.

I'd keep an open mind as to what qualifies as a hack. :) Think about this, is a hierarchy drawn in Windows Explorer any different from the "visual hierarchy" drawn in the project manager? What makes them different? To my mind, these are both equal "hacks" in the sense that they visualise an otherwise flat series of numbers in a datastore; one a file system, the other a MongoDB database.

We can use that to reap the benefits of visualisation, without getting bogged down by thinking hierarchically in our tools. Best of both worlds!

[mkolar]

All right then :), I think I'm getting your approach here.

Extra data members that hold this information, should be straightforward and clear.

I was initially thinking that the path template could have access to all of .data[] which may still be intuitive. It'd be a matter of passing it to the folder create method, as template = work_template.format(**all_data)

That's exactly what I'd imagine.

It looks like the formatting is happening ad-hoc in multiple places as a direct string.format(). Was there a reason not to use something more robust like lucidity or pather? Just out of interest.

Anyways I'll dig through the code to see how many things would be influenced by this and try to add hierachy data member, for now, that should at least allow for arbitrary nesting in the most rudimentary fashion.

@BigRoy are you currently using the visualParent to categorize things in the loader, or are you reconstructing the hierarchy from the asset name?

[mottosso]

Was there a reason not to use something more robust like lucidity or pather?

Those are relevant when going the other way; from getting data out of a path. When putting data into a path, I don't know if there is any advantage to those. In Avalon, the path shouldn't have to be used as a container of data; all relevant data is accessed via the database. If you find yourself needing to get data out of a path, then that's a bug.

[mottosso]

It looks like the formatting is happening ad-hoc in multiple places as a direct string.format().

Perspective perspective. :) I don't know if .format qualifies as ad-hoc any more than lucidity or pather. .format is part of the standard library after all.

[BigRoy]

@BigRoy are you currently using the visualParent to categorize things in the loader, or are you reconstructing the hierarchy from the asset name?

Definitely using the visualParent for hierarchy. I'm not parsing underscores or anything or splitting on strings, always going from this "explicit" data in the database.

And I'm using the label for the name (if present in .data) else it just falls back on the name.

So:

s01e01
- shot010 

The shot010 would be "unique asset name": s01e01_shot010 yet readable and labeled as just shot010 (which doesn't have to be unique).

[mkolar]

So I got it working quite well now. I'll clean it up a bit a push so we can discuss the approach. I've just added one hierarchy key to the template which I build on the fly before formatting from hierarchy data member on the asset that holds a list of all the parents.

When I create all the parent directories as assets and link them via visualParent everything works in the loader as well, so that's all cool.

It does introduce one strange inconsistency, which doesn't affect anything functionally, but should probably be addressed

We have silo and visualParent. They both act as some way of categorization of assets, however, visualParent must be an asset in the database, while silo is just metadata on the assets and on its own doesn't actually exist in the db.

Wouldn't it make sense to unify these and either make them all just metadata, or make silos assets too?

[mottosso]

Wouldn't it make sense to unify these and either make them all just metadata, or make silos assets too?

Isn't visualParent also just metadata? Making silo into an asset is also an option; only it might not make sense to start working on one, but that could be an organisational limit rather than a technical one.

[mottosso]

Shouldn't be any technical reason for visualParent to require the parent it's pointing at to exist, but it'd make sense for it to do so. I think this is merely how the GUI is implemented, and doesn't put constraints on the system itself. @BigRoy?

[mkolar]

Sure. Functionally of the system isn't an issue at all as you said. I'm only mentioning it for consistency reasons.

I'll refer to this hierarchy below.

- episodes (silo metadata)
   -> ep101 (asset in db)
      -> sq01 (asset in db)
         ->101sh010 (asset in db)
         ->101sh020 (asset in db)
      -> sq02

Shouldn't be any technical reason for visualParent to require the parent it's pointing at to exist,

True but I just realized, that some changes would have to be made to how e.g. the loader creates the visual hierarchy. Right now it takes 'sh010', finds its visualParent (sq01), then takes the 'sq01' asset, grabs its visualParent and so on it cascades all the way, stopping at silo because that is excluded (already categorized in 'tabs')

If 'sq01' doesn't exist as an asset in the db, it won't find its further parents. This could, of course, be remedied, by simply storing the full hierarchy in the asset data as a list of parents.

[BigRoy]

Shouldn't be any technical reason for visualParent to require the parent it's pointing at to exist, but it'd make sense for it to do so. I think this is merely how the GUI is implemented, and doesn't put constraints on the system itself. @BigRoy?

Actually. It must exist. The visualParent is an id and without it being present it would be impossible to store a name of the parent. So it would need to exist to be possible to know something about it.

Making silo into an asset is also an option; only it might not make sense to start working on one, but that could be an organisational limit rather than a technical one.

Theoretically it could work, but I think this is the exact distinction we made at the beginning. The "silo" is never worked on, it's just the parent holder. The "ep101" instead, could have tasks.

So I got it working quite well now. I'll clean it up a bit a push so we can discuss the approach. I've just added one hierarchy key to the template which I build on the fly before formatting from hierarchy data member on the asset that holds a list of all the parents.

Actually, what does this do? This hierarchy data is just another key added to the path template? Yet the assets aren't actually "parented" to each other? Or how must I see this?

[mkolar]

Theoretically it could work, but I think this is the exact distinction we made at the beginning. The "silo" is never worked on, it's just the parent holder. The "ep101" instead, could have tasks.

Aha. didn't know about that decision, that clears it up somewhat. I personally don't mind that, but it should be documented somewhere eventually.

Actually, what does this do? This hierarchy data is just another key added to the path template? Yet the assets aren't actually "parented" to each other? Or how must I see this?

I'm doing this now:

templates

[template]
work = "{root}/{project}/{hierarchy}/{asset}/work/{task}/{app}"
publish = "{root}/{project}/{hierarchy}/{asset}/publish/{subset}/v{version:0>3}/{subset}.{representation}"

code that formats the path

# prepare hierarchy data
hierarchy = asset['silo']
for parent in asset['data']['hierarchy']:
    hierarchy = os.path.join(hierarchy, parent)

template_data = {"root": root,
                         "project": PROJECT,
                         "silo": asset['silo'],  # technically not needed in our path formating
                         "hierarchy": hierarchy,
                         "asset": ASSET,
                         "subset": subset["name"],
                         "version": version["name"]}

template_publish.format(**template_data)

asset['data']['hierarchy'] is a list of all parents (their names) excluding silo

So for forming the path I only need the hierarchy key from the asset data, that is a list of strings, so I don't have to query every parent from the database, but all the parents also exist as assets linked as visualParent to each other

My path formatting uses the hierarchy data from asset['data'], while your loader uses the visualParent.

So there is a disconnect now between what is needed to build a visual hierarchy in the loader (visualParent) and what is needed to build a path (asset['data']['hierarchy']).

[BigRoy]

Isn't visualParent also just metadata? Making silo into an asset is also an option; only it might not make sense to start working on one, but that could be an organisational limit rather than a technical one.

Actually, I've given this some more thought. In my head it makes sense to drop the required default silo and simplify it away from the requirements of avalon core.

The only thing I'm currently not sure about is how we can transition away from it, without breaking backwards compatibility. Our projects use silo, that's fine. But somehow we should be able to "work" on them in the future as tools allow the changes of removing "silo" as a thing of its own.

In practice it can really just be a parent asset of anything else. I'd be good to really get to the ground of this discussion on why or why not to ditch it - maybe an issue of its own. But last time tinkering about this I started feeling more and more confident that removing silo in the required way that it is will actually simplify things without losing too much.

[mottosso]

Yeah, agreed. The direction I see this going is for silo to become one of many tags, I think similarly to what @mkolar had in mind.

{
  "name": "sh2010",
  "tags": ["seq03", "ep12"],
  "icon": "boat"
}

And then used in a path template (or not).

{root}/{project}/{tags}/{asset}/work/{task}/{user}/{app}

Where maybe tags would be parsed prior to being passed to the path.format() function..

tags = "/".join(tags)

# Or rather..
tags = os.sep.join(tags)

This way you get both the reusable metadata per asset, that you could use to sort, search or filter with - like finding all shots for seq03 and/or ep12 - along with their simplest use to construct an actual path.

For backwards compatibility, we could do what we did for Pyblish, where family started as a single value but became families, a list. On startup, ask for silo and put it in tags, then use tags exclusively. silo could then remain indefinitely without affecting the remainder of the system.

Bottom line is that this should enable an arbitrary hierarchy depth, whilst preserving the simplicity of the current object model, where there only are Project, Asset, Subset, Version and Representation to worry about for pipeline tools. The rest being pure cosmetics.

Thoughts?

[BigRoy]

Don't think tags is the best name for it, but yes. It definitely sounds about a good way going forward.

Aside from that, the only remaining issue is the asset's name - it must be unique. So sh2010 can't be present twice. How did you think of tackling that? We're currently prefixing the name with the hierarchy and just using the label for the short name. Hmm...

[mottosso]

Aside from that, the only remaining issue is the asset's name - it must be unique. So sh2010 can't be present twice.

This is somewhat of an independent issue to this one about hierarchy, but in short you can do this (already, locally, and independent of Avalon): With Mongo, you can make this assertion at a low-level, via the Unique Index functionality. Currently, assets are stored in what is effectively a list, their names are stored as members of a dictionary. But you can make that name forcibly unique by making it a "unique index". That way, Mongo should prevent two names from ever being the same by throwing an error.

[BigRoy]

Ah sure, my question was not regarding forcing uniqueness and ensuring an artist does not create the same asset name twice. It's just that the artist might make sh2010 in episode one, and do the same in episode two. The second cannot be named sh2010 because it's the same as the other.

So:

s01e01/sh010
s01e02/sh010

The shot names clash and are not allowed - they are not unique, even though they are unique in their hierarchy. Or am I missing your point?

[mottosso]

Ah I see what you mean. I would use tags only as a cosmetic addition and pick a name that is unique on its own. For example, one of those can be called `s01e01sh010` and the other `s01e02sh010` and then give them tags to help with visual sorting and directory layout, and maybe a label to reduce visual clutter in loader and launcher, like `sh010` for one of them. In practice, I would probably establish somewhat of a convention on names that are a little easier on the eyes and to remember. Like `kr0250` or `pr1500`, where the initial two characters are the name of a sequence such like "Kongo Returns" and "Prologue". The first two numbers could be the episode or what not and the latter a shot. That's the convention used at Framestore per film and I still remember several letter number combinations. :) Point is, silo is just cosmetics and so is tags. They can help you visually organise things but shouldn't affect pipeline functionality. Sticking to that should enable an arbitrary directory depth for example, whilst still being compatible with tools built for projects with a shallow layout.

…

On Sat, 24 Mar 2018 16:28 Roy Nieterau, ***@***.***> wrote: Ah sure, my question was not regarding forcing uniqueness and ensuring an artist does not create the same asset name twice. It's just that the artist might make sh2010 in episode one, and do the same in episode two. The second cannot be named sh2010 because it's the same as the other. So: s01e01/sh010 s01e02/sh010 The shot names clash and are not allowed - they are not unique, even though they are unique in their hierarchy. Or am I missing your point? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#308 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACDZPqocfgC35VMvnavC8jUO58gURm5qks5thnQ7gaJpZM4RvEhl> .

[mkolar]

I applaud where this is heading!

I agree that tags is probably not the best name for it. I'd go with something that clearly states what it's for. hierarchy, parents, ancestors or my favorite: breadcrumbs

In terms of asset names. I'd restrain from putting any restriction on these apart from having to be unique. The system simply shouldn't allow you to make a second sh010 shot in the project. How the studio deals with it then, is purely up to them. 101sh010 s01ep01sh01 ep101sh010 are all ok if a studio decides so.

[BigRoy]

@mkolar did you ever get around to taking another look at this?

[mkolar]

Long time sleeper topic :).

We've gotten around to this recently and did a test implementation in our fork of avalon-core.

We found that silo was actually present as a requirement in many places so removing it isn't trivial at all. Instead as a intermediate step we opted for adding option to have silo exists in the db as any other asset as we needed to allow tasks to be created on it. This asset is identical to all others, with the only difference being that it's silo data is set to None. We then simply find all assets with this silo == none and treat them as silos the gui (meaning they become the tabs on top)

For the hierarchy we're using exactly what was proposed here however we've named the data member parents which we felt represents what it is the best. To make this bulletproof, parents list should probably become a required property rather than a member of data.

here are the PRs in our fork to have a look at. removing of silo is somewhat dependent on allow hierarchy in the system.

https://github.com/pypeclub/avalon-core/pull/12/files
https://github.com/pypeclub/avalon-core/pull/4/files

[mkolar]

Closing this as it's been implemented, quite some time ago :)