Support multiple actions, NLA tracks

[donmccurdy]

Fixes #150, #118, #112, #39.

Tasks:

• Export multiple animations based on actions used in NLA tracks
• Support morph targets / shape keys.
• Support "bake skinning" option.

Attempting to implement support for multiple actions. There seems to be no good way to detect which actions could actually affect an object, so instead I'm using NLA tracks for this purpose. This means you must select which actions to export, otherwise only the active action is used.

In the NLA editor, you must Push Down each action that should be exported. Using Stash might also work, I didn't test that. In order for the action to be listed, you must open the Dope Sheet and make that action active, first.

Once you've done that for each action, they should be listed regardless of whether any are active:

/cc @sketchpunk @motorsep @cdata @nicocornelis if you are able to test this, I'd appreciate it. 🙂

Updated exporter:
io_scene_gltf2.zip

@cdata this is similar to your approach from #111, I think, but I've (attempted to) export 1 glTF animation for 1 Blender action, rather than separate for each object. Not sure whether that is the right approach. There is also some extra work to handle caching now.

[donmccurdy]

Definitely some bugs to iron out, but e.g. this character is animating correctly:
Archive.zip

[motorsep]

Nice! Why not to just export all actions (each as an individual anim) and let user delete unnecessary ones upon importing glTF into UE4, for example? That's how it works with FBX right now.

[donmccurdy]

@motorsep Yeah, I expect we’ll need an export option or two in the end, e.g. if two characters share a “walk” action, are they intended to be played separately or together? Depends on context... But hopefully we can get something basic working first and unblock most workflows.

[motorsep]

@donmccurdy As a game dev I usually export one character at a time. Selected mesh + respective armature (or selected armature). Actions should be unique to each armature.

It would be also a good idea to add prefix to exported anims. For example, you mesh (object) is called Char1. So you could export its Actions with prefix Char1_*. For example "walk" action as Char1_walk. I believe FBX exporter does that (or maybe UE4 does that on import, not sure).

[donmccurdy]

Makes sense. There’s not going to be a one size fits all solution here, since others may want a single “animation” that targets multiple objects in sync, so we’ll have to allow some options for sure. I think the default should probably not be N actions * M meshes = N * M animations, because that has led to some misunderstandings and bad loader implementations before. I’d like to have defaults that work reasonably well for runtime use, since users might not be editing this in an engine at all, and then add in more advanced options stuff from there.

Should also note, this PR uses NLA tracks only as indications of which actions are active. NLA tracks composing multiple actions with transitions will need to be baked to a single strip and action to export correctly.

[motorsep]

What if animator doesn't use NLA at all? (I never do and I have no clue how I'd use it for the purpose of just exporting to glTF)

Also wondering why not to make a copy of the scene on export, bake everything, export and then discard temporary copy. That would allow animator to keep actual scene unbaked, providing nondestructive workflow (I believe we had that in Doom 3 md5 exporter)

[donmccurdy]

The main thing to do first is just getting this tested enough to find errors in the code, you only have to press “Push Down” on each action you want included with the character. We can figure out which options to add and perhaps do NLA track baking once something basic is in place.

[donmccurdy]

Added a ZIP with the updated exporter in the OP.

[emackey]

I tried adding a location change (2 animation keyframes) to the default Blender cube, clicking "push down" to make an NLA track, and got an error on export. Did I miss a step?

  File "blender-2.79-windows64\2.79\scripts\addons\io_scene_gltf2\gltf2_animate.py", line 204, in animate_location
    joint_cache = export_settings['gltf_joint_cache'][action_name]
KeyError: 'CubeAction'

[donmccurdy]

@emackey Thanks — should be fixed now.

[inkthorne]

This is awesome, @donmccurdy! I had started looking into fixing this, too, after my comments on #39. I grabbed your .zip and tried it out with a simple scene. I know you're in just "getting it to work" mode, so some of these comments might be for the future:

Don't hate me, but I think you're going to want to keep the existing exporter behavior for animations[0] (i.e. if objects have an assigned action, those actions get exported into a single animation). It's ideal for things like Sketchfab, Facebook, and game cinematics where you want to play a single animation and have the whole scene animate. The separate animations would then only be exported from the NLAs of an object.

I think the default should probably not be N actions * M meshes = N * M animations,

I completely agree, however, I think that only should apply at the accessors[] level. If the same action is assigned as an NLA to 2 or more objects, each object should have it's own animations[] entry with a duplicated samplers[] entry which references the same accessors[] entry. Currently, I'm getting all objects that have a 'Slide' NLA grouped into a single 'Slide' animation, so animating them separately would be difficult.

It would be also a good idea to add prefix to exported anims. For example, you mesh (object) is called Char1. So you could export its Actions with prefix Char1_*. For example "walk" action as Char1_walk.

It's convenient for the exporter to output uniquely named animations, but there are some authoring drawbacks to this approach. Example, the door of a house is parented to a hinge. It's the hinge that needs the animation, but "hinge_open" isn't an ideal animation name compared to "door_open". My preference would be to export the animations with just the name of the action OR use the name of the NLA track if it has been changed from the default "NlaTrack"*. This way folks can specify what they want the animation called and if they want it to be uniquely named.

[donmccurdy]

For comparison, here are the settings exposed by the "Better COLLADA" exporter and FBX exporter:

COLLADA

• All actions (writes all actions for 1st armature found, to separate DAE files)
• Skip (-noexp) actions (skip control actions)

FBX

• NLA strips (export each non-muted NLA strip as AnimStack, if any, instead of global animation
• Export each action as separated AnimStack, instead of global scene animation (some objects will get all actions compatible with them, others will get none at all)

These aren't perfect parallels, because both are exchange formats where it's pretty much assumed you're doing more processing in e.g. UE4, whereas glTF can be used for runtime, but it's a start. And of course there are options for baking keyframes but that's out of scope for this PR.

---

From various comments in this thread:

If objects have an assigned action, those actions get exported into a single animation

Why not to just export all actions (each as an individual anim) and let user delete unnecessary ones upon importing glTF into UE4, for example?

My preference would be to export the animations with just the name of the action OR use the name of the NLA track if it has been changed from the default "NlaTrack"*.

...let me try to propose a couple of export scenarios / settings:

1. No NLA tracks are used, multiple actions are present. A single global animation is written, with each object moving according to its active action, if any. Good for cinematic case, e.g. Facebook.
2. No NLA tracks are used, multiple actions are present. Each action is exported as a separate animation for each object to which it could apply, e.g. N * M animations for the first Armature selected. Good for import into an engine where further editing happens, e.g. UE4.
3. NLA tracks are used. Each NLA track is exported as an animation, named according to the track name (if non-default) or <object_name>_<action_name> as fallback. If multiple NLA tracks share the same name, they are merged into a single glTF animation track. Good for runtime use, e.g. three.js or A-Frame, where we want to play different clips dynamically, without exporting bloated files.

Assuming those three cases are available by some combinations of settings and defaults, am I missing anything necessary?

[inkthorne]

For the use cases I'm aware of, those cover it.

If multiple NLA tracks share the same name, they are merged into a single glTF animation track.

If that can be used to put an attacker's 'punch' action into the same animation as the defender's 'dodge' action, that is amazingly cool! :)

FWIW, I'm not a huge fan of 2. It doesn't add export capability like 1 or 3. It may offer a slightly nicer workflow in certain situations, but it also complicates the exporter (i.e. how robust does determining if an armature is compatible with an action need to be?). But, I might be overlooking something.

I like that 1 & 3 are compatible with each other, so no option toggle would be needed, things would 'just work'. I'm definitely interested in hearing others' opinions on this.

[inkthorne]

As a side-note:

Using Stash has some advantages over using Push Down. Both will create an NLA track with the given action, but Stash creates a locked & muted NLA track, so it won't affect your object's current animation by trying to combine with the current action or the other NLAs. It's also nice because, by definition, a stashed action is an unmodified, single-action NLA (which is the only kind of NLA that can be exported, at least to begin with).

[inkthorne]

I'm sorry, @donmccurdy, for duplicating your effort. I had already started on my own changes before seeing your tweet about this, and I just continued with it. If there's anything in this version of the exporter you'd like to take, please do.

The changes accommodate 1 & 3 from above:

• All the currently assigned actions of objects are combined into a single unnamed animation at animations[0]. This is how the exporter previously worked, so there's no change in behavior here.
• An object's NLA tracks are exported as additional named animations. If the NLA track has been renamed from the default, the animation name is the NLA track name. If the NLA track has not been renamed from the default, the animation name is <object_name>_<action_name> in order to be unique.
• Animations that have the same name are combined into a single animation with that name.

I've run some simple test cases for skinned & articulated scenes, it also appears to successfully export the TwoAnimCharacter model as well. If you have time, I would be interested in hearing if it's behaving as you'd expect.

NLA exporter: nla_export.zip

[motorsep]

@inkthorne I am just trying to understand the thinking behind bunching all actions into one animation. The whole purpose to have multiple actions is that one can mix and match those in any order, whether it's in Blender's NLA Editor or in UE4's anim system / Sequencer or in whatever Unity has. How'd you know where individual anims begin and end in glTF once they are bunched into one anim at animations[0] ?

[inkthorne]

@motorsep, the way the exporter currently works (exporting all currently assigned actions into a single animation) is ideal for full-scene (cinematic) animations. This scenario is useful when you want to display an animated 3D scene in things like Sketchfab, Facebook, or even game cinematics. In these cases, you want to load up the scene and then just play a single animation that animates everything. In that case, every object is assigned an action that animates over the entire timeline period.

However, with the changes, the single-action NLAs are exported as separate animations (i.e. run, walk, jump, attack, etc). You get a separate animation for each NLA (i.e. run @ animations[1], walk @ animations[2], etc). This is the scenario that us gamedevs are mainly interested in, because the animations are played & blended dynamically based on user input.

Did that answer your question? I think we might be tripping up over the Blender terminology.

[motorsep]

@inkthorne yep, makes sense. I am not sure why you refer to NLAs, since there is no point to even bother with NLA when working with individual Actions. I've animated and exported to numerous engines, starting with Quake 1 and ending with UE4 and I never even opened NLA Editor. Export add-ons would also grab Actions directly, never even touching NLA tracks.

[donmccurdy]

Thanks @inkthorne! More approaches to this are certainly welcome. :) From this snippet of your code...

process_object_nlas()

    def process_object_nlas(blender_object, nla_animations):
        """
        Iterate through all of the NLA tracks and make animations out of their actions.
        Append the newly created animations to 'nla_animations'.
        """
        if blender_object.animation_data is None:
            return

        nla_tracks = blender_object.animation_data.nla_tracks

        if nla_tracks is None:
            return

        for nla_track in nla_tracks:
            nla_channels = []
            strips = nla_track.strips

            if strips is not None and len(strips) == 1:
                strip = strips[0]
                blender_action = strip.action
                process_object_animation(blender_object, blender_action, nla_channels, samplers)

                if len(nla_channels) > 0:
                    # Use the NLA track name as the animation name, if it's been changed from the default.
                    if not nla_track.name.startswith("[Action Stash]") and not nla_track.name.startswith("NlaTrack"):
                        nla_animation_name = nla_track.name
                    # Otherwise, create a unique name for the animation from the object & action strip.
                    else:
                        nla_animation_name = blender_object.name + "_" + strip.name

                    # Attempt to find an animation with the same name.
                    nla_animation = None
                    for animation in nla_animations:
                        if animation['name'] == nla_animation_name:
                            nla_animation = animation
                            break

                    # Combine animations with the same name.
                    if nla_animation is not None:
                        nla_animation['channels'] = nla_animation['channels'] + nla_channels
                    # Otherwise, create a new animation.
                    else:
                        nla_animation = {
                            'channels': nla_channels,
                            'name': nla_animation_name,
                        }
                        nla_animations.append(nla_animation)

... it looks like what would happen is that if there are multiple actions sequentially in an NLA track, they'll all be written to the same glTF animation, but stacked on top of one another and starting at the same time. Am I misunderstanding, or do you know if there's a way around that? I think we can "bake" the NLA to an action and then export that, but it would be nice to avoid re-sampling if it's possible...

About single-action NLAs, it sounds like there's no reason to think of that as a special case. If someone wants a single-action clip they can stash it, or mark it active, or use an All actions option if/when we add that. And if they did make a single-action NLA it should "just work" anyway, as long as we handle complex NLAs right. So I'm not too worried about the distinction right now.

[inkthorne]

... it looks like what would happen is that if there are multiple actions sequentially in an NLA track, they'll all be written to the same glTF animation, but stacked on top of one another and starting at the same time.

@donmccurdy, it shouldn't. The code only exports an NLA track if there's only a single strip and then only exports the strips[0].action as the animation. It seemed like a reasonable limitation for 2 reasons:

1.) Blender NLA tracks are specific to the object they're assigned to and can't be reassigned or shared with other objects like Blender actions. While it's completely possible you might want to author a complex NLA that only that object uses, if you're depending on the exporter for the baking, you're now limited to authoring that animation using a single NLA track.

2.) Complex animations are often put together not just by using multiple strips in a single NLA track, but by also combining multiple NLA tracks. In Blender, you can then bake these into a single action which can then be shared and used to animate other objects. This action can then be assigned as a stashed action (single-action NLA) for export.

That said, there may not be anything wrong with the exporter baking out a multi-strip NLA track as an animation (as long as it doesn't bake single-strip NLAs and make them unnecessarily larger). But, given 2 above, it wasn't needed to get things working. :)

[inkthorne]

Export add-ons would also grab Actions directly, never even touching NLA tracks.

@motorsep, you're right; however, glTF doesn't currently support soft targets for animations. Each animation channel in glTF must be targeted to a specific node or bone by a (hard) index rather than a (soft) name. This means that in order for an action to be exported, it has to be specifically assigned to an object.

In a Blender file where you have a character and 3 actions: walk, run, & jump; only 1 of these actions can be assigned to your character at a time. In order to associate multiple actions with the character, the NLA tracks are used. Another way to deal with this would be to export all actions as if they were assigned to the first (or all) valid object(s). That works for simple scenes (e.g. a single character), but not for complex ones.

In either case, these strategies are just ways to give an exported glTF animation hard target(s). IMO, if an animation.channel.target could be a name OR an index, none of these work-arounds would be necessary. But, I realize changing the spec is a much bigger deal than changing the exporter. :)

[motorsep]

@inkthorne I still don't get the issue. Select Armature, select mesh(es) and export. Export one Action for selected Armature at a time. Like I mentioned in the past, none of the exporters I used had any issues with multiple Actions. For example, you can see what Doom 3 skel/anim exporter does: https://github.com/motorsep/blender-idtech4-md5 (I didn't code that beast, only did some tweaks)

Also worth mentioning that if you have multiple Armatures in your scene, whether it's a cinematic anim or anims for AI-driven characters, you still want to export one Armature at a time (per glTF scene) and then assemble multiple glTF scenes/files into one scene in the game engine of your choice.

Since glTF was never meant to be an exchange format as FBX or COLLADA, I don't see why one would want to have multiple armatures inside glTF scene.

[donmccurdy]

The code only exports an NLA track if there's only a single strip and then only exports the strips[0].action as the animation.

@inkthorne Ah that makes sense, thanks!

Export one Action for selected Armature at a time ... export one Armature at a time ... and then assemble multiple glTF scenes/files into one scene in the game engine of your choice.

@motorsep For my use case, the glTF file is for last-mile transmission to a WebGL application running in a user's browser. There is no easy way to assemble different actions into a single animation later, and having any unused actions attached to an Armature means unnecessary file size for end-users. The expectation of exporting one armature at a time is (usually) fine, the entire scene does not need to be in one file.

There's no need to pick and choose workflows, I think we can support what everyone is describing here... just trying to clarify some of the use cases. I still think options (1), (2), and (3) from #166 (comment) would accommodate this, but let me know if I've missed something.

[pjcozzi]

@donmccurdy anything you need here to bump this along?

[NoxWings]

Just to reanimate the discussion I'd like to give some feedback. I like the 3 options proposed by @donmccurdy.

About the 2nd option:
This is how I would have expected the exporter to work by default since my workflow also involves a single character per blend file.

About the 3rd option:
BUT after learning about the difference of how blender doesn't really link actions with armatures and how the gltf spec needs a "hard" link, I really like the idea about exporting non-muted NLA tracks. It just looks like THE CORRECT CHOICE now.

But the first time I read it I just thought "why would I have to tinker with the NLA editor (3)? I just want all my actions exported (2)". So I consider having option (2) too would add value to the proposition too.

[emackey]

Thanks great feedback @karroffel, thanks!

[donmccurdy]

@IARI does your example fail both with and without the version of the exporter in this PR?

@karroffel this is a fine place to post, but because this PR is taking a while to finish, I think it would be better to start a new issue. Thanks!

[donmccurdy]

This PR is ready to merge, if there are no regressions I've missed here. For now the change to current behavior is simple enough: stashed actions and single-strip NLA tracks are written to their own animation clips.

I thought there was a regression with joint matrices, but it's on master as well (#253). A few other good points have come up in this thread that are not addressed, but I don't think they need to block this PR. Please open new issues for any further improvements we can make here.

[IARI]

@donmccurdy you earlier asked "does your example fail both with and without the version of the exporter in this PR?" - sorry for replying thsi late:
in short: I think the problem is not specific to this PR.

long: I'm not 100% sure, but think this is related to Blender not being able to apply modifiers on objects that have shape keys in general. I think theres no trivial solution to this anyways.

[emackey]

@donmccurdy I'm seeing a validation error:

Animation channel has the same target as channel 1.

Test file: ParentingTests_animated.zip

            "channels" : [
                {
                    "sampler" : 0,
                    "target" : {
                        "node" : 2,
                        "path" : "rotation"
                    }
                },
                {
                    "sampler" : 0,
                    "target" : {
                        "node" : 2,
                        "path" : "rotation"
                    }
                }
            ],

There's another error here too, unrelated to this branch, in that parenting some objects and exporting with "Selected objects only" causes some meshes & nodes to be exported multiple times, resulting in This object may be unused messages. I haven't nailed that one down yet so I don't know if that's related to the multiple animation targets error. The animation targets error does seem to be local to this branch though.

[emackey]

@donmccurdy I found the root cause. Don't waste any of your cycles on the problem I posted, it's not local to this branch and I will have a fix that's completely separate from this.

[emackey]

Awesome work here @donmccurdy. Would be great to have options for combining a particular animation across multiple nodes/objects, but that can be a future PR.

[IARI]

Wohoo! :D thanks for merging this, I now am one happy developer ;)