Progressive Token Variant Specificity

[CITguy]

In build systems like Flutter (likely Android, as well), convention for variants is to add specificity when a variant differs from the default. If foo is the default, foo-dark is its dark variant.

I'm finding it difficult to adhere to this naming convention, given the current specifications.

To illustrate, consider the following example:

Desired Tokens

Say you need to define the following SCSS vars from your design tokens:

1. $color-background
2. $color-background-highContrast (high-contrast variant of $background)
3. $color-background-dark (dark variant of $background)
4. $color-background-dark-highContrast (high-contrast variant of $background-dark)

Problem

1. Unless I'm misinterpreting the spec, there is no way to define the desired tokens.
   • It would require a JSON property to be a group name, token name, or both.
   • Token discovery normally stops traversing the JSON hierarchy once a $value property is found, because the current spec implies that further processing is unnecessary.
2. Given the current definition of a Token in the spec, you cannot nest tokens within tokens in the JSON hierarchy.
   • Thus, there's no way to define a token that matches the subpath of another token (i.e., I cannot define token foo.bar when token foo.bar.fizz exists, because foo.bar needs to be defined as a group for foo.bar.fizz).
3. The spec is unclear if the JSON property name indicates token presence (top-down) or if the $value property indicates token presence (bottom-up). Because of this confusion, the spec is unclear if token identification should continue after the first occurrence of $value, assuming a top-down discovery strategy.

[CITguy]

Idea 1: All JSON properties are group names

1. All JSON properties are considered group names.
2. The presence of a $value property is the critical condition for identifying token presence.
   • Token identification is based on occurrences of $value properties in the JSON hierarchy.
   • Token identification should include ALL occurrences of $value, not just the first in a path (when resolving top-down).
   • A tokens.json file will define N tokens, where N is the count of $value properties in the file.
     • A tokens file with 4 $value properties will define 4 tokens.
   • Token paths are calculated from the path to a specific $value property.
     • #/foo/bar/$value resolves to a token path of foo.bar

Example:

{
  // GROUP: #/color
  "color": {
    "$description": "All the colors",
    "$type": "color",

    // GROUP: #/color/background
    "background": {
      "$value": "#eaeaea", // TOKEN: color.background

      // GROUP: #/color/background/highContrast
      "highContrast": {
        "$value": "#ffffff" // TOKEN: background.highContrast
      },

      // GROUP: #/color/background/dark
      "dark": {
        "$value": "#555555", // TOKEN: color.background.dark

        // GROUP: #/color/background/dark/highContrast
        "highContrast": {
          "$value": "#000000" // TOKEN: color.background.dark.highContrast
        }
      },
    }
  }
}

which should resolve to the following, flattened token hierarchy:

{
  "color.background": {
    "$type": "color",
    "$value": "#eaeaea"
  },
  "color.background.highContrast": {
    "$type": "color",
    "$value": "#ffffff"
  },
  "color.background.dark": {
    "$type": "color",
    "$value": "#555555"
  },
  "color.background.dark.highContrast": {
    "$type": "color",
    "$value": "#000000"
  }
}

Challenges

1. Extraneous properties cannot be easily differentiated from group identifiers.
   • For efficiency, you'd only want to traverse deeper if you know you need to.
   • Requires traversing EVERY property to identify tokens at all levels.
2. Given just a token path (e.g., foo.bar) there's no way to tell if the path refers to a group or a token.
   • This will only get worse if the spec switches to use JSON pointers for aliases (unless the JSON pointer alias points directly to a $value property).

[CITguy]

Idea 2: group $token prop

1. A group is any JSON object that DOES NOT contain a $value property.
   • same/similar to current spec
2. A token is any JSON object that CONTAINS a $value property.
   • same/similar to current spec
3. An explicit token is one defined using the $token property, all other tokens are considered implicit.
   • All tokens defined by the current spec are considerd implicit.
   • Explicit tokens are necessary if a JSON path cannot reliably identify a token vs a group.
4. The path of an implicit token is defined by the parent path of the $value property.
5. The path of an explicit token is defined by the parent path of the $token property.
6. All tokens inherit shared properties defined by their ancestor groups (e.g., $type, $extensions, etc.)

Example:

{
  "color": {
    "$description": "All the colors",
    "$type": "color",

    "background": {
      "$token": {
        "$value": "#eaeaea"
      },

      "highContrast": {
        "$value": "#ffffff"
      },

      "dark": {
        "$token": {
          "$value": "#555555"
        },

        "highContrast": {
          "$value": "#000000"
        }
      },
    }
  }
}

should resolve to the following, flattened token hierarchy:

{
  "color.background": {
    "$type": "color",
    "$value": "#eaeaea"
  },
  "color.background.highContrast": {
    "$type": "color",
    "$value": "#ffffff"
  },
  "color.background.dark": {
    "$type": "color",
    "$value": "#555555"
  },
  "color.background.dark.highContrast": {
    "$type": "color",
    "$value": "#000000"
  }
}

Walkthrough

Task: Define the following tokens

• foo.bar = '#abcabc'
• foo.bar.fizz = '#defdef'

Step 1: foo.bar.fizz

{
  "foo": {
    "bar": {
      "fizz": {
        "$type": "color",
        "$value": "#defdef"
      }
    }
  }
}

Step 1 Progress

• foo.bar = '#abcabc'
• foo.bar.fizz = '#defdef'

Step 2: foo.bar

Now let's define foo.bar...

{
  "foo": {
    "bar": {
      "$type": "color",
      "$value": "#abcabc",
      
      "fizz": {
        "$type": "color",
        "$value": "#defdef"
      }
    }
  }
}

Step 2 Progress

• foo.bar = '#abcabc'
• foo.bar.fizz = '#defdef'

Step 3: Correct definitions

Uh oh! By defining #/foo/bar/$value in step 2, the token discovery algorithm stopped looking for tokens once foo.bar was found.

Let's use the $token group property to fix this issue.

{
  "foo": {
    "bar": {
      "$token": {
        "$type": "color",
        "$value": "#abcabc"
      },
      
      "fizz": {
        "$type": "color",
        "$value": "#defdef"
      }
    }
  }
}

Step 3 Progress

• foo.bar = '#abcabc'
• foo.bar.fizz = '#defdef'

Step 4: Optimize JSON

We were able to define both tokens, but we can simplify the JSON structure by having both tokens inherit $type from the #/foo/bar group.

{
  "foo": {
    "bar": {
      "$type": "color",
      
      "$token": {
        "$value": "#abcabc"
      },
      
      "fizz": {
        "$value": "#defdef"
      }
    }
  }
}

[CITguy]

Relates to #97

[c1rrus]

Yup. Your 2nd idea, the $token property for groups, is essentially the same as the $default one I proposed in this comment on #97. :-)

Out of the two ideas in your OP, I would prefer that approach for the same reasons you've identified: The "special" group $token can have its own token-specific properties independently of its parent group. For example, the group and its $token could each have their own description, extensions, etc.

I don't have a strong preference what this "group-level" token should be called though - $token, $default or something else.

[CITguy]

After looking at my proposals, they both feel like a hack to work around the strict syntax imposed by JSON itself.

As-is, a JSON property is overloaded, because it can implicitly be any of the following...

• group name
• token name
• property name

Yet, to accomplish what I'm asking for, the syntax needs to be explicit. Which requires doing one of two things within the limitations of unique JSON properties.

1. Defining special, reserved properties for a "group" object to explicitly define a token at the same path as the group (as proposed above)
2. Defining special syntax for naming properties/keys, such that you can define sibling properties with the same name, but get around the unique key restriction. For example...

   {
       "color": { // implicit Group
           "token:background": { /* explicit Token (color.background) */ },
           "group:background": { // explicit Group
               "highContrast": { $value: "...", ... }, // implicit Token (color.background.highContrast)
               "token:dark": { /* explicit Token (color.background.dark) */ },
               "group:dark": { // explicit Group
                   "highContrast": { $value: "...", ... } // Implicit Token (color.background.dark.highContrast)
               }
           }
       }
   }

Unfortunately, neither solution is very intuitive, nor does it feel like the right solution.

---

Side note... It's because of this that I'm starting to wonder if JSON will be able to keep up with the evolving requirements of the spec. I've been looking into seeing if there are any existing, alternative file formats that would provide the needed flexibility. Unfortunately, I've not found anything that could be easily adapted, so I'm experimenting with a custom syntax, along with a CLI tool to translate this syntax into universal JSON for input into translation tools. I'll share progress as I can.

[CITguy]

I just thought of a solution that would require minimal changes.

We know that the problem with progressive token definition is being able to define both a group and token name at the same level in JSON. It doesn't make sense to change the name of groups, because the JSON syntax naturally implies grouping. Instead, it would make more sense to explicitly differentiate a token name using some sort of naming convention.

Given that we use dot-path syntax for token alias reference, what if we allow a . prefix for JSON props corresponding to token definitions?

• The . prefix is entirely optional.
  • This invalidates current spec for § 5.1.1 "Character restrictions"
• Token definitions using the current syntax are still valid.
• Downstream tool maintainers would need to add support for the optional .<token> syntax.
  • However, it's much simpler to implement than supporting an entirely new syntax for both tokens and groups.
  • I'd argue that this might be a reason to need to explicitly specify the schema version in token JSON files, so that tool authors can adjust internal logic based on supported schemas.

Example

{
    "color": {
        // opt-in syntax, explicit Token (color.background)
        ".background": { $value, $type, ... },
        "background": {
            // current syntax, implicit Token (color.background.highContrast)
            "highContrast": { $value, $type, ... }, 

            // opt-in syntax, explicit Token (color.background.dark)
            ".dark": { $value, $type, ... },
            "dark": {
                 // current syntax, implicit Token (color.background.dark.highContrast)
                "highContrast": { $value, $type, ... }
            }
        }
    }
}

The above would then parse out 4 tokens...

color.background
color.background.highContrast
color.background.dark
color.background.dark.highContrast

[jgornick]

Just checking in to see if something like this makes sense to implement? Could it be implemented with a custom preprocessor?

In this scenario, one could just create a "default" variant and all would be well. But what happens when you have a "default" color name that has different variants? Something like color.background.default.default doesn't read to well.

[CITguy]

The root of the problem I'm trying to solve is tokens that have multiple, contextual values (a.k.a., "modes"). Currently, the only means of differentiating modes is via naming, which doesn't provide a lot of flexibility. I'm not sure my previous idea will suffice, though. I'm still thinking of potential workarounds, because we're going to need runtime, contextual "modes" in our next major release of our tokens.