Tokens group $extends property proposal

[jorenbroekema]

Tokens group $extends property proposal

I would like to propose adding a special $extends property on token groups to signify that the group extends from another group.
This proposal is mostly focusing on components being tokenized rather than core/base or semantic tokens, although perhaps you can use this for such tokens as well.

For example, an input-amount component may conceptually extend the input component, with 95% overlap and only a few design decisions that differ.

Instead of duplicating 95% of the tokens, it's more likely that token authors will only tokenize the parts of input-amount that differ from the input. For more context, I wrote a bit about token explicitness here

Throughout this proposal I'm using an example with 2 components. An input component with 2 tokens: field width and field background. An input-amount component that reuses the regular input's field background, but changes the field width, so it has only 1 token but reuses another.

Base input.tokens.json:

{
  "input": {
    "field": {
      "width": { 
        "$value": "100%"
      },
      "background": { 
        "$value": "#FFFFFF"
      }
    }
  }
}

Extension input-amount.tokens.json:

{
  "input-amount": {
    "$extends": "{input}",
    "field": {
      "width": { "value": "100px" }
    }
  }
}

The $extends prop signifies that the input-amount token group extends from the input token group.

Use cases

I'll try to explain some use cases here to show why this information could potentially be valuable for design token tools. Feel free to add your own!

Token analysis

Let's say we want to analyze our tokens and create a big diagram to show relationships between tokens. When one component's tokens conceptually extends another's tokens, this information is relevant to such diagrams, otherwise you won't get the full picture and it will look like the input-amount only has 1 token.

Discoverability

When token authors are changing, removing or adding token values, it helps to be able to see that one token group extends another. Without this information, they may change tokens in the wrong location e.g. they want to change the background-color of all inputs, but by accident they end up doing it for every extension input individually, creating unnecessary duplication of tokens.

Autosuggest/complete

When your input-amount tokens only have a field width property, consumers of these tokens or the output of these tokens might get confused.

Let's imagine for a second that we do CSS in JS, so we exported our tokens to a custom JS format. A developer will now implement this component separately.

import { css } from 'lit';
import { inputAmountTokens as t } from './input-amount-tokens.js';

const inputAmountStyles = css`
  .btn {
    width: ${t.width};
    background-color: ${t.background}; /* ?? :( */
  }
`;

t.background will error as being undefined, because unknowing to the developer, this token is the same for input and input-amount, therefore they should import that from the input-tokens.js. This may not be intuitive, however, with the $extends information we may be able to give an auto-suggestion on hover or at least display in the hover on inputAmountTokens that it inherits from the regular inputTokens.

Explicit tokens output

Usually when there's a lot of overlap between component tokens, I personally tend to say you should be minimal and not duplicate your tokens everywhere, reuse what you can, at least in your tokens source of truth. This makes maintenance easier.

However, I can imagine it might be preferable for some platforms to have very explicit token output. So in our example, instead of only getting:

• input-field-background
• input-field-width
• input-amount-field-width

You would also get input-amount-field-background, even though the value is exactly the same as input-field-background and therefore duplicate.

In the token parsing and formatting process, the $extends property would be used to essentially do something like this:

before:

const tokens = {
  input: {
    field: {
      width: { 
        $value: "100%"
      },
      background: { 
        $value: "#FFFFFF"
      }
    }
  },
  "input-amount": {
    $extends: "{input}",
    field: { // no background prop
      width: { "value": "100px" }
    }
  }
}

after:

const tokens = {
  input: {
    field: {
      width: { 
        $value: "100%"
      },
      background: { 
        $value: "#FFFFFF"
      }
    }
  },
  "input-amount": {
    ...tokens.input,
    field: {
      ...tokens.input.field, // has the background prop
      width: { "value": "100px" }
    }
  }
}

Basically, a deep merge will be done using the $extends, which can chain up recursively e.g. if input extends another component tokens file etc.

In this way you will have the full result of any tokens that you're extending from, explicitly in the output format.

Theming

When theming, most of the design decisions remain the same across themes, and a few ones change based on theme. I'm not exactly sure if there is yet a best practice for managing themes inside design tokens, but I can imagine that if you go with a "parent-child-inheritance" type of syntax, this spec proposal may be applicable for theming. However, my personal preference is combining theme values inside a single token value, but that's just me..:

{
  "$value": {
    "light": "{colors.text.dark}",
    "dark": "{colors.text.light}"
  }
}

Let me know what you guys think about the proposal, whether there is indeed a need, how you would change the API, other use cases, etc.

[c1rrus]

I like this proposal.

@TravisSpomer described something similar that his team were doing in the discussion in #97. That was essentially allowing references to groups as a shorthand for creating aliases to all its nested tokens in another group. However, there was no way to then selectively override or add to the reference group.

I think your $extends proposal can cover that use-case as well. For example:

{
  "original-group": {
    "token-1": {
      "$value": "#123456",
      "$type": "color"
    },
    "token-2": {
      "$value": "2rem",
      "$type": "dimension"
    }
  },

  "copied-group": {
    "$extends": "{original-group}"
    // makes this group behave as though it contained 2 tokens
    // called "token-1" and "token-2" which are aliases of
    // "{original-group.token-1}" and "{original-group.token-2}"
    // respectively
  },
  
  "overriding-group": {
    "$extends": "{original-group}",
    "token-2": {
      "$value": "5rem",
      "$type": "dimension"
    },
    
    // behaves as though there was another token
    // in this group called "token-1", which was a reference
    // to "{original-group.token-1}"
  }
}

There are some details that need to worked out though:

• Presumably $extends can only be used on groups and must be a reference to another group, right? Extending tokens doesn't make sense to me (how would that be different to an alias token?)
• If group B extends group A, should B also inherit properties from A like $description, $type, etc.?

Thoughts?

[TravisSpomer]

However, there was no way to then selectively override or add to the reference group.

The system I've built is basically exactly what's proposed here, including the ability to override and add to, except I use "aliasOf": "original-group" instead of "$extends": "{original-group}". I chose aliasOf because that's the syntax I use for tokens that reference other tokens, and feels like the same concept applied to groups, so it seemed at the time that obviously it should have the same syntax. But now that I know what I know now, and mentioned later in #97, I think it may help a lot to have aliases of groups not be called aliases. A new term "extends" seems like it might make it easier for some people who struggled with the idea of aliasing groups to comprehend, while not making it any more challenging for the others, so I think I like this proposal slightly better than what I had built.

[TravisSpomer]

We'd have to be very explicit about how things work when you extend a group that has subgroups, and then you override the same subgroups.

1. B extends A
2. Both A and B specify a subgroup C
3. A.C has tokens D and E
4. B.C only has token D

Does the token B.C.E exist? That is, does B.C also automatically extend A.C without explicitly specifying that? In Joren's proposal it does, and that's how I'd expect it to work because it seems like it makes the most sense for the component token scenario. But it wouldn't necessarily have to be that way; one could just as easily interpret it to mean that B.C replaces A.C rather than merging it, if B.C doesn't explicitly state that it extends A.C.

[jorenbroekema]

• Presumably $extends can only be used on groups and must be a reference to another group, right? Extending tokens doesn't make sense to me (how would that be different to an alias token?)
• If group B extends group A, should B also inherit properties from A like $description, $type, etc.?

Thoughts?

Yes to both those questions for me. Indeed, alias is the extends equivalent for a single token, or vice versa, extends is the alias equivalent for a token group. And yes, I think it makes sense to inherit the other properties, saves a lot of duplicate work I think.

@TravisSpomer with regards to your extension question, I would say we should follow something along the lines of what deepmerge does, which if I recall correctly is something along the lines of this:

const obj1 = {
  group: {
    nested: { 
      nestedValA: 'foo',
      nestedValB: 'qux',
    },
    val: 'bar',
  },
};


const obj2 = {
  group: {
    nested: { 
      nestedValB: 'something',
      nestedValC: 'else',
    },
  },
};

// obj2 extends obj1 or aka obj2 deepmerges with obj1
// so it becomes
const obj3 = {
  ...(obj1 || {}),
  ...(obj2 || {}),
  group: {
    ...(obj1.group || {}),
    ...(obj2.group || {}),
    nested: {
      ...(obj1.group.nested || {}),
      ...(obj2.group.nested || {}),
    },
  },
}

// which is equivalent to
const obj3 = {
  group: {
    nested: { 
      nestedValA: 'foo', // obj1
      nestedValB: 'something', // obj2 override
      nestedValC: 'else', // obj2 override
    },
    val: 'bar', // obj 1
  },
};

I want to add however something to @c1rrus 's comment:

// makes this group behave as though it contained 2 tokens

This is the right wording I think, I think it's up to design token tools (like style-dictionary), not the spec itself, to decide what to do with this $extends metadata:

• Leave it as metadata
• Hard-copy the super-group's (for a lack of a better term) properties/tokens into the group that extends it (e.g. for style-dictionary you would, optionally, do this during parsing the design tokens and putting it in the dictionary object like that)

The reason why I'd leave it as meta-data is for example because I don't want to bloat my CSS Custom Properties with tokens that are just duplicates of tokens in the super-group from which it extends, but I do want to have the metadata in order to help consumers of my design tokens to understand where they need to look to find the token they need. Let me know if that makes sense or if I should elaborate on this further with examples.

[nesquarx]

If alias and extends are token and group level keywords for the same activity. Would it decrease or increase confusion to have a single keyword for it?

…

On Tue, 29 Mar, 2022, 6:58 pm Joren Broekema, ***@***.***> wrote: - Presumably $extends can only be used on groups and must be a reference to another group, right? Extending tokens doesn't make sense to me (how would that be different to an alias token?) - If group B extends group A, should B also inherit properties from A like $description, $type, etc.? Thoughts? Yes to both those questions for me. Indeed, alias is the extends equivalent for a single token, or vice versa, extends is the alias equivalent for a token group. And yes, I think it makes sense to inherit the other properties, saves a lot of duplicate work I think. @TravisSpomer <https://github.com/TravisSpomer> with regards to your extension question, I would say we should follow something along the lines of what deepmerge <https://www.npmjs.com/package/deepmerge> does, which if I recall correctly is something along the lines of this: const obj1 = { group: { nested: { nestedValA: 'foo', nestedValB: 'qux', }, val: 'bar', },}; const obj2 = { group: { nested: { nestedValB: 'something', nestedValC: 'else', }, },}; // obj2 extends obj1 or aka obj2 deepmerges with obj1// so it becomesconst obj3 = { ...obj1, ...obj2, group: { ...obj1.group, ...obj2.group, nested: { ...obj1.group.nested, ...obj2.group.nested, }, },} // which is equivalent toconst obj3 = { group: { nested: { nestedValA: 'foo', // obj1 nestedValB: 'something', // obj2 override nestedValC: 'else', // obj2 override }, val: 'bar', // obj 1 },}; I want to add however something to @c1rrus <https://github.com/c1rrus> 's comment: // makes this group behave as though it contained 2 tokens This is the right wording I think, I think it's up to design token tools, not the spec itself, to decide what to do with this $extends metadata: - Leave it as metadata - Hard-copy the super-group's (for a lack of a better term) properties/tokens into the group that extends it The reason why I'd leave it as meta-data is for example because I don't want to bloat my CSS Custom Properties with tokens that are just duplicates of tokens in the super-group from which it extends, but I do want to have the metadata in order to help consumers of my design tokens to understand where they need to look to find the token they need. Let me know if that makes sense or if I should elaborate on this further with examples. — Reply to this email directly, view it on GitHub <#116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEKS36B27HNE5ZR4LD4K4C3VCMAO3ANCNFSM5QI7I2JQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jorenbroekema]

I think the behavior is a bit more complex on token groups. An alias is just a reference to another token and we don't use alias as a keyword, we use {}. With extending a token group, overrides also come into play, so it's more than just a flat reference.

I think $alias instead of $extends would be confusing because it implies direct/flat reference, rather than an extension with potential overrides.

Edit: so I guess my comment about equivalency wasn't actually correct.

[nesquarx]

Ah, then keeping them distinct makes sense.

…

On Fri, 1 Apr, 2022, 4:31 pm Joren Broekema, ***@***.***> wrote: I think the behavior is a bit more complex on token groups. An alias is just a reference to another token and we don't use alias as a keyword, we use {}. With extending a token group, overrides also come into play, so it's more than just a flat reference. I think $alias instead of $extends would be confusing because it implies direct/flat reference, rather than an extension with potential overrides. — Reply to this email directly, view it on GitHub <#116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEKS36F7TUSZN3MIF3CZW6DVC3JPBANCNFSM5QI7I2JQ> . You are receiving this because you commented.Message ID: ***@***.***>

[WanaByte]

This discussion looks to be related to #123.

[CITguy]

The whole point of using JSON as the chosen format is to ensure cross-platform interoperability using a well-defined data structure. By adding this type of syntax to the spec, we're adding implied behavior in addition to the data structure. Doing so ventures into the realm of defining a domain-specific language (DSL), which reduces downstream interoperability, because translation tools would need to add an extra layer of complexity in order to parse the DSL into the data structure before they can even begin translating the data.

Conversely, if you know you can perform the "extends" behavior using another language, why not use that language to generate your fully-defined tokens.json? If the tokens.json is already in a fully-defined state, there's no need for translation tools to parse a DSL to begin translating the data.

[jorenbroekema]

I think I might not have been clear enough in the initial post of this issue, but the $extends property would be pure metadata, it doesn't go as far as to imply "behavior" in tools that consume tokens.

In the section "Explicit tokens output" I'm merely suggesting that certain tooling could use this metadata property to configure how the tokens could be outputted to certain platforms.

In essence, this property is just a hint to a relationship between the current token and a parent token from which it inherits (or conversely, from which it extends), but what is done with this relationship is completely up to the tooling, all this proposal aims to do is standardize this "hint to a relationship between tokens/token groups" so that cross-tool interoperability is easy. The reason why I think this is important is because parent-child relationships between UI components is so common, e.g. an input-amount component is often the extension of a base input text component, both in design and code, so it would make sense that they have a parent-child relationship in the design tokens. This feature request is to make the relationship at least visible as metadata, for reasons I mentioned in my OP: discoverability, autocomplete/suggest, visualizing your tokens (e.g. in a graph/flow diagram).

[drwpow]

Another usecase this could cover for is giving the spec an equivalent of OpenAPI/JSONSchema’s allOf property for tokens:

{
  "typography": {
    "$type": "typography",
    "base": {
      "$value": {
        "fontFamily": ["Inter"],
        "fontWeight": "400",
        "fontStyle": "normal",
        "fontSize": "16px",
        "letterSpacing": "0.0625em",
        "lineHeight": "1.4",
      }
    },
    "body": {
      "$extends": "{typography.base}",
      "$value": {
        "fontSize": "14px",
      }
    },
    "heading1": {
      "$extends": "{typography.base}",
      "$value": {
        "fontSize": "18px",
      }
    }
  }
}

This could prevent a ton of errors if you wanted to automatically inherit properties from a “base token” and only provide minimal overrides where needed.

Even if the larger questions are unanswered about how groups do/don’t get merged (I’m personally struggling to see how $extend would work on groups with tokens of different $types without throwing validation errors), I think having $extends on the token level could yield many benefits without adding complexity to the spec.

Alternate proposal:

Many people in this thread have identified the overlap between $extends as-proposed and $alias, and how the two could possibly be combined. We could simply just steal JSONSchema’s solution outright, and combine both $extends and $alias into an array structure:

{
  "typography": {
    "styleC": {
      "$allOf": ["{typography.styleA}", "{typography.styleB}"],
      "$value": {
        "fontSize": "18px"
      }
    }
  }
}

Here, the idea would be that multiple composite tokens could be combined and merged in order, and optionally overridden. $value would be optional if $allOf were provided. Also not stuck on the name $allOf at all; just trying to outline slightly-different approach without getting hung up on naming.

As an aside, JSONSchema also has oneOf and anyOf but I don’t think those would be good fits for the DTCG spec, personally