Support Composite Tokens

[wraybowling]

Lately I have been bumping into Style Dictionary's lack of support for composite tokens. At the time of this writing, composite tokens are part of the w3c spec draft, most notably for typography https://second-editors-draft.tr.designtokens.org/format/#typography

As an example of a tool making use of the spec, Figma Tokens is already providing composite tokens for typography. The token-transformer utility then outputs composite tokens. Finally, style dictionary is the weak link in making use of them, outputting scss variables as [object Object]
see related issue: tokens-studio/figma-plugin#1088 (comment)

Details

Consider the following section from the attached tokens.json file which came from the Figma Tokens plugin

    "heading": {
      "level-1": {
        "value": {
          "fontFamily": "{fontFamily-brand}",
          "fontWeight": "Light",
          "lineHeight": "48",
          "fontSize": "{fontSize.2xl}",
          "letterSpacing": "0%",
          "paragraphSpacing": "0",
          "textDecoration": "none",
          "textCase": "none"
        },
        "type": "typography",
        "description": "Heading 1"
      }
    }

When the above is passed through token-transformer, and then Style Dictionary, the following SCSS (abridged) is produced:

$font-size-4xl: 64;
$font-size-5xl: 96;
$normal: 140%;
$heading-level-1: [object Object];
$heading-level-2: [object Object];
$heading-level-3: [object Object];
$body-thin: [object Object];
$body-regular: [object Object];
$body-medium: [object Object];
$body-bold: [object Object];
$display-medium: [object Object];
$display-large: [object Object];
$caption-regular: [object Object];
$xs-bold: [object Object];
$background-white: #ffffff;
$background-light: #f9f9f9;
$background-blue: #0059B8;

To Reproduce

Steps to reproduce the behavior:

1. Create headings
2. export tokens
3. feed into token-transformer
4. feed transformed tokens through style dictionary to scss

Alternatively, download my test project zip file and run npm run build

Expected behavior

I expect to see on-spec typography tokens converted into valid SCSS variables

[moarpie]

@wraybowling I am struggling with the same issue as well and thought there was some workaround, but do I understand you correctly that this is simply just not possible at the moment?

[mthines]

I got it working using a custom transformer, that I called figma/web/flatten-properties :)

const StyleDictionary = require('style-dictionary');

StyleDictionary.registerTransform({
	type: 'value',
	transitive: true,
	name: 'figma/web/flatten-properties',
	matcher: ({ type }) => {
		return ['typography', 'composition'].includes(type);
	},
	transformer: ({ value, name, type }) => {
		if (!value) return;

		const entries = Object.entries(value);

		const flattendedValue = entries.reduce(
			(acc, [key, v], index) =>
				`${acc ? `${acc}\n  ` : ''}--${name}-${StyleDictionary.transform['name/cti/kebab'].transformer(
					{ path: [key] },
					{ prefix: '' },
				)}: ${v}${index + 1 === entries.length ? '' : ';'}`,
			`${name.includes(type) ? '' : `${type}-`}${name}-group;`,
		);

		return flattendedValue;
	},
});

I hope it solves your issue :)

[wraybowling]

I got it working using a custom transformer, that I called figma/web/flatten-properties :)

I appreciate seeing a workaround in the thread, but this issue should not be closed until Style Dictionary can handle the w3c spec without it being something custom. Perhaps we could use your transformer as the starting point for the new default.

[jakobe]

Hey, we've experienced the same issue, so I'll just chime in with our findings / solution:

When outputting typography css classes, it's really useful to have all the properties for a given typography grouped under value (as in the example from the OP ☝🏻 ).
This way we can easily create a formatter that uses the path of the composite token as the css selector and the child values as the css declaration block.
E.g. continuing from the example above (assuming the tokens have been transformed):

.heading-level-1 {
   font-family: "Some Brand Font Face";
   font-weight: 300;
   line-height: 48px;
   font-size: 24px;
   letter-spacing: 0em;
   text-decoration: none;
   text-transform: none;
}

On top of that it makes sense to group:

closely related style properties that are always applied together

as mentioned in the draft from the W3C Design Tokens Community Group on Composite Tokens in general as well as the specific types for typography.

BUT: When outputting this structure as flattened tokens (e.g. as css or scss) we get the [object Object] problem as mentioned above. The transformer that @mthines suggests above kinda solves this issue, but only for css variables, and not for scss nested maps as those maps in the current implementation both fails on outputting the expanded tokens/scss variables as well as the references to them from within the nested maps:

$heading-level-1: [object Object];
...
$tokens: (
  ...
  'heading': (
    'level-1': $heading-level-1
    )
  ),
  ...
);

Or solution is to:

1. Expand the composite tokens into separate tokens for each child value in the dictionary.allTokens array. This solves the [object Object] problem and the scss/variables format works out of the box with the current implementation 🥳
   We do this this by creating a copy of the dictionary and manipulating the allTokens|allProperties array and then use this expanded dictionary for the formatter:

const expandToken = (
  token: TransformedToken,
  nameTransformer: (token: TransformedToken) => string
): TransformedToken | TransformedToken[] => {
  if (typeof token.value !== "object") {
    return token;
  }

  const { attributes, name, path, value: _value, ...rest } = token;
  return Object.entries(token.value).map(([key, value]) => {
    const childPath = [...path, key];
    var childName = nameTransformer({
      ...token,
      path: childPath,
    });

    return {
      ...rest,
      ...(attributes ? { attributes: { ...attributes, subitem: key } } : {}),
      name: childName,
      path: childPath,
      value,
    };
  });
};

// Create a shallow copy - we'll create new tokens in `allTokens|allProperties` when expanding composite tokens below:
const expandedDictionary = { ...dictionary };
// Expand composite tokens
// Note: we need to overwrite both `allTokens` and `allProperties` as long as the latter deprecated alias exists
// See: https://amzn.github.io/style-dictionary/#/version_3?id=style-properties-%e2%86%92-design-tokens
expandedDictionary.allTokens = expandedDictionary.allProperties =
  dictionary.allTokens
    .map((token) => expandToken(token, nameTransformer))
    .flat();

This produces the following output (continuing with the example):

$heading-level-1-font-family: "Some Brand Font Face";
$heading-level-1-font-weight: 300;
$heading-level-1-line-height: 48px;
$heading-level-1-font-size: 24px;
$heading-level-1-letter-spacing: 0em;
$heading-level-1-paragraph-spacing: 0,
$heading-level-1-text-decoration: none;
$heading-level-1-text-case: none;
...
$tokens: (
  ...
  'heading': (
    'level-1': (
       'fontFamily': $heading-level-1-font-family,
       'fontWeight': $heading-level-1-font-weight,
       'lineHeight': $heading-level-1-line-height,
       'fontSize': $heading-level-1-font-size,
       'letterSpacing': $heading-level-1-letter-spacing,
       'paragraphSpacing': $heading-level-1-paragraph-spacing,
       'textDecoration': $heading-level-1-text-decoration,
       'textCase': $heading-level-1-text-case
    )
  ),
  ...
);

This also solves the [object Object] problem for the flat scss variables output that are referenced from within the nested scss maps in the scss/map-deep format

2. Enhance the scss/map-deep format (actually just the part of the template that outputs the value prop) to also expand the child values for composite tokens into a nested map:

if (typeof obj['value'] === 'object') {
   // if we have found a composite group of child values, use the Sass group "(...)" syntax and loop on the children:
   var compositeProp = obj['value'];
   output += '(\n'
   output += Object.keys(compositeProp).map(function(subKey) {
      var indent = '  '.repeat(depth+1);
      var subvalueName = nameTransformer({...obj, path: [...obj.path, subKey]});
      return `${indent}'${subKey}': $${subvalueName}`;
   }).join(',\n');
   output += '\n' + '  '.repeat(depth) + ')';
} else {
   // if we have found a leaf (a property with a value) append the value
   output += `$${obj.name}`;
}

For option 2 to work we have to enhance the map-deep.template (this could probably be good to do anyway) but we also need to expand the composite token so that the scss variables referenced within the nested scss maps get outputted.
That essentially means we need to override the current scss/map-deep format, but I don't see anywhere else where we can get to the dictionary after it has been resolved and transformed? Maybe a pre-format action or...?

Might need @dbanksdesign to chime in here (nudge-nudge 😉) for any thoughts on that part - as well as him being part of the W3C Design Tokens Community Group and might elaborate as to wether this approach is fit with their current thinking on composite tokens.

Jakes 😊

[dbanksdesign]

First, apologies for being a bit MIA (personal life has been extraordinarily busy the past few months). To start, the core of Style Dictionary does support composite tokens, but as you all know the pre-built transforms and formats do not understand how to deal with composite tokens. Here is an example which uses color tokens as composites of hue, saturation, and lightness: https://github.com/amzn/style-dictionary/tree/main/examples/advanced/transitive-transforms Style Dictionary supports composite tokens in its core architecture by how tokens get identified, transformed, and resolved. I created this quick example to show that transforms (at least for non-composite tokens used inside composite tokens) and resolutions work. https://stackblitz.com/edit/style-dictionary-example-vkhiqj?file=build/test.json

The reason you see [object Object] in the output is the built-in formats assume a token's value has been transformed into a string at that point.

The difficulty with composite tokens is there is usually more than one way to output them. In this issue there are 2 different potential correct outputs: splitting the token up into separate variables (like the SCSS example), and outputting a CSS helper class like .heading-1 {}. Also, what would a typography token look like for other platforms like Android or iOS? This is not an unsolvable problem, but just requires some deep thinking, and I appreciate all the thinking that has gone into these comments. At first glance I think the more sensible default behavior would be to split tokens up as this would make the least amount of assumptions about the user's setup and be the most cross-platform compatible.

@wraybowling could you fill in what your expected output would look like?

The other difficulty with composite tokens is they require a specific structure to the value. Non-composite tokens are just strings or numbers, but now a composite token's value structure or type is dependent on the composite type (border has a size, color, and style). The W3C spec is still looking for feedback on the structure of these composite types, for example: design-tokens/community-group#102

I think this is definitely something we should support this in some way in version 4.

[jbarreiros]

I was intrigued by jakobe's idea to split composite tokens into individual tokens.

Worked out a similar solution using a custom parser (stackblitz), which works pretty good.

The one thing I can't settle on is whether to persist the composite token after splitting it up. Keeping it around makes it easy to achieve the following:

--border-thin-width: 1px;
--border-thin-style: solid;
--border-thin-color: rebeccapurple;
--border-thin: 1px solid rebeccapurple;   /* note, for this to work, the token's path is `border.thin.@` */
--card-border: 1px solid rebeccapurple;   /* alias! `card-border: { value: "{border.thin.@}" }` */

Though, that pattern may only work well for composite tokens that map to a CSS shorthand property. For example, the typography composite token includes a letter-spacing property which isn't part of the font shorthand.

Furthermore, aliasing wouldn't be possible without persisting the composite token.

[jkinley]

You all know about Lukas Oppermann's repo, right? https://github.com/lukasoppermann/style-dictionary-utils

[lfantom]

@jakobe I really like this approach - at what stage are you implementing this in order to mutate the dictionary prior to passing to the formatter?

Thanks

[jorenbroekema]

To summarize, there are 3 solutions in this thread:

• Expanding the object value token into separate tokens, 1 token for each property -> parser/preprocessor-level
• Creating a CSS class and putting each prop inside as a separate CSS rule -> format level
• Converting the object value into a CSS shorthand, which supports only a limited subset of typography token -> transform level

Obviously all three are great ways to tackle this, but I think the first option should be something that Style Dictionary should come out of the box with as an opt-in, which means you as a user can still opt for the other alternatives. It might be cool to know that both sd-transforms and style-dictionary-utils have a transform for the third option (CSS shorthand)

Fortunately, I created an expand composites utility in sd-transforms already because Tokens Studio has had composite type tokens for a while now, and this supports many edge cases such as references inside such tokens, cross-file references, etc. It wouldn't be much work to add this feature into Style Dictionary itself, it has 100% test coverage and is mostly re-using Style Dictionary utilities already, making it a good fit.

Suggestion for API:

{
  "source": ["tokens.json"],
  "expand": {
    // only expand for typography / run function for typography tokens
    "include": { "typography": true }, // or Function
    // expand for all composite types except for typography, there we don't expand or we run function to check
    "exclude": { "typography": true }, // or Function
    
    // not specifying either exclude or include means we run expand on all tokens, analogous to "expand": true
    // except this way allows you to specify a typesMap
    
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

Where "typography" can be any token type to granularly control which types should be expanded.
You can also do "expand": true to expand all object value type tokens as a shortcut, rather than defining it per token type.
true and false (default) are valid values, but you can also supply a callback function (if you're in JS context, not possible in JSON) which allows you to conditionally expand on a per token basis, for ultimate granular control.

@dbanksdesign what do you think?

[lfantom]

@jorenbroekema Agreed, in an ideal world I feel we should be able to control this at a platform level, for example, I want to keep my composite token format for Figma and Web, as they both handle Typography tokens (in Figma, and with CSS Shorthand).

However for Android and iOS I may want to expand composites as we don't have the option for certain shorthand properties, so need to be handled individually.

My thoughts are that this should be controlled at the format level, but almost like a pre format stage, where we can expand the composite tokens into a group, for example, and pass the updated dictionary back so it can then go through the relevant template, e.g Android. I'd be interested to know your thoughts, or if anyone has taken this approach?

[jorenbroekema]

Great points @lfantom , in v4 we have something called preprocessors which allows processing the dictionary object after the parsing step, this happens on a global level before any transforms are done, which is platform specific.
https://style-dictionary-v4.netlify.app/reference/hooks/preprocessors/ docs about that here, though fair warning these docs are highly work in progress and the domain will change in the future, it's just a temporary dump.

I'm considering that perhaps we need a postprocessors hook that allows you to do the exact same but after transforms, so it's platform specific. I also discussed this with Danny in a private message and he seemed to agree with this concept.

This means that the expandTokens utility can be done on a preprocessor (global) OR postprocessor (platform-specific) level.
The suggested API can still apply, you'd be able to put it on a global config level meaning it'll be applied as a built-in preprocessor, or you can put it on the platform config level meaning it'll be applied as a built-in postprocessor.

Thoughts?

[lfantom]

@jorenbroekema Yes I think the idea of adding a post-proccessors options is a good one - I've been testing out an implementation where this step is added after the transforms have taken place, and that seems to work well, as I've found it's necessary for the references to be resolved prior to expanding the tokens.

One issue that I have found with this however, is when expanding say typography tokens, I'm able to assign the new type to each new token e.g fontSize is expanded into a single token and given the relevant type fontSize. However because this stage happens after the transforms, any custom attribute cti structure is then ignored, so the tokens carry their original attribute structure. Sometimes this is a problem, if using a component based naming structure.

I hope that makes sense, I'd be interested to hear your thoughts on how to handle this? The only option I could see, was to add the additional attribute structure at the point of expanding the composite tokens, but that feels like the wrong place for it. Or maybe you could use your suggested api structure with expand: true but then also pass a callback function, to attach the new attribute structure?

[jorenbroekema]

@lfantom can you show a small example of what you mean with the attributes structure not being passed correctly when expanding a token in a postprocessor hook?

I was thinking that for this expand postprocessor, we could run the transformToken function which executes all applied transforms on a token, for each newly created/expanded token. Would that solve the problem you are describing?

[jorenbroekema]

@lukasoppermann I think I arrived at a somewhat elegant API now #848 (comment)

[lukasoppermann]

Hey @jorenbroekema you can use json5 for the code fence to get rid of the red error messages.

I don't quite get it. Would this be valid?

{
  "source": ["border.json"],
  "expand": {
    "include": { "border": true }, // or Function
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

What about this?

{
  "source": ["border.json"],
  "expand": {
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

Couldn't you just do an array instead for include?

{
  "source": ["border.json"],
  "expand": {
    "include": [ "border", fnExpandTypography],  // fnExpandTypography would return typography or undefined
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

I think I still prefer this:

{
  "source": ["border.json"],
  "expand": {
      "border": {
        "width": "borderWidth"
      },
      "typography": true,
      "shadow": expandShadow // fn
    }
  }
}

const expandShadow = (token, platform) => { // idk which arguments it would get
  if(!condition) return false // maybe undefined could also work
  return {
    "blur": "dimension"
  }
}

[jorenbroekema]

Oh yeah true thanks for the tip.

I don't quite get it. Would this be valid?

Yes, that would only expand border tokens, or if it's a function it will run the function for each border token to determine per token if it should be expanded.

What about this?

Yep, that would expand all tokens, similar to "expand": true except now you can pass a typesMap to configure that.

Couldn't you just do an array instead for include?

Yes possibly, perhaps we should consider that when you use a Function, you have access to the token.type, so it doesn't really make sense to have functions wrapped in composite types keys, so maybe this:

{
  "source": ["border.json"],
  "expand": {
    "include": [ "border", "typography"],
    // OR (not in JSON but in JS in this case)
    "include": (token, config, platformCfg) => true,
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

and then for exclude it would be the same API except reverse effect:

{
  "source": ["border.json"],
  "expand": {
    "exclude": [ "border", "typography"],
    // OR (not in JSON but in JS in this case)
    "exclude": (token, config, platformCfg) => true, // true means it will get excluded, false means included
    "typesMap": {
      "border": {
        "width": "borderWidth"
      }
    }
  }
}

I think I still prefer this

I just think that it's a bit awkward that it's either false, true OR Object which kinda means true but with extra attached meta information, but it's definitely not a bad choice.
What it misses is a way to expand all except for a few exceptions, meaning with a growing number of composite types it might get a little verbose to specify for each type when you just want to exclude 1 type.

[jorenbroekema]

Support for expanding composite type tokens on the preprocessor level (either globally or per platform) was released in prerelease.27: https://v4.styledictionary.com/reference/config/#expand

In the next prerelease (28), if such composite tokens are not expanded into separate tokens, there will now be built-in transforms for CSS (also included in the css, scss and less transformGroups by default) that will transform these object-value tokens into CSS shorthands, but keep in mind that not every CSS shorthand supports every single composite type token property (e.g. typography -> letterSpacing)

[mikeriley131]

@jorenbroekema Is there a resolution for the letterSpacing thing or should it just be ignored? I'm getting the following warnings after running SD:

Unknown CSS Font Shorthand properties found for 9 tokens, CSS output for Font values will be missing some typography token properties as a result:

letterSpacing for token at textStyle.heading-3 in src/fts-tokens/tokens.json

I'm not using the letterSpacing fields in Figma Tokens Studio and tokens build correctly so it's not a big deal, just annoying.

[jorenbroekema]

I'm not using the letterSpacing fields

Style Dictionary disagrees apparently, can you show me the src/fts-tokens/tokens.json file? otherwise I can't reproduce why you're getting the error, whether erroneously or not.

You can turn it off btw: https://v4.styledictionary.com/reference/logging/ (warnings -> 'disabled')

[mikeriley131]

Oh, I'm mistaken. I am using that field.

"heading-3": {
  "mobile": {
    "value": {
      "fontFamily": "{font.family.sans}",
      "fontWeight": "{font.weight.semibold}",
      "fontSize": "2rem",
      "lineHeight": "1.125",
      "letterSpacing": ".15em"
    },
    "type": "typography"
  },
  "tablet": {
    "value": {
      "fontFamily": "{font.family.sans}",
      "fontWeight": "{font.weight.semibold}",
      "fontSize": "2.25rem",
      "lineHeight": "1.125",
      "letterSpacing": ".15em"
    },
    "type": "typography"
  },
  "desktop": {
    "value": {
      "fontFamily": "{font.family.sans}",
      "fontWeight": "{font.weight.semibold}",
      "fontSize": "2.5rem",
      "lineHeight": "1.125",
      "letterSpacing": ".15em"
    },
    "type": "typography"
  }
},

However letter-spacing is not part of the font shorthand like the others and they don't seem to transpile to a CSS letter-spacing property so it gets "lost".

$font-style-heading-3-desktop: 600 2.5rem/1.125 'Source Sans 3', -apple-system, blinkmacsystemfont, helvetica, sans-serif;

$font-style-heading-3-mobile: 600 2rem/1.125 'Source Sans 3', -apple-system, blinkmacsystemfont, helvetica, sans-serif;

$font-style-heading-3-tablet: 600 2.25rem/1.125 'Source Sans 3', -apple-system, blinkmacsystemfont, helvetica, sans-serif;

[jorenbroekema]

Yeah so if you're using properties that don't fit into the CSS shorthand, it's best to expand such object-value tokens into separate tokens for each property: https://v4.styledictionary.com/reference/config/#expand

Also make sure to give this a read if you're using Tokens Studio / sd-transforms: https://github.com/tokens-studio/sd-transforms?tab=readme-ov-file#using-expand

[mikeriley131]

Thanks for the info, @jorenbroekema, but when I add the expand property and run the build, I get the following error:

Error: tries to reference font.family.sans, which is not defined.

and points to node_modules/style-dictionary/lib/utils/references/resolveReferences.js/183.

Here's my config:

const sd = new StyleDictionary({
  source: ['**/fts-tokens/tokens.json'],
  preprocessors: ['tokens-studio'],
  expand: {
    typesMap: expandTypesMap,
  },
  platforms: {
    ...
  }
});

This is the related section of my tokens.json file:

{
  "base": {
    "font": {
      "family": {
        "sans": {
          "value": "'Source Sans 3', sans-serif",
          "type": "fontFamilies"
        }
      }
    }
  }
}

Everything built without issue prior to adding the expand property.

[mikeriley131]

Any idea what might be going on here, @jorenbroekema?

Let me know if you'd prefer I create a new issue for this.

[thomasmattheussen]

@mikeriley131 This looks to be related to excludeParentKeys (see https://github.com/tokens-studio/sd-transforms?tab=readme-ov-file#options)

[mikeriley131]

@mikeriley131 This looks to be related to excludeParentKeys (see https://github.com/tokens-studio/sd-transforms?tab=readme-ov-file#options)

@thomasmattheussen could you clarify what you mean by "related to"?
I am using excludeParentKeys: true currently in the register function. Is that causing the issue?

[thomasmattheussen]

Well, I see that your error says something about font.family.sans while it's actually base.font.family.sans according to your tokens.json file. So my mind just jumped to "has probably got something to do with the excludeParentKeys option" 😅

[jorenbroekema]

Yeah so the problem is that currently in v4.0.0 the custom preprocessor (excludeParentKeys) runs AFTER the expand utility. This means that when the expand utility is doing its thing, references are "broken" because excludeParentKeys util hasn't ran yet.

This is fixed in #1268 which will be in the next release. I'll try to get that out today.

[mikeriley131]

Just upgraded to v4.0.1 and my issues are resolved. Thanks!