Advice on parsing style-dictionary tokens as a third party lib/tool

[jorenbroekema]

Hi,

I noticed something interesting, the dictionary object has a tokens/allTokens prop which contains your tokens, and it contains a little bit of meta data about these tokens. However, the dictionary object you get after .extend(config) seems to have less meta data when comparing it to the dictionary object that is passed into formats for example.

From the code, this seems to be because different platforms can have different transforms, which need to run before you can establish this enriched version of the tokens, so therefore transformations and reference resolutions are done per platform.

Compare:

{
  color: {
    font: {
      secondary: {
        filePath: 'tokens/color/font.json',
        isSource: true,
        value: "{color.base.green.value}",
      }
    }
  }
}

To an enriched version of this after exportPlatform:

{
  color: {
    font: {
      secondary: {
        value: "#00ff00",
        filePath: "tokens/color/font.json",
        isSource: true,
        original: {
          value: "{color.base.green.value}"
        },
        name: "sd-color-font-secondary",
        attributes: {
          category: "color",
          type: "font",
          item: "secondary"
        },
        path: [
          "color",
          "font",
          "secondary"
        ]
      }
    }
  }
}

I understand that this hard coupling to platforms is necessary due to how transforms work to adjust the source of truth so I'm not really suggesting that the lib changes anything.

I'm basically looking for best practice / advice on how I can best parse style-dictionary (design token spec compliant in general) design tokens without doing the same work that style-dictionary is already doing (gathering meta data, resolving token references, flattening tokens, etc.). Parsing the tokens would be simpler if I already knew about value versus original.value, similarly I would love it if I already knew about meta data like the token "Type" as well --> related: design-tokens/community-group#89 (review)

What I'm leaning towards is having my tool be provided with 1) the style-dictionary instance 2) platform to parse for, and then doing something like this in my tool:

const createDictionary = require('style-dictionary/utils/createDictionary');

// Assume we have the sd instance here, in my case style-dictionary-play.dev will probably pass it by iframe with postMessage to my tool

const enrichedTokens = sd.exportPlatform('css'); // platform to parse for is 'css' in this case
const enrichedDictionary = createDictionary({ properties: enrichedTokens });
console.log(enrichedDictionary.tokens, enrichedDictionary.allTokens); // profit! lots of the work is done for me :) now I can use them in my tool

Thoughts?

[dbanksdesign]

If I am understanding correctly: you want to use the dictionary object that has been transformed, resolved, and meta-data applied? What you have in the code snippet at the end is how I would achieve what I think you are trying to achieve. For more context could you zoom out a bit, what are you trying to acheive just so I make sure I properly answer your question. Thanks!

[jorenbroekema]

Yep that's correct.

What I'm trying to achieve is essentially passing a style-dictionary-play.dev project ID and then loading the transformed + resolved + meta-data applied tokens in a Figma plugin. Then I can import tokens straight into Figma, having the meta-data applied and references resolved really helps here.

The playground is just an example, we have style-dictionary integration in Backlight.dev which is where tokens will eventually be imported from as "the real use case", since our single source of truth with regards to design tokens is Backlight, but it will work very similar.

I have a POC working right now that I could share in a DM if you're curious.

Speaking more generically, I think given the Design Token Spec, a lot of tools may start writing their own parsers/processors/transformers for the spec, but style-dictionary already does this well and apart from positioning as "exporting design tokens to every platform" I think this lib should also be used as the go-to "Javascript parser for Design Tokens Spec".

Perhaps it makes sense to document in more detail on how users should do this (e.g. my use case), and perhaps clean up the API for this a bit and expose the entrypoints for example:

import { parseTokens } from 'style-dictionary';

const { tokens, allTokens } = parseTokens(tokens); // resolved + meta-data applied tokens
const { tokens, allTokens } = parseTokens(tokens, { dictionary, platform: 'css' }); // transformed + resolved + meta-data applied tokens

[dmackerman]

@jorenbroekema Very interested in your solution for using style-dictionary as a source of truth for Figma as well. Perhaps you could share your plugin code?

[jorenbroekema]

Hm yeah so what I do is I have my style-dictionary output in GitHub, and then use https://www.figma.com/community/plugin/843461159747178978/Figma-Tokens plugin for making use of these design tokens, it has a Sync with Github feature that works pretty well.

One thing to note here is that this Figma Tokens plugin does not use the exact same JSON format that you would get from the out of the box "json" formatter in style-dictionary, so I had to write some stuff to make it play well with this Figma Tokens plugin, I added some comments to explain what the transformations do and why, hope it helps:

function figmaCompatibilityParser(tokens) {
  // Recursively goes through values and trims `.value` away from it
  // Since reference values in figma tokens plugin trim this
  // Also trims away the name prop, figma tokens plugins does grouping itself
  const _trim = (_obj) => {
    Object.keys(_obj).forEach((key) => {
      if (key === 'value') {
        const val = _obj[key];
        const reg = /^\{(.*)\}$/g;
        const matches = reg.exec(val);
        if (matches && matches[1]) {
          _obj[key] = val.replace('.value', '');
        }
      } else if (key === 'name') {
        delete _obj[key];
      } else if (typeof _obj[key] === 'object') {
        _trim(_obj[key]);
      }
    });
  };
  

  // In order for reference tokens to work in the figma plugin
  // we actually want the original value as the value instead of the
  // resolved final value that style-dictionary gives us.
  const _original = (_obj) => {
    Object.keys(_obj).forEach((key) => {
      if (key === 'original') {
        _obj.value = _obj.original.value;
      } else if (typeof _obj[key] === 'object') {
        _original(_obj[key]);
      }
    });
  };
  

  // Move `"global": true` marked token files to "global" tokenset,
  // e.g. base tokens like colors, spacing, typography etc.
  // other files will be its own tokenset,
  // these are usually tokens for individual UI components
  const _global = (_obj) => {
    _obj.global = {};
    Object.keys(_obj).forEach((key) => {
      if (typeof _obj[key] === 'object') {
        Object.keys(_obj[key]).forEach((_key) => {
          if (_key === 'global' && _obj[key][_key] === true) {
            delete _obj[key][_key];
            _obj.global[key] = _obj[key];
            delete _obj[key];
          }
        });
      }
    });
  };

  const newObj = { ...tokens }; // clone original
  _trim(newObj);
  _global(newObj);
  _original(newObj);

  return newObj;
}

export default {
  source: ['**/*.tokens.json'],
  format: {
    figmaTokensPluginJson: (opts) => {
      const { dictionary } = opts;
      const parsedTokens = figmaCompatibilityParser(dictionary.tokens);

      return JSON.stringify(parsedTokens, null, 2);
    },
  },
  platforms: {
    json: {
      transformGroup: 'js',
      buildPath: '/tokens/',
      files: [
        {
          destination: 'tokens.json', // <-- this will be put on github and read by figma tokens plugin
          format: 'figmaTokensPluginJson',
        },
      ],
    },
  },
};

Disclaimer btw, this is pretty experimental, result of a first working proof of concept.

Essentially I see two ways to grab the transformed, resolved, and meta-data applied tokens from style-dictionary, the first approach is in my previous comment to grab it through JS directly, another way which I'm using now is just JSON stringifying (albeit with some extra transforms applied to it) and putting it in a file, and having the third party tool read from that file.

Here's the tokens file that I end up generating for my design system (just has base tokens and a button component atm): https://github.com/divriots/rev/blob/json-tokens/tokens/tokens.json

which looks like this in Figma