How to generate the stylesheet for tokens when using both TextMate and Monarch grammars simultaneously

[fabiospampinato]

I've mostly added TextMate grammars support in an app of mine, but that app still uses a Monarch grammar too, and I haven't been able to figure out how to generate the right CSS for the tokens properly, without having the tokens defined in the Monarch stylesheet override the ones in the TextMate stylesheet and viceversa.

I'm hoping I won't have to rewrite the Monarch grammar to TextMate as that would have quite a few downsides:

• Most of the times that Monarch grammar will be the only one my users will need, so no need to load Oniguruma at all in that common scenario.
• The Monarch grammar is most probably going to be faster, as it uses JS regexes.
• Also importantly I already have the Monarch grammar at hand, but translating that for TextMate is going to be error prone at best and very time consuming.

I'm guessing the root issue is that I basically have 2 registries for grammars that don't know about each other, but how could I merge them when the registry for TextMate only seems to handle TextMate grammars?

Any ideas on how to address this issue?

/cc @bolinfest, in case you might have already encountered this issue. Thanks for monaco-tm by the way!

[alexdima]

@fabiospampinato
The only problem of mixing Monarch and TM is the color map. You want that colors with id 1, 2, 3 ... mean the same color to both Monarch and TM.

You basically need to collect all the colors used by the Monarch theme and all the colors used by the TM theme and create a single color map that contains all of them. A color map is just an array with the colors. This is used to encode/decode the color of a token as a small integer.

Once you compute the color map, you can define it for vscode-textmate when calling Registry.setTheme:

export declare class Registry {
    setTheme(theme: IRawTheme, colorMap?: string[]): void;
}

For Monarch, you can define it via monaco.languages.setColorMap:

declare namespace monaco.languages {
	export function setColorMap(colorMap: string[] | null): void;
}

I think that would make it possible to have text models using both tokenizers at the same time.

[fabiospampinato]

@alexdima Thanks for the help, it sounds doable in principle, but if ids [1..N] must have the same colors in both registries I suppose the ids themselves should also correspond to the same tokens under the hood, like if the first color is for strings in one registry it can't be for numbers on the other registry. Is it possible to enforce that somehow? 🤔

Alternatively what if I reserve the first 50 tokens or whatever for the Monarch grammar and then force the TM grammars to start from 51 for example? I suppose that should work in principle too and perhaps it would be much easier to implement? I'm thinking for that I may only need to add 50 dummy initial colors/tokens to one of the registry, compute the stylesheet for both, and make sure to insert them in the DOM in the right order basically. Does this sound reasonable/easier to you? If so I tried setting this to 100 but it didn't work, I'm guessing there's an array of colors to pre-fill somewhere or something, any pointers would be greatly appreciated.

[fabiospampinato]

Alternatively what if I reserve the first 50 tokens or whatever for the Monarch grammar and (...)

If that approach makes sense and it would be easier to implement it then I think I could do the following:

• TextMate theme: set a TextMate theme normally.
• Monarch theme: generate one dynamically from the TextMate one, ensuring that strings are colored the same way for example, should be ~easy.
• Monarch stylesheet: generate it normally and attach it to the DOM, no issues there.
• TextMate stylesheet: generate it normally, then patch it increasing all counters by 50 or whatever before attaching it to the DOM.
• Minimap colorization: generate both color maps, pad the Monarch one by 50 or whatever it is, concatenate the two and set the result.
• Off-editor colorization with editor.colorize: generate it normally, then patch the generated HTML increasing the counters by 50 too.
• In-editor colorization: this is the tricky bit I'm not sure how to approach, I suppose I could patch the output of grammar.tokenizeLine2 increasing the counters there too, but it's less clear to me if that would work at all and most importantly I have no clue about what the format for the Uint32Array object is 🤔

Does this sound reasonable?

[fabiospampinato]

I have no clue about what the format for the Uint32Array object is

It looks like that contains some metadata, but not exactly which id to use for the token, I'm not sure where that's resolved or if it would be feasible to patch that.

[fabiospampinato]

Actually maybe I could just increment the bits the encode the foreground color to achieve this. I'll have to try this, it sounds promising.

https://github.com/microsoft/vscode/blob/2f077172cb0fcbd06f5820b0a5125215e8d38435/src/vs/editor/common/modes.ts#L172

[fabiospampinato]

It actually worked!

For posterity, I did the following:

• Shifted all counters by 100 for the TextMate stylesheet, this ensures the classes for tokens from both registries won't overlap:

css.replace ( /mtk(\d+)/gi, ( _, nr ) => `mtk${Number ( nr ) + 100}` );

• Shifted the counters by 100 for tokens generated by TextMate grammars, this ensures both in-editor and off-editor tokens match the classes in the stylesheet:

for ( let i = 1, l = tokens.length; i < l; i += 2 ) {
  tokens[i] += 100 << 14;
}

• Padded by 100 the color map from the Monarch/global registry and concatenated that with the color map from the TextMate registry, this ensures the right colors are displayed in the Minimap:

const colorsMonarch = Monaco.TokenizationRegistry._colorMap.slice ( 0, 100 ),
      colorsTextMate = Style.getColors ( registry ),
      colorPadding = Monaco.Color.Format.CSS.parseHex ( '#00000000' ),
      colors = new Array ( 100 + colorsTextMate.length ).fill ( colorPadding );

for ( let i = 0, l = colorsMonarch.length; i < l; i++ ) {
  if ( !colorsMonarch[i] ) continue;
  colors[i] = colorsMonarch[i];
}

for ( let i = 0, l = colorsTextMate.length; i < l; i++ ) {
  if ( !colorsTextMate[i] ) continue;
  colors[100 + i] = colorsTextMate[i];
}

Monaco.TokenizationRegistry.setColorMap ( colors );

• What's left basically is just reusing the TextMate theme for Monarch grammars, I haven't addressed that yet but that should be a bit tedious at worst, and not really needed perhaps.

I love Monaco! <3