Single JSON file containing shortcodes

[nolanlawson]

This is a small feature request, and I totally understand if it's out-of-scope for the project.

As of emojibase-data v6, the data.json file and shortcode file need to be stitched together to have both the shortcodes and emoji data in the same file (#64).

However, for my own use case (nolanlawson/emoji-picker-element#47), having a single file was a nice convenience. By default, the library just fetches the JSON from jsdelivr (this is designed for folks not using a build step). Switching to multiple files is possible, but pushes complexity either to me or my consumers (or both).

To ease the process of upgrading from v5 to v6, perhaps a new JSON file could be added – call it data-with-shortcodes.json – which just includes the default Emojibase shortcodes? This would provide the most direct upgrade path to v6, while also allowing users to stitch together the files themselves if they prefer some other shortcode set.

I can only speak for my own library, but this would definitely help with the upgrade process. 🙂 Thanks for creating emojibase!

[milesj]

@nolanlawson How are you loading the datasets? If you use fetchEmojis, it will do the stitching for you and the resulting dataset will be similar to v5. https://emojibase.dev/docs/api#fetchemojis

There's also this API if you have the datasets already: https://emojibase.dev/docs/api/#joinshortcodes

[milesj]

So I dug around your source code a bit, and it looks like you're using the full CDN URL. I understand how this can be problematic.

The problem with data-with-shortcodes.json is that it will be immediately out of date come spring once the next version of emoji are released. The previous emojibase shortcodes are deprecated and will no longer be updated. The suggested path forward is the new emojibase shortcodes (which are quite different), or another preset. This is about a 6 month window where time may be better spent thinking of an alternative?

Spitballing here, but option may be to update dataSource to support a string or an object, where the object is emojibase settings, and the string is for external sources (file system, other CDNs, etc).

dataSource: { 
  locale: 'de', // Should probably be inherited from the other setting
  shortcodes: ['cldr', 'emojibase'],
  compact: false,
  version: 'latest',
},

[nolanlawson]

The problem with data-with-shortcodes.json is that it will be immediately out of date come spring once the next version of emoji are released.

Ah, I didn't realize that the emojibase shortcodes are going away. I guess in that case, I would prefer for some shortcodes to be the default, so that there could be a single "default" JSON file.

fetchEmojis is an option, but the problem is that I designed emoji-picker-element to be as efficient as possible, which I can only do if I handle the fetch myself. In particular, emoji-picker-element caches the ETag so that, on second load, it can just issue a simple HEAD request and only fetch the full JSON if necessary. I could use fetchEmojis, but I would have to compute a checksum or something from the JSON rather than using the more efficient ETag method. (Incidentally, I already do this, but this is just a fallback for Safari which doesn't support cross-origin ETag headers.)

Also, I like the idea of fetching a single JSON file rather than two – it's a small optimization, but it's an optimization.

I guess if data-with-shortcodes.json does not exist, then I could either 1) create a separate repo to build it myself, and then use unpkg.com as the CDN, or 2) require consumers to stitch together the JSON themselves. I'm less a fan of # 2 because I like it that emoji-picker-element works out of the box…

Anyway, I understand that I'm using emojibase in a slightly unusual way, and I'm not directly using the APIs, so perhaps I'm the only one who's running into this issue. Thanks for at least entertaining my suggestion. 🙂

[milesj]

Yeah sorry, I have no plans of merging shortcodes back into the primary dataset for 2 reasons.

1. The old shortcodes were the biggest complaint about emojibase, as they did not align with GitHub/Slack/Discord -- all 3 of which are different from each other. I no longer want to force certain shortcodes on the consumer, as they should choose which preset they want.
2. Removing shortcodes reduced all dataset file sizes by quite a lot.

I'd argue that multiple HTTP requests is a non-issue, as the browser/fetch is rarely the bottleneck. I avoid subsequent requests by caching the response in local/session storage (https://github.com/milesj/emojibase/blob/master/packages/core/src/fetchFromCDN.ts#L46). I would suggest doing something similar, coupled with your ETag implementation, and maybe supporting a shortcodesSource setting.

[nolanlawson]

Thanks for your response. Using LocalStorage/SessionStorage is a reasonable approach, but I'm avoiding it because 1) it's synchronous, 2) accessing it means reading the entire emoji database into memory, and 3) accessing it also means running JSON.parse on the entire database. My technique is to use IndexedDB, so that all the querying can be done asynchronously, avoiding these costs on subsequent page load. Some details here.

I agree that in HTTP/2 having one additional network request is not a huge deal. I may look into caching the two ETags separately, or writing a small npm package to stitch the files together and publish the result.

Thanks again for creating emojibase! I appreciate all the hard work you've put into this, and dealing with users with very meticulous demands. 🙂