Implement app example using webpack

[rxaviers]

WIP fix for #464

Requires:

• Runtime code (smaller and quicker) for production #398
• rxaviers/globalize-webpack-plugin

ToDo:

• Declarative development - don't worry about loading CLDR or using pre-compiled instead.
• Fast Live Reload - use dynamically-created-generators. (via npm run start in this example)
• Optimized for Production (it's using a temporary branch with runtime support - need update when Runtime code (smaller and quicker) for production #398 lands).
• Compiled-Globalize-Code in separate chunks. (via npm run build in this example)
  • for the development locale (en in this example).
  • for each supported locale (en, es and zh in this example).
• Example instructions README.md.
• Using the same hello world example for the app. While the example must be concise, ideally we should provide one closer to real life. Should we? Ideas?
  • In the end, the demo implemented here was not very much a real life thing. But, at least, slightly better than the hello world one.
• Proper translation messages for supported locales.
• Update README linking to this example.

About this implementation:

• Application doesn't require developer to handle CLDR loading as can be observed at https://github.com/jquery/globalize/blob/fix-464-webpack-example/examples/full-app-npm-webpack/app/index.js. Globalize plugin will handle that.
• Webpack relevant configuration is
  https://github.com/jquery/globalize/blob/fix-464-webpack-example/examples/app-npm-webpack/webpack-config.js#L38-L44

new globalizePlugin({
    production: options.production, // true: production, false: development
    developmentLocale: "en", // locale to be used for development.
    supportedLocales: [ "en", "es", "zh" ], // locales that should be built support for.
    messages: "messages/[locale].json", // messages (optional)
    output: "globalize-compiled-[locale].[hash].js" // build output.
})

production is a boolean that tells the plugin whether it's on production mode (i.e., to build the precompiled globalize data) or not (i.e., to be in development mode - will use Live AutoReload HRM).

developmentLocale tells the plugin which locale to automatically load CLDR for and have it set as default locale for Globalize (i.e., Globalize.locale(developmentLocale)).

supportedLocales tells the plugin which locales to build/produce compiled-globalize-data for.

output is the name scheme of the built files.

Development mode

Start the Live AutoReload server by running the command below.

npm run start

Then, point your browser to http://localhost:8080/.

Production mode

Build the production bundles by running the command below.

npm run build

The build files are placed in dist. Point your browser at ./dist/index.html to check.

[sompylasar]

• Using a webpack loader or a function to load the messages (not just from plain JSON).

The real-world use case. I'm migrating a large code base to a set of modern tools, including webpack and, probably, Globalize for i18n, but my messages are in PHP (and that cannot be changed during this migration because they are reused by the backend i18n system). We're currently developing a loader that provides the messages to our frontend i18n system in the required format, and it would fit well if we decide to replace our frontend i18n with Globalize.

• Runtime language switching (and asynchronous loading of the required locale which is not loaded during initial page load).

The real-world use case is obvious: language switching without page reload.

[rxaviers]

• Using a webpack loader or a function to load the messages (not just from plain JSON).

I see. Can you provide examples of the not-just-from-plain-JSON-alternatives please?

Runtime language switching (and asynchronous loading of the required locale which is not loaded during initial page load).

👍

[sompylasar]

@rxaviers I'll give you an example but please do not write a specific loader for my specific example. A generic ability to specify a loader (or maybe better a plain function which could delegate to any other code if needed) would be better.

Our i18n system is Yii-based and uses a notion of per-module "category" and "key" (not just "key" which Globalize uses AFAIK), so a transform is required. But the system is highly customized to support multi-level overrides and fallbacks for each language, so this should be somehow handled by our custom loader (we actually delegate to the PHP component which knows how to do overrides and fallbacks, it does this for the server-side).

Here's an example of a single message file (of a single "category" for a "dialogs" module):

<?php

return [

    "_meta" => [
        "selector" => [
            "en",
        ],
    ],

    "dlg.message.title" => "Message",

    "dlg.message.button.close" => "Close",

    "dlg.error.title" => "Error",

    "dlg.info.title" => "Info",

    "dlg.confirm.title" => "Confirmation",

    "dlg.confirm.button.yes" => "Yes",

    "dlg.confirm.button.no" => "No",

    "dlg.prompt.title" => "Input",

    "dlg.prompt.button.yes" => "OK",

    "dlg.prompt.button.no" => "Cancel",

];

It's time for one more unexpressed thought that I've written about before. We need to be able to load different sets of messages for different apps based on a single platform (each app has one or more webpack entry points). Imagine two different applications with a set of common modules, each module depends on its own "category" with translations, each translation may be overridden at the application level. We should be able to build these apps separately and bundle only the translations being required by each app's components.

[rxaviers]

Sure, the messages option can be made to either accept the string (as documented above) or a function that passes locale as its first argument. For example:

new globalizePlugin({
    production: options.production, // true: production, false: development
    developmentLocale: "en", // locale to be used for development.
    supportedLocales: [ "en", "es", "zh" ], // locales that should be built support for.
    messages: function( locale ) {
        // Messages in the JSON format for requested locale.
        return getMessagesFor( locale );
    }
    output: "globalize-compiled-[locale].[hash].js" // build output.
});

Will include that...

We should be able to build these apps separately and bundle only the translations being required by each app's components.

The current implementation (rxaviers/globalize-webpack-plugin) is smart to bundle the translations being required only.

[sompylasar]

The current implementation (rxaviers/globalize-webpack-plugin) is smart to bundle the translations being required only.

Wow, that's pretty cool if I get you correct! How would you require specific message files from component files then, not having a gigantic "messages" object?

[sompylasar]

the messages option can be made to either accept the string (as documented above) or a function that passes locale as its first argument.

Does it by chance support asynchronous loading e.g. via a Promise returned from the "messages" function?
How will that function understand what translation files have been required by the components the entry points of the current compilation depend on?

[rxaviers]

Making myself clear, the current implementation requires a gigantic "messages" object to be available for the webpack plugin. But, the plugin is smart to use the messages being required only (and therefore, to generate the final bundle with the translations being required only).

Does it by chance support asynchronous loading e.g. via a Promise returned from the "messages" function?

Async might be tricky in webpack. Most of the compilation plugins (for example, [1], [2]) are sync calls.

I'd like to understand how you handle your custom messages and where async comes in. For example, are you handling files locally or remote? Does the processing require async? We could chat on IRC (http://irc.jquery.org/, #globalize).

How will that function understand what translation files have been required by the components the entry points of the current compilation depend on?

It's possible to pass, as another argument, the filename of the JS file being handled (in webpack terminology, the request variable):

    // For example:
    messages: function( locale, request ) {
        // Messages in the JSON format for requested locale.
        // locale: E.g. "en", "en-GB", etc.
        // request:  E.g., "my-app/index.js", "my-app/components/foo.js", so your function
        //           could look into "my-app/index.messages.json" or
        //           "my-app/components/foo.messages.json" for example
        return getMessagesFor( locale, request );
    }

Did I answer your question?

[sompylasar]

Async might be tricky in webpack. Most of the compilation plugins (for example, [1], [2]) are sync calls.

I'd like to understand how you handle your custom messages and where async comes in. For example, are you handling files locally or remote? Does the processing require async? We could chat on IRC (http://irc.jquery.org/, #globalize).

The current implementation executes an external process (a PHP console command which composes the messages using our custom override logic and emits them to stdout) which is an async operation.

I'm sorry for confusing you, we are currently using a plugin, not a loader (we tried loader before but failed). Because of lack of documentation on webpack plugin features we are trying to learn from existing plugins (including yours from now). The current implementation looks hacky and requires us to include the i18n bundle separately, not using webpack autoloading system. The code is not public, but I can show you some generic pieces we came up with.

We use the normal-module-factory to gather the categories from require requests (each module requires a special module path, e.g. i18n/my-category, which can be captured by a RegExp). We have to replace the request with a dummy file.

compiler.plugin('normal-module-factory', function (nmf) {
        langCategories = [];

        nmf.plugin('before-resolve', function (data, beforeResolveCallback) {
// ...
                                langCategories.push(langCategory);
                                data.request = __dirname + '/.i18n-dummy';

Then we use the emit compile step with asynchronous resolution (compilationCallback). We emit a compilation asset with a JS code that uses a globally available object to inject the translations it contains.

compiler.plugin('emit', function (compilation, compilationCallback) {
// ...
            compilation.assets['some-prefix-' + language + '.js'] =  source;
//

It's possible to pass, as another argument, the filename of the JS file being handled (in webpack terminology, the request variable):

Yes, this would be better, I think. This makes the "messages" function return a per-request object instead of the gigantic one, and Globalize then merges all the messages together, right?

[rxaviers]

Webpack is very powerful, but it has been really laborious to develop the plugin. I've also been using another plugins as a baseline given the lack of doc, plus I have also been reading the source code and have counted with help from bebraw and sokra.

Thanks for sharing your excerpts. I understood that your require categories return a no-op source. So, I didn't understand how your 'some-prefix-' + language + '.js' assets get loaded by your application? In other words, I didn't understand the link between the code that requires a category and the asset that is generated based on it.

Yes, this would be better, I think. This makes the "messages" function return a per-request object instead of the gigantic one, and Globalize then merges all the messages together, right?

Ok, will include it or suggest something else after thinking more about it.

[sompylasar]

I understood that your require categories return a no-op source. So, I didn't understand how your 'some-prefix-' + language + '.js' assets get loaded by your application?

Yes. That's the hacky flavor this solution gets. We access the translations via a singleton component which is populated with the messages. We have to side-load these files by a plain script tag or our own script loader, not by the webpack auto-loader. The messages get injected via a global variable shared between the core JS and the messages JS.

It looks like this (the code is pseudo-code):

// some-prefix-en.js
window.globalI18nManager.addMessages("some-category", {
    "some-key": "Some message"
}, "en");

// bundle.js
window.globalI18nManager.setLanguage("en");
$someElement.text( window.globalI18nManager.formatMessage("some-category", "some-key") );

[rxaviers]

It generates stuff like:

    <script src="vendor.07c80e6dfcd89796c7d5.js"></script>

    <!--
    Load support for the `en` (English) locale.

    For displaying the application in a different locale, replace `en` with
    whatever other desired supported locale, e.g., `pt` (Portuguese).

    For supporting additional locales simultaneously and then having your
    application to change display dynamically, load the multiple files here.
    Then, use `Globalize.locale( <locale> )` in your application to dynamically
    set it.
    -->

    <!-- <script src="i18n/zh.07c80e6dfcd89796c7d5.js"></script> -->

    <!-- <script src="i18n/pt.07c80e6dfcd89796c7d5.js"></script> -->

    <!-- <script src="i18n/es.07c80e6dfcd89796c7d5.js"></script> -->

    <script src="i18n/en.07c80e6dfcd89796c7d5.js"></script>

    <script src="app.07c80e6dfcd89796c7d5.js"></script>

[rxaviers]

@sompylasar your usecase should be tracked by rxaviers/globalize-webpack-plugin#1 and rxaviers/globalize-webpack-plugin#2

[rxaviers]

Content team, @kswedberg, @arthurvr, if anyone has any spare time, it would be great to count with your continued help reviewing this PR as well. Specially in:

• https://github.com/jquery/globalize/pull/481/files#diff-96890900e38239964de166018af8e295
• https://github.com/jquery/globalize/pull/481/files#r37784158

[sompylasar]

@rxaviers Thanks!

[kswedberg]

thanks, @rxaviers . I should be able to do a quick review this evening.

[arthurvr]

Dunno how we generally do that, but if a package is marked private giving a name is kinda moot.

[rxaviers]

👍

[rxaviers]

ef8dd29

[rxaviers]

Thank you all

[unindented]

Small typo, extra t at the end of pluralización.

[rxaviers]

Thanks, d317164.

[juhamust]

I wonder what is the correct place to report the issues with webpack-globalize, so I'll a quick on here.

Feels that globalize compiler is trying to be too smart for its own good (too much optimisation?): Methods like Globalize.numberFormatter are considered undefined time to time. Assigning them to variable like var numberFormatter = Globalize.numberFormatter helps, but only sometimes. I haven't found the pattern, yet.

---

Also, evaluation may consider variables undefined, while they're not:

// Angular translation filter
angular.module('app')
  .filter('t', function($filter) {
    return function (translationKey) {
      return Globalize.formatMessage(translationKey);
    };
  });

// Compilation error
./node_modules/globalize-webpack-plugin/GlobalizeCompilerHelper.js:72
      throw e;
            ^
ReferenceError: translationKey is not defined
    at eval (eval at extractor (./node_modules/globalize-compiler/lib/extract.js:67:9), <anonymous>:3:36)
    at ./node_modules/globalize-compiler/lib/compile-extracts.js:54:23
    at Array.reduce (native)

If I change the translationKey into valid translation-key string, it works. But obviously always returns the same translation. And all this worked at some point :(

Any wise words to help compiler to make it work like expected?

[rxaviers]

Hi @juhamust,

The problem you're facing is because you're using a dynamic construction in your Globalize options and the compiler is static. It doesn't mean you cannot use dynamic patterns in your code, but you have to change your code like the below:

var availableMessageFormatters = {
  foo: Globalize.messageFormatter(<foo>),
  bar: Globalize.messageFormatter(<bar>)
  ...
};

// Angular translation filter
angular.module('app')
  .filter('t', function($filter) {
    return function (translationKey) {
      return availableMessageFormatters[translationKey];
    };
  });

Note this is actually a good constrain for your code. Because, you could potentially run into a case where you are trying to format a message in production that you have never provided translation for.

Did I answer to your question? Would you change anything in our docs to make this clear?

PS: Anyway, if you need to report errors to webpack-globalize, do so at https://github.com/rxaviers/globalize-webpack-plugin.

[juhamust]

@rxaviers thanks a lot for your prompt and informative response. That really took me surprised, so perhaps the documentation really could be improved about the topic.

Duh. I tried building multilingual lookup table, but it still has that dynamic translationKey variable there.

// Build translation message lookup table
var messageLookupTable = {};
var messages = require('json!../common/locales/messages/en.json');

_.each(['fi', 'en', 'es', 'de'], function(languageCode) {
  messageLookupTable[languageCode] = {};
  // Using english translation keys as a reference
  _.each(messages['en'], function(msg, translationKey) {
    messageLookupTable[languageCode][translationKey] = Globalize.messageFormatter(translationKey);
  });
});

So I really need to generate JS versions from all of those JSON message files? That's something I would expect the compiler is doing on my behalf?

Thanks again.

UPDATE: I created a simple from JSON to JS conversion step in building phase and it seems to work as expected.

[rxaviers]

@juhamust, can you give a little more details on what you are trying to accomplish? Is this part of a frontend or backend application? Where are you going to use messageLookupTable later in your application? I'm asking, because Globalize is pretty flexible and knowing your application helps me to give a more accurate answer. Thx

[juhamust]

@rxaviers I responded in separate ticket instead of polluting this PR thread

[rxaviers]

👍 I've replied you there.