Ability to display the "uncommon" fonts in the platform

[dpalomino]

As discussed, we need a way to properly display "uncommon" fonts not currently supported in the platform (and specifically the font used by a particular partner).

This is a follow up from #1549 issue (please see details there).

[clash99]

The ones mentioned yesterday: KNU, Karen, Padauk (Burmese) and Noto Myanmar.

[clash99]

We should probably have a discussion related to this for the bigger picture. Scalability and maintenance could get difficult very quickly. Some options off the top of my head:

• Is a reasonable solution to rely on users to have these fonts installed locally?
• We could detect these uncommon fonts and provide links in a alert message to download them (would this be from us or someone like google? - but what if they aren't provided by Google...)
• Other options?

[amplifi]

Flagging this -- even if we can get the character set mapping to work, for any non-Unicode charset portions of the platform will still not function (at all, or may exhibit unpredictable behavior), including but not limited to: any type of sorting, search, string validation, export...

This is not just a font issue; this is a character set/encoding issue.

[alukach]

Regarding @amplifi's comment, I think it's important that we are very clear about which problems we're talking about. There seem to be two issues at hand:

1. What do we do with non-unicode data? Should it be supported? What do we do if it's not supported? If we are supporting it, how do we handle those fonts?
2. If the data is encoded in unicode, how do we ensure that its characters are represented correctly on our front-end (i.e. how do we avoid "tofu")?

These are two distinct issues and can be addressed separately. @amplifi seems to be suggesting that the answer to #1 should be "no, non-unicode is not supported". I vote a +1 to this (simply for the sake that it's easier than suggesting that it is supported) and suggest that we agree on a policy that "Cadasta expects all data to be UTF-8 compliant" until we get to a place where we are confident that it is actually needed. It may be prudent to add some tooling to error on non-unicode input (maybe this already exists as part of our sanitization tooling).

I'd suggest renaming this Issue to read "Ability to display the "uncommon" unicode characters in the platform" to specify that it only relates to dilemna #2.

With regards to #2 I'd like to add a point to this solution's wishlist:

• If we are loading fonts via URL (such as from Google or from our own /static dir), we should ideally aim for a solution that only loads fonts that are needed as loading font-faces for every possible language seems a bit excessive. Not sure what the right way to do this would be. We could ask for a user to select the used languages when creating a project. Or, aim for some magic language detection based on characters (https://stackoverflow.com/questions/4545977/python-can-i-detect-unicode-string-language-code, https://github.com/wooorm/franc) and dynamically load fonts as-needed for that representation (this feels far fetched but a dev can dream).

[alukach]

Looks like my request for as-needed font loading is already built in to modern-browsers: https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/webfont-optimization#unicode-range_subsetting

[clash99]

@alukach

For Q1, I agree we should not support non-unicode data. Karen is pseudo-unicode meaning that it is a combination of unicode and not so do we throw it all out? If its all converted to unicode then I think we are good there.

For Q2, since the users at-hand have the fonts installed locally we don't have to do anything with the "visual images that typically represent the parts of language" display issue on the platform right now. :)
I think we need to set aside some time to discuss this in Madrid. I think we should put this into a larger discussion regarding how we handle all files that feed into the Cadasta UI (cdn, etc). It will be good to collaborate and set some thoughtful best practices.

[clash99]

@SteadyCadence and @dpalomino - Since it seems like the users we are currently supporting have these fonts locally installed, I suggest we table this for a larger discussion in Madrid (see note above). I recommend we move this from high priority to medium priority. Let me know if you agree.

[alukach]

@clash99 I'm not sure what you mean by pseudo-unicode. My understanding is that all of S'gaw Karen characters should be included in the Myanmar Unicode block, meaning it is valid Unicode. Am I mistaken? Are there more characters outside of that Unicode block that make-up the S'gaw Karen written language?

Are we confident that all S'gaw Karen speaking clients have a Myanmar Unicode font installed and available on their machines? For example, some of the input data was provided in a non-Unicode Karen font. Furthermore, are we certain that all browsers will utilize that Myanmar Unicode font when it encounters characters from the Myanmar Unicode block (I think this is a "yes")? I would recommend that @SteadyCadence ask a few of our S'gaw Karen speaking partners to open the Wikipedia link and to verify that there are no tofu symbols in the block: https://en.wikipedia.org/wiki/Myanmar_(Unicode_block)

[clash99]

@alukach I don't know if this is an issue or not but it seems like one symbol ( ြ) might be a problem, or at least worth verifying that it is not a problem based on what I've read.

Good questions for @SteadyCadence. When I was screen sharing with Katrina last night they were installing fonts on the machines they were using so I'm pretty sure they have everything locally that they need (for this particular partner at least). But yes, we should verify. At this point I think it would be better to install the fonts locally.

[SteadyCadence]

@alukach @clash99 Looking at these tables with Mabu, he says that there are some characters in the Myanmar table that are not used in Karen and then there are some characters in the extension A and extension B that are used in Karen: https://en.wikipedia.org/wiki/S%27gaw_Karen_Script

Ill try to print out the sheet and have him highlight the characters that Karen uses... not sure how much that will help.

[seav]

It should be OK if there are unused characters. What we should worry about is if there are characters they use that aren't in Unicode.

[SteadyCadence]

I lied. Apparently, there are characters missing in that Wikipedia page. This file has all of the alphabet and the corresponding unicode character:
Payap unicode complain.pdf

[alukach]

@SteadyCadence thanks for the update. I take it that all of the characters rendered properly when you opened the S'gaw Karen webpage? Even the extension? As @seav said, the issue isn't unused characters. Right now we just want to verify that our target users have that font installed on their machines and that it plays well with their browsers and Unicode text.

[clash99]

I've read the emails @SteadyCadence and I've created a branch called font/padauk that I believe will work. The KNU font mentioned wasn't available in the link and this Paudauk font was recommended in the second email. I think it will work based off my before/after screenshots below. Can you verify this is the readable? If so, I will make a PR.

BEFORE

AFTER

[SteadyCadence]

Thank you! I'll try to get someone to take a peek. I know I will go into the office on Wed, so I can certainly get it done by then.

I appreciate your hard work on this! I think it will make the team super happy, esp because they have had some doubts.

[dpalomino]

First, thanks to all @clash99 @alukach @seav @amplifi and @SteadyCadence !!! This is s a very very tricky, difficult but important thing.

For Q1, I agree we should not support non-unicode data. Karen is pseudo-unicode meaning that it is a combination of unicode and not so do we throw it all out? If its all converted to unicode then I think we are good there.

I agree, indeed as far as I understood is our only option...

For Q2, since the users at-hand have the fonts installed locally we don't have to do anything with the "visual images that typically represent the parts of language" display issue on the platform right now. :)

If we can confirm (@SteadyCadence?) that now all the team have the correct fonts installed and there are no problems working with the data then I think it should be ok. If this is not working for all the team, I think we should make an exemption an try to include it via loading font-face the proper font only for this case, given the importance of the partner.

I think we need to set aside some time to discuss this in Madrid. I think we should put this into a larger discussion regarding how we handle all files that feed into the Cadasta UI (cdn, etc). It will be good to collaborate and set some thoughtful best practices.

Totally agree, and thanks for bringing this into the conversation!

[SteadyCadence]

Disregard what I wrote earlier if you received it in an email. I got very confused talking with Lawplah-- he is confusing sometimes.

I think we can successfully read KNU with the font/KNU branch. The font works well in Chrome and Safari. I couldnt test on IE yet. (They use Chrome and IE mostly). In Firefox, there are characters missing.

I think we should go ahead and merge the Paudauk and KNU branch asap. Both languages will be used. This gets us one step closer to an ideal readability.

Paudauk is the Burmese language. It is different than Karen.
In the KNU fonts showing in Cadasta, you can see some ticks off of the letters, which represent vowels. On the site, you only see one tick but in Karen that would be two ticks. So there are some small errors but this is a GOOD first step. Also they would definitely still need to install the KNU Unicode keyboard.

Overall, let's merge the branches. It's our best option so far!

[dpalomino]

@SteadyCadence, we need to check if we need this landed urgently or we can wait until having a broader solution. Can you confirm if all the partner's team can see properly their data just having the local fonts installed?

[clash99]

Reference: http://scripts.sil.org/cms/scripts/page.php?site_id=projects&item_id=graphite_fontdemo

[SteadyCadence]

The font is looking good on staging. Its working!!

Request: We need the KNU and Paudauk font to be LARGER. Like size 16 or more. It's super hard to read.

[clash99]

@SteadyCadence Since we aren't using the lang tag to change the font of the whole platform (just the project data fields), there isn't a quick solution to this. I will look into this more but it will take more time.

[wonderchook]

@SteadyCadence for the time being are you having them enlarge the text in the browser as a work around?

[SteadyCadence]

@wonderchook. They are zooming in, which is not an ideal UI experience, since they cannot view the map or many questions and answers at a time. But it is a workaround.

On Tue, Jun 13, 2017 at 7:16 AM Kate Chapman ***@***.***> wrote: @SteadyCadence <https://github.com/steadycadence> for the time being are you having them enlarge the text in the browser as a work around? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1554 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYKfFiXtobQmV1ThsRolaNyyjrMxLCmuks5sDpnFgaJpZM4NtI-6> .

-- [image: Cadasta Logo] [image: Grey Divider] Katrina Engelsted Field Program Support Technologist + 1 774 239 8699 Skype: katrina.engelsted | Cadasta.org <http://cadasta.org/> <https://twitter.com/CadastaOrg> <https://www.facebook.com/CadastaOrg/> <https://www.linkedin.com/company/6417003>Subscribe to our youTube channel <https://www.youtube.com/channel/UC_Q7Ty-YCSC4RY6g6sGQocg> to get the latest documentation!