rfc19: propose a standard (alternate) encoding for FLUIDs

[grondo]

As discussed in flux-framework/flux-core#3038, this PR proposes a standard alternate encoding for FLUIDs, most useful in decreasing the space required for JOBID output in tools such as flux jobs. I figured we'd get the idea down in the RFC so we could refer to it during development (assuming there is concensus).

[grondo]

Pushed a commit to update the example FLUID (to one with more human interest.)

Also thinking that FSE is too close to FSD, and maybe a better name for this "standard" FLUID encoding is simply "f58".

[garlick]

Fuzzbunny - Love it.!

[grondo]

Yeah, who is against fuzzy bunnies? I dare you to speak up!

[SteVwonder]

LGTM! :) Thanks @grondo!

[grondo]

Looks like the Wikipedia reference was literally just deleted. I have never seen that before (is un serendipitous a word?)

Maybe drop the reference since the concept of baseN should be clear? There's also a rfc draft, but it is expired.

[grondo]

Pushed a fixup commit to replace the Wikipedia reference with a reference to the Bitcoin wiki description of Base58. Honestly we could also probably drop the reference since we're just representing 64-bit integers in base58, not describing a generic binary encoding.

[garlick]

Looks good! Please set MWP when you're ready.

[grondo]

Unfortunately, once I had an implementation mostly complete, I did run into a problem with Unicode support as @SteVwonder feared. If the current locale is set to a non-UTF-8 locale, and attempt to print a jobid encoded in F58 fails with:

flux-jobs: ERROR: 'ascii' codec can't encode character '\u0192' in position 2: ordinal not in range(128)

sharness explicitly set the locale to C, so this problem hits the sharness tests.

It seems like this is a well known issue such that PEP 538 (coercion to UTF-8 locale) was proposed, but it seems like that PEP only made it into 3.7.

Not sure if we can do a similar thing and coerce locales to a valid UTF-8. This kludge does seem to work:

    # Re-open stdout with utf8 encoding
    sys.stdout = open(sys.stdout.fileno(), mode='w', encoding='utf8')

Only needed in flux-*.py commands that output JOBIDs encoded as F58 strings.

Also seems that PYTHONIOENCODING=utf-8:surrogateescape works.

I guess if these minor changes work around the issue in sharness, we shouldn't have troubles in various locales that might be set on production systems, but also maybe it would be safer to use ascii f as the F85 default prefix.

(Sorry for doubting you earlier @SteVwonder, this unicode thing in Python is indeed a mess!)

[garlick]

I'm fine with the coersion as it seems like that's where python is heading. The unicode output looks cool and we do have a fallback plan if problems arise.

[grondo]

In the implementation proposed in flux-framework/flux-core#3045, a new fluid_parse() function was introduced that allows decoding a FLUID/JOBID string from any supported encoding using the following rules:

• if the string contains a ., decode as dothex
• if the string contains a -, decode as words (mnemonic)
• if the string starts with ƒ or f, decode as F58
• if the string starts with 0x decode as hexidecimal
• O/w, decode as decimal

E.g.:

$ flux job id ƒLUX  
65512
$ flux job id 0000.0000.0000.ffe8
65512
$ flux job id modular-archive-academy--academy-academy-academy
65512

Should these rules be made explicit in this RFC? Or is that overkill? Just throwing it out there.

[garlick]

Should these rules be made explicit in this RFC? Or is that overkill? Just throwing it out there.

Yeah, that IMHO would be useful to document.

[grondo]

I've pushed an update here that reworks the "Representation" section to include descriptions of all supported FLUID encodings, as well as a set of "rules" for decoding FLUID strings in supported representations.

If this looks ok (might need some wordsmithing), then I can squash all the commits together.

[garlick]

Nice! That works for me. Please squash and set MWP when you're ready.

[grondo]

Cast in stone!