Revamp REWORD API: Evaluate value expressions; /case option; binary! support; trailing delimiters

[rebolbot]

Submitted by: rgchris

REWORD has been a design experiment, temporarily in mezzanine form, suggested by Carl in one of his blogs. Ever since the article on REWORD it has been getting a lot of feedback that has helped nail down the final list of design changes that are needed.

Requested changes (requester):

* Binary support, and working tag support (BrianH)
* /case for case-sensitive comparison (BrianH)
* Specifying leading and trailing delimiters with /escape (i.e. for HTML/XML entities) (BrianH)
* Being able to specify none as an /escape, equivalent to "" now (HostileFork)
* Evaluating value expressions directly in a block spec, rather than requiring the spec to be reduced, and /only to turn that off (rgchris, BrianH, prompted by GrahamChiu's RebolBot code)
* Evaluating block values as code like functions, to cut down on function creation overhead, and /only to turn that off (rgchris)
* Special-case treatment of set-word keys as if they were regular words, so block and object specs can be compatible (BrianH)
* Special-case treatment of lit-word keys as if they were regular words, to be compatible with fully reduced specs (BrianH)
* Directly using values specs that already match what we need internally, to save overhead (BrianH)
* Better-phrased doc strings that are short enough to not wrap on the screen (BrianH)

Spec and behavior in the example code. Making this a native is the subject for another day.

>> help reword
USAGE:
        REWORD source values /case /only /escape char /into output

DESCRIPTION:
        Make a string or binary based on a template and substitution values.
        REWORD is a function value.

ARGUMENTS:
        source -- Template series with escape sequences (any-string! binary!)
        values -- Keyword literals and value expressions (map! object! block!)

REFINEMENTS:
        /case -- Characters are case-sensitive
        /only -- Use values as-is, do not reduce the block, insert block values
        /escape -- Choose your own escape char(s) or [begin end] delimiters
                char -- Default "$" (char! any-string! binary! block! none!)
        /into -- Insert into a buffer instead (returns position after insert)
                output -- The buffer series (modified) (any-string! binary!)

>> reword/escape "ba" [a 1 b 2] none
== "21"  ; /escape none like /escape ""

>> reword "$a$A$a" [a 1 A 2]
== "222"  ; case-insensitive, last value wins
>> reword/case "$a$A$a" [a 1 A 2]
== "121"  ; case-sensitive, even with words

>> reword/escape "!bang;" [bang "!"] ["!" ";"]  ; not using a real XML entity because of CC, so pretend they use ! instead
== "!"  ; Leading and trailing delimiters considered
>> reword/escape "!bang;bang;" [bang "!"] ["!" ";"]
== "!bang;"  ; One-pass, continues after the replacement
>> reword/escape "!bang;bang;" [bang "!"] ["!"]
== "!;bang;"  ; No trailing delimiter specified

>> reword "$a" ["a" ["A" "B" "C"]]
== "C"  ; evaluate a block value as code by default
>> reword/only "$a" ["a" ["A" "B" "C"]]
== "ABC"  ; Just insert a block value with /only

>> reword "$a$b$+" [a 1 + 1 b 2 + 2]
== "24$+"  ; Value expressions evaluated, keys treated as literals
>> reword/only "$a$b$+" [a 1 + 1 b 2 + 2]
== "122"   ; With /only, treated as raw values, + keyword defined twice in this case
>> reword "$a ($b)" ["a" does ["AAA"] "b" ()]
== "AAA ($b)"  ; Evaluates function builders and parens too. Note that b was undefined because () evaluates to unset
>> reword/only "$a ($b)" reduce ["a" does ["AAA"] "b" ["B" "B" "B"]]
== "AAA (BBB)"  ; With /only of explicitly reduced spec, functions still called even though blocks are just inserted

>> reword "$a" [a: 1]
== "1"  ; It should be easy to switch from block to object specs
>> reword "$a" context [a: 1]
== "1"  ; ... like this, so we should special-case set-words
>> reword "$a" ['a 1]
== "1"  ; It should be easy to explicitly reduce the same spec
>> reword "$a" reduce ['a 1]
== "1"  ; ... like this, so we should special-case lit-words
>> reword/escape "a :a /a #a" [a 1 :a 2 /a 3 #a 4] none
== "1 2 3 4"  ; But otherwise let word types be distinct

>> reword to-binary "$a$A$a" [a 1 A 2]
== #{010201}  ; binaries supported, note the case-sensitivity, same key rules, values inserted by binary rules
>> tail? reword/into to-binary "$a$A$a" [a 1 A 2] #{}
== true  ; /into option behaves the same
>> head reword/into "$a$A$a" [a 1 A 2] #{}
== #{020202}  ; string templates can insert into binaries, note the case-insensitivity
>> head reword/case/into "$a$A$a" [a 1 A 2] #{}
== #{010201}  ; ... and this time it's case-sensitive
>> head reword/into to-binary "b$a$A$ac" [a 1 A 2] ""
== "b121c"  ; Binary templates can insert into strings, using string insert rules, still case-sensitive

>> reword/escape <a href="http://blah/blah?a!b" /> ["!" "!bang;"] none
== <a href="http://blah/blah?a!bang;b" />  ; Yes, tag source should work too
>> head reword/escape/into {a href="http://blah/blah?a!b" /} ["!" "!bang;"] none to-tag ""
== <a href="http://blah/blah?a!bang;b" />  ; ... and output as well

^{CC - Data [ Version: r3 master Type: Wish Platform: All Category: Mezzanine Reproduce: Always Fixed-in:r3 master ]}

[rebolbot]

Submitted by: BrianH

OK, your requested features are a few of a long list from many people since I wrote that article on REWORD, and they all need implementing. So I did. Rather than make a dozen tickets, I'll just rework this one until it covers everything that has been requested. This will be the end-all be-all API revamp that will let us declare the experiment over and have it be time to make it native.

[rebolbot]

Submitted by: BrianH

Here is the original text of this ticket in case someone is curious:

I'm looking to recode my 'form-date (http://reb4.me/r/form-date) function for R3 and am evaluating the 'reword function to this end. There are two design obstacles that I'd like to address:

Case: my function uses case-dependent keys so as to be compatible with the 'strftime function from other languages. At this time, 'reword is case-equivalent.

Block Arguments: Currently 'reword handles a block as any other value and 'appends the block to its output. I'm currently using a format akin to 'switch where a block is evaluated for its value:

    >> my-reword-like-func "$a $A" ["a" [now/day] "A" [now/month]]
    == "9 3"

Advantages to this is include that blocks are only evaluated as needed, can be bound just once per instance nor requires functions for this behaviour. An /ONLY refinement could restore the ability to insert the values within blocks:

    >> reword/only "$a" ["a" ["a" "b" "c"]]
    == "abc"

Below is a version of the function that implements the desired block/only usage:

reword: func [
    {Substitutes values into a template string, returning a new string.}
    source [any-string!] "Template series (or string with escape sequences)"
    values [map! object! block!] {Pairs of values and replacements (will be called if functions)}
    /escape {Choose your own escape char (no escape for block templates)}
    char [char! any-string!] "Use this escape char (default $)"
    /into {Insert into a buffer instead (returns position after insert)}
    output [any-string!] "The buffer series (modified)"
    /only "Inserts a block as-is, no evaluation of content" ; better description?
    /local vals word a b c d
][
    output: any [
        output
        make source length? source
    ]

    vals: make map! length? values

    either all [block? values not only] [
        while [not tail? values] [
            a: first+ values
            set/any 'b do/next values 'values
            unless string? :a [a: to string! :a]
            unless empty? a [poke vals a unless unset? :b [:b]]
        ]
    ][
        foreach [w v] values [
            unless string? :w [w: to string! :w]
            unless empty? w [poke vals w unless unset? :v [:v]]
        ]
    ]

    word: make block! 2 * length? vals

    foreach w vals [
        word: reduce/into [w '|] word
    ]

    word: head remove back word

    escape: [
        c: word d: (
            ; output: insert insert/part output a b vals/(copy/part c d) :b
            output: insert insert/part output a b case [
                block? c: select vals copy/part c d [
                    either only [c] :c
                ]
                function? :c [apply :c [:b]]
                'else :c
            ]
        )
        a:
    ]

    char: to string! any [char "$"]

    either empty? char [
        parse/all source [
            a: any [b: [escape | skip]]
            to end (output: insert output a)
        ]
    ][
        parse/all source [
            a: any [to char b: char [escape | none]]
            to end (output: insert output a)
        ]
    ]

    either into [output] [head output]
]

print {; reword "$a" ["a" ["A" "B" "C"]]}
probe reword "$a" ["a" ["A" "B" "C"]]
print {; reword "$a ($b)" ["a" does ["AAA"] "b" ()]}
probe reword "$a ($b)" ["a" does ["AAA"] "b" ()]
print {; reword/only "$a" ["a" ["A" "B" "C"]]}
probe reword/only "$a" ["a" ["A" "B" "C"]]
print {; reword/only "$a ($b)" ["a" does ["AAA"] "b" ()]}
probe reword/only "$a ($b)" ["a" does ["AAA"] "b" ()]
print {; reword "$a" context [a: ["A" "B" "C"]]}
probe reword "$a" context [a: ["A" "B" "C"]]
print {; reword/only "$a" ["a" ["A" "B" "C"]]}
probe reword/only "$a" context [a: ["A" "B" "C"]]
print {; reword/only "$a ($b)" context [a: does ["AAA"] b: none]}
probe reword/only "$a ($b)" context [a: does ["AAA"] b: none]

[rebolbot]

Submitted by: BrianH

I adjusted the summary, description, example code, and reworked the tests into something that can be adapted for rebol-tests. I have an implementation which I could package to attach to this ticket, but just as easily could make into a pull request. Tomorrow.

[rebolbot]

Submitted by: rebolek

Why not paren! instead of block! for code?

[rebolbot]

Submitted by: BrianH

Because that would conflict with the inline evaluation, and wouldn't look right. To use paren you'd need to QUOTE it in the spec or use /only, so it just seems awkward. Parens are usually used for immediate evaluation, and what Chris wants is delayed and repeated evaluation (we went over this at length in SO chat, before he made this ticket for me). It's the same reason we don't use parens for code blocks in the DO control functions like IF and LOOP.

REWORD is shaping up to be a simple dialect, but it's one that will normally be called in DO code, so we want to make it fit in with the style of DO code. It's been a bit tricky to get the balance right.

[rebolbot]

Submitted by: rebolek

The style of DO code is that paren! is always evaluated. Block! is block!. Why can't we have it same in REWORD?

[rebolbot]

Submitted by: BrianH

The style in DO code is that paren is always evaluated immediately and that if you want the evaluation deferred you either make a function or use a block which you DO later. That is why IF takes a block rather than a paren. REWORD will support both, parens being evaluated immediately before any replacements start, and blocks deferred to be executed at the point of each replacement, like a parameterless function.

[rebolbot]

Submitted by: BrianH

Fixed some last bugs to work around otherwise special-case R3 behavior for tags and binaries, and optimized it about as well as you can in mezzanine. It's ready for submitting.

Pull request here: rebol/rebol#103

Note that the expression and block evaluation features will make #539 applicable here as well. We'll likely need to go native eventually to solve that issue, though we were planning to do that anyway.