New lt-merge command to merge LU's from BEG to END tag

[unhammer]

The reason for this is to be able to use CG (or any tool inside the stream really) to mark which parts of the stream should be shielded from translation, e.g. quotes (but perhaps not all quotes).

The pipeline then becomes something like this:

So lt-merge runs after tagger:

$ echo '^ikke/ikke<adv>$ ^«/«<lquot><MERGE_BEG>$^så/så<adv>$ ^veldig/v<adv>$^»/»<rquot><MERGE_END>$ ^bra/bra<adj>$' | lt-merge

^ikke/ikke<adv>$ ^«så veldig»/«så veldig»<MERGED>$ ^bra/bra<adj>$

If bidix does <i><w/><s n="MERGED"/></i> then this passes through unharmed (similarly generator <i><w/></i><p><l/><r><s n="MERGED"/></r></p>), and we get

^ikkje<adv>/ikkje$ ^«så veldig»<MERGED>/«så veldig»$ ^bra<adj>/bra$

after generation. Then unmerge

$ echo '^ikkje<adv>/ikkje$ ^«så veldig»<MERGED>/«så veldig»$ ^bra<adj>/bra$' |lt-merge --unmerge

^ikkje<adv>/ikkje$ «så veldig» ^bra<adj>/bra$

drops it into the stream before cg-proc genprefs.

The tags are hardcoded, MERGE_BEG, MERGE_END → MERGED. I don't see a reason for making them configurable (unless someone else starts using this and has a use-case), they're only used within the lt-merge tool so that should have no effect on existing pairs.

We need to be able to pass MERGED stuff unchanged through biltrans and generator – this PR adds ANY_CHAR (lsx <w/>) support to lt-proc -b. This is the only change here to existing code, it should have no effect unless you for some reason named your bidix tag ANY_CHAR :)

Effects on people not using this

• ANY_CHAR is now treated specially when using lt-proc -b (just like it is in lsx)
• there is a new binary called lt-merge (should that name be used for something else?)

---

Example language pair usage

apertium/apertium-nno-nob@8ca111d
implements the necessary language pair changes for using lt-merge to protect anything between «» (if the user has requested so with AP_SETVAR / style preferences). In summary:

• an rlx file adds MERGE_BEG / MERGE_END tags
• two lsx files create a simple pass-through entry for bidix/generator
• the mode needs three new entries: merge-quotes.rlx, lt-merge, lt-merge --unmerge
• makefile needs to build the rlx and lt-append the lsx onto bidix/generator

---

Detailed escaping details

Most of the above is simple, but escaping can look a bit messy (this is why we need the --unmerge).

If any of the LU's have word-bound blanks, the [] need escaping:

$ echo '^«/«<lquot><MERGE_BEG>$[[tf:i:a]]^veldig/veldig<adv>$[[/]]^»/»<rquot><MERGE_END>$' | lttoolbox/lt-merge

^«\[\[tf:i:a\]\]veldig\[\[\/\]\]»/«\[\[tf:i:a\]\]veldig\[\[\/\]\]»<MERGED>$

to ensure we have legal stream format.

If any of the forms contain already escaped chars, these now need double-escaping. Why? We need to know the difference between a \[ meaning word-blank or \\[ meaning literal [.

We run an "unmerge" step towards the end of the pipeline, while still outputting Apertium Stream Format, which extracts merged forms and drops one level of escaping.

Example of typical quoted char @:

$ echo '^ikke/ikke<adv>$ ^«/«<lquot><MERGE_BEG>$^til/til<pr>$ ^x\@y.com/x\@y.com<email>$^»/»<rquot><MERGE_END>$ ^da/da<adv>$' | lttoolbox/lt-merge

^ikke/ikke<adv>$ ^«til x\\\@y.com»/«til x\\\@y.com»<MERGED>$ ^da/da<adv>$

If we run lt-merge between analysis and wblank-attach, then after the lt-proc -b generator.bin step we should have e.g.

^ikkje<adv>/ikkje$ ^«til x\\\@y.com»<MERGED>/«til x\\\@y.com»$ ^då<adv>/då$

We extract the merged form:

$ echo '^ikkje<adv>/ikkje$ ^«til x\\\@y.com»<MERGED>/«til x\\\@y.com»$ ^då<adv>/då$'|lt-merge --unmerge

^ikkje<adv>/ikkje$ «til x\@y.com» ^då<adv>/då$

Here \\\@ turned into \@ – we removed one layer of quoting, but this is still in the apertium stream so special chars stay quoted, e.g. cg-proc -g leaves it alone:

$ echo '^ikkje<adv>/ikkje$ ^«til x\\\@y.com»<MERGED>/«til x\\\@y.com»$ ^då<adv>/då$'|lt-merge --unmerge |cg-proc -1ng nob-nno.genprefs.rlx.bin

ikkje «til x\@y.com» då

until the final tf-inject removes the last escape.

And the word-blanks only have a single \ so lt-merge --unmerge ensures they take effect:

$ echo '^ikkje<adv>/ikkje$ ^«\[\[tf:i:a\]\]s\\\^å»<MERGED>/«\[\[tf:i:a\]\]s\\\^å»$' | lt-merge --unmerge

^ikkje<adv>/ikkje$ «[[tf:i:a]]s\^å»

which after cg-proc -g becomes

$ echo '^ikkje<adv>/ikkje$ «[[tf:i:a]]s\^å»' |cg-proc -1ng nob-nno.genprefs.rlx.bin

ikkje «[[tf:i:a]]s\^å»

which tf-inject is happy to handle.