Support for RTL and bidirectional documents in OpTeX #132
Replies: 12 comments 36 replies
-
|
It should be very interesting to implement support for bi-directional documents for OpTeX. I haven't need more yet than to apply |
Beta Was this translation helpful? Give feedback.
-
|
Indeed section 3.3 is not very detailed, nor close to be complete. I found a nice explanation about Both Since \_pagerightoffset =0pt
\fontfam [newcm]
\textdir TRT\pardir TRT\pagedir TRT\bodydir TRT
\margins/1 a4 (7,7,,,)cm
שלום
\byeYou can see how changing the value of the left margin or right margin doesn't affect the output. It can be fixed in the following manner \_pagerightoffset =0pt
\_def\margins/#1 #2 (#3,#4,#5,#6)#7 {\_def\_tmp{#7}%
\_ifx\_tmp\_empty
\_opwarning{\_string\_margins: missing unit, mm inserted}\_def\_tmp{mm}\_fi
\_setpagedimens #2 % setting \_pgwidth, \_pgheight
\_ifdim\_pgwidth=0pt \_else
\_hoffset=0pt \_voffset=0pt \_pagerightoffset =0pt
\_if$#3$\_if$#4$\_hoffset =\_dimexpr (\_pgwidth -\_hsize)/2 \_relax
\_pagerightoffset =\_dimexpr \_hoffset -1in \_relax
\_else \_hoffset =\_dimexpr \_pgwidth -\_hsize - #4\_tmp \_relax % only right margin
\_pagerightoffset =\_dimexpr \_hoffset -1in \_relax
\_fi
\_else \_if$#4$\_hoffset = #3\_tmp \_relax % only left margin
\_pagerightoffset =\_dimexpr \_hoffset -1in \_relax
\_else \_hsize =\_dimexpr \_pgwidth - #3\_tmp - #4\_tmp \_relax % left+right margin
\_hoffset = #3\_tmp \_relax
\_pagerightoffset =\_dimexpr \_hoffset -1in \_relax
\_xhsize =\_hsize \_setxhsize % \_xhsize used by \output routine
\_fi\_fi
\_if$#5$\_if$#6$\_voffset =\_dimexpr (\_pgheight -\_vsize)/2 \_relax
\_else \_voffset =\_dimexpr \_pgheight -\_vsize - #6\_tmp \_relax % only bottom margin
\_fi
\_else \_if$#6$\_voffset = #5\_tmp \_relax % only top margin
\_else \_vsize=\_dimexpr \_pgheight - #5\_tmp - #6\_tmp \_relax % top+bottom margin
\_voffset = #5\_tmp \_relax
\_fi\_fi
\_if 1#1\_shiftoffset=0pt \_def\_prepoffsets{}\_else \_if 2#1% double-page layout
\_shiftoffset = \_dimexpr \_pgwidth -\_hsize -2\_hoffset \_relax
\_def\_prepoffsets{\_ifodd\_pageno \_else \_advance\_hoffset \_shiftoffset \_fi}%
\_else \_opwarning{use \_string\_margins/1 or \_string\_margins/2}%
\_fi\_fi\_fi
}
\fontfam [newcm]
\textdir TRT\pardir TRT\pagedir TRT\bodydir TRT
\margins/1 a4 (7,,,,)cm
שלום
\byeNote that I did not address double page layout and that the meaning of the arguments of |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I did read from section 3.3 that |
Beta Was this translation helpful? Give feedback.
-
|
As always, why questions are hard to answer, but to be more concrete, my guess is that it was more important when more directions were implemented. LuaTeX implemented the multi-directional model from the Omega and Alpha projects, and originally there were 16 possible directions, not just 4. Maybe it was just easier to separate the offsets to several registers and read an offset register at shipout in correlation to the current As for why there is a need for both I don't think there is any documentation for this topic, maybe to best source for answers would be LuaTeX's mailing list. Probably Javier Bezos has some better understanding than me on the subject, as he implemented a package meant to be used with Aleph (If I remember correctly this package is called |
Beta Was this translation helpful? Give feedback.
-
|
I tried to do some changes in OpTeX macros, see my last commit 56fd378. |
Beta Was this translation helpful? Give feedback.
-
|
I've tried the new commit on a document I'm working on and the output looks great. I don't think that the information you have added to the documentation is entirely true. Consider the following document: \textdir TRT
Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph Just some text to fill the paragraph.
\byeAs you can see, the lines don't run outside the paper. What you are describing is more similar to a situation where Slide 9 of the presentation I linked previously contain a demonstration what happens when Note that there is still some problem in the output routine, but I could not trace the source of it. When
which from the reason describe earlier, causing the body to be positioned outside of the page. Here is a minimal example: If I uncomment the first line, the output is correct (except the are no colors, of course), so I guess Currently I'm adding to my document
so that I could use colors, I didn't encounter any problems with this hack so far, but it does not really explains what happens. The other solution I suggested for that, using caused some problems, for example when I had overfull |
Beta Was this translation helpful? Give feedback.
-
|
Honestly I have very little idea about the primitives used. But what I gather is that some text direction or whatever property is set on boxes when they are created and just like with attributes this property is taken as the current value at box creation time. But maybe I just didn't test properly and I am not even going to pretend to try to understand what the text direction etc. properties mean or do. |
Beta Was this translation helpful? Give feedback.
-
This shouldn't be a problem. The property is on the box, i.e. on the thing the pointer points to, not on the pointer itself. So the fact that the copy is a pointer copy ("a shallow copy") means that the page direction property or whatever is "copied" as well. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, I can confirm that your solution seems to work for a 20 page document I'm currently working on. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW some pointers to source code:
Now why it behaves differently when run from Lua and when run from TeX I have node idea and also no time to investigate further. (I am testing with TeX Live 2022 version of LuaTeX, if you can reproduce the issue with newest version and with |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the pointers, I'll try to test some cases and send a question to LuaTeX's mailing list. |
Beta Was this translation helpful? Give feedback.
-
|
When I use your example then the lines run outside the paper in my PDF output. First of all, we need exact documentation of primitives |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, Maybe I wasn't clear, but I meant that this example should be compiled with plain LuaTeX, not OpTeX. If you compiled this example with OpTeX then the lines are outside of the paper because of the problem in the output routine I mentioned, since |
Beta Was this translation helpful? Give feedback.
-
|
In plain LuaTeX you can get the same output with That is essentially what causing the problem in the output routine in OpTeX, somehow |
Beta Was this translation helpful? Give feedback.
-
|
I tried a second attempt to solve this issue. 1056c12 |
Beta Was this translation helpful? Give feedback.
-
This approach introduce a new problem. When there is an overfull \_def \_optexshipout #1{%
\_setbox0=\_completepage
\_preshipout0\_box0
\_pagedirection=0 \boxdirection0=0 \_shipout\_box0
}
\fontfam [newcm]
\textdir TRT\pardir TRT\bodydir TRT
\hbox{aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}
טקסט הנועד למלא את הפסקה טקסט הנועד למלא את הפסקה טקסט הנועד למלא את הפסקה טקסט הנועד למלא את הפסקה טקסט הנועד למלא את הפסקה טקסט הנועד למלא את הפסקה.
\bye |
Beta Was this translation helpful? Give feedback.
-
|
OK, that's a strong argument why use the |
Beta Was this translation helpful? Give feedback.
-
|
I returned back the |
Beta Was this translation helpful? Give feedback.
-
I don't know of any RTL language that have hyphenation so it is hard to tell about that specifically. TeX cares about directions at line breaking stage, especially if
Don't think so, but if you will take a look at some of the Here is a minimal LaTeX file that use \documentclass{article}
\usepackage{fontspec}
\setmainfont{NewCM10-Regular}
\usepackage{luatexbase}
\directlua{
dofile(kpse.find_file("babel-bidi-basic.lua"))
function Babel.pre_otfload_v(head)
if Babel.numbers and Babel.digits_mapped then
head = Babel.numbers(head)
end
if Babel.bidi_enabled then
head = Babel.bidi(head, false, dir)
end
return head
end
function Babel.pre_otfload_h(head, gc, sz, pt, dir)
if Babel.numbers and Babel.digits_mapped then
head = Babel.numbers(head)
end
if Babel.bidi_enabled then
head = Babel.bidi(head, false, dir)
end
return head
end
luatexbase.add_to_callback('pre_linebreak_filter',
Babel.pre_otfload_v,
'Babel.pre_otfload_v',
luatexbase.priority_in_callback('pre_linebreak_filter',
'luaotfload.node_processor') or nil)
luatexbase.add_to_callback('hpack_filter',
Babel.pre_otfload_h,
'Babel.pre_otfload_h',
luatexbase.priority_in_callback('hpack_filter',
'luaotfload.node_processor') or nil)
}
\begin{document}
Hello World שלום עולם Hello World \hbox{שלום עולם}
\end{document}Of course you will need the file BTW, How should I implement a similar thing in OpTeX? |
Beta Was this translation helpful? Give feedback.
-
Thanks, works great. |
Beta Was this translation helpful? Give feedback.
-
|
Everything relevant I could find: https://ctan.org/tex-archive/obsolete/systems/omega (some information in the README, something maybe also in the http://www.pragma-ade.nl/general/manuals/aleph.pdf http://www.pragma-ade.nl/general/manuals/followingup.pdf (Chapter 6: Directions) The last one is about changes in LuaMetaTeX, but mentions some history and workings of LuaTeX / Omega. |
Beta Was this translation helpful? Give feedback.
-
|
Actually, reading LuaMetTex manual it seems they dropped |
Beta Was this translation helpful? Give feedback.
-
|
If I understand well, Hebrew doesn't use hyphenation. OK, I added Hebrew to the lang-decl.opm with dummy hyphenation patterns in my last commit. It would be good to create the |
Beta Was this translation helpful? Give feedback.
-
|
My problem is that I don't know what result is desired in Hebrew document. Suppose, we have a section If I knew what is required then I can solve it at macro level. Moreover, the mentioned directlua doesn't work in my computer. When I use your document I get the error message |
Beta Was this translation helpful? Give feedback.
-
It should be the first option:
Yes, there seems to be a problem with this approach and the |
Beta Was this translation helpful? Give feedback.
-
|
OK, I introduced |
Beta Was this translation helpful? Give feedback.
-
I turns out that it was just the pdf reader. Attached are two files with lua code that should work. they are in .txt format because github won't allow .lua attachments. Here is an example file, note that the The idea of this module is to implement unicode bidi algorithm so that short left-to-right text could be written inside right-to-left paragraph, and vice versa. Note how it displays the section header correctly and the english text inside a hebrew paragraph without the need of manual |
Beta Was this translation helpful? Give feedback.
-
Thanks, section and page numbers look great. Should |
Beta Was this translation helpful? Give feedback.
-
|
IMHO, the RL printing using OpTeX macros is solved after my last commit with Usage of luabidi is another issue. I can imagine that a |
Beta Was this translation helpful? Give feedback.
-
Yes, I'm sincerely apologize it took so long, I was certain I've provided the example file already. In any case, I've made a pull request with an example file, I filled all the necessary data for hebrew in
You are probably right. I've had some success with Khaled Hosny's code, but it has a lot of bugs. It seems that implementing the bidi algorithm in pure lua is not an easy task. Unrelated to that, I think that OpTeX should assign two registers called |
Beta Was this translation helpful? Give feedback.
-
Hello OpTeX users,
I've started to take an interest in the OpTeX format, and as a native Hebrew speaker one of the first things I've wondered after reading the documentation and the source is if OpTeX can be used to typeset RTL documents.
I decided to start this discussion to share the problems I've encountered and possible solutions to them (luckily OpTeX's source is pretty elegant and easy to patch for such a task). I started to work on a package, I'll share it when it will be ready.
As a side note, I was really surprised how easy it was to get a working RTL document, as it is not a trivial thing and I found no indication that OpTeX was meant for that. It seems that allot of things that usually break with latex works here "out of the box".
Problems
The first thing I've observed is that the text block is not centered when
\pagedirection=1. Since OpTeX sets\_pdfhoriginto get rid of the annoying1infrom\_hoffset, it means that that the default\_pagerightoffset=1inis not desired, and should be changed to\_pagerightoffset=0pt.The
\marginsmacro does not take into account\_pagerightoffset. It could be easily solved by adding after each assignment of\_hoffset, assignment of\_pagerightoffsetto the same length minus1in.It seems that if the page is shipped while
\textdiris not the same as\pagedirit somehow changes\bodydirin the output routine. my suspicion is that it is somehow related to the operation that\_preshipoutdo, since if I remove it it does not happen. I was able to solve it by adding\boxdirection0=\pagedirectionbefore\_shipout\_box0, or by adding\textdirection=\pagedirectionsomewhere before the call of\_preshipout. I believe that the former solution is better, but I'm not fully sure why it happened in the first place, if someone can shed more light on that topic it would be great.Some adjustment to the formatting of the output routine are needed, mainly just adding conditionals to use
\_pagerightoffsetinstead of\_hoffsetfor the calculations when\pagedirection=1.I guess that
\urlshould be patched as well, as normally it should be typeset LTR unrelated to\textdir, but I did not try anything yet.Counters are displayed RL inside RL context, when they should not. fix that by adding conditionals of directionality nodes in the correct places, but there might be a better solution.
Ideas
A nice feature would be to implement unicode bidi algorithm in pure Lua, so that short LR text inside RL paragraph won't need a macro use each time (it will also solve the counters problem). I think that
Babelis doing that whenluatexis used, but the code is a bit entangled and I found it hard to extract the implementation. There is also this repository but it is outdated and not functional asluatexhas changed allot.More directions could be considered, for cjk and Mongolian for example.
Beta Was this translation helpful? Give feedback.
All reactions