JDK-8210970 : MathML editing issues in HTMLEditor

[scientificware]

The purpose of this issue is to make mathematic editing as simple as possible.

This is referenced in Java Bug Database as

• JDK-8210970 : MathML editing issues in HTMLEditor

This is tracked in JBS as

• JDK-8210970 : MathML editing issues in HTMLEditor

Fluent Mathematic Writing.

Situation

Presently, the patch for JDK-8147476 (javafxports/openjdk-jfx Rendering issues with MathML token elements. w3c/mathml#117) allows to display MathML formula in WebView. Consequently we can also work with MathML in HTMLEditor, the MathML rendering is pretty good.

Justification

• A common user can not easily insert a mathematic formula in a HTMLEditor.
  To insert this simple quadratic formula :
  
  You need to insert the following code :

<math display="block"> 
   <mrow> 
      <mi>x</mi> 
      <mo>=</mo> 
      <mfrac> 
         <mrow> 
            <mo>−</mo> 
            <mi>b </mi> 
            <mo>±</mo> 
            <msqrt> 
               <mrow> 
                  <msup> 
                     <mi>b</mi> 
                     <mn>2</mn> 
                  </msup> 
                  <mo>−</mo> 
                  <mn>4</mn> 
                  <mi>a</mi> 
                  <mi>c</mi> 
               </mrow> 
            </msqrt> 
         </mrow> 
         <mrow> 
            <mn>2</mn> 
            <mi>a</mi> 
         </mrow> 
      </mfrac> 
   </mrow>
</math>

• trying to modify a formula will occurate some trouble in your document. Modifiy the previous formula, by removing 2a from the fraction ! ...

Conclusions and Roadmap.

HTMLEditor and underlying WebView i.e. embedded WebKit implementation are clearly not ready yet for formula editing. It still needs some work to solve issues.

• First : Identify all issues.
• Secondly : Understand where are problems.
• Third : Propose a solution.

Details

• Old known bug All bug I encountered and who are already know, will be reported here.
  • JDK-8170082 JavaFX clipboard erroneously supposes that text/??? are UTF-8.
  • JDK-8133337 [Linux] Pasting HTML from Firefox does not work.
• New bug not related to MathML All new encountered bug independant from MathML use will be reported here.
• MathML related bug Some bugs couldn't be detected before JDK-8147476. They will be reported here. Since I work with HTMLEditor, I noticed that :
  • HTMLEditor doesn't create a new line when one presses Enter.
    • Done. Shouldn't affect JavaFX 11 :
      • It works with JavaFX 8u171 and JavaFX 11-internal+0-2018-07-02-110935
      • but not with JavaFX 9.0.4, JavaFX 10.0.1. Post a message at [email protected] because this affects HTMLEditor without the patch for JDK-8147476 too.
  • Deferred work. MathML content is not updated after an insertion. This only occurs when MathML Token are in a table like in the Mozilla MathML Torture Test.
  • Work In Progress. Understood. Cursor disappears in <mi> token when the identifier name has only one character. If the name of the identifier is longer than 1 character, the cursor appears normaly. Note : in W3C <mi> description, The default font style should (typically) be "normal" (non-slanted) unless the content is a single character, in which case it should be "italic". This not occurs in <mn> and <mo>. It's a WebCore issue, then transfered in my WebKit Repository Issue#1.
  • Work In Progress. Understood. In <mo>, the caret disappears too but only with the minus sign. Related to Hyphen. It's also a WebCore issue, then transfered in my WebKit Repository Issue#2.
  • Work In Progress. Understood. The caret has the same size than the character (but not the font size, see JavaFX and MathML. javafxports/openjdk-jfx#71). This problem is specific to the MathML support. With normal text, the caret height is the font height. Solution : Anyway the implemented logic is not correct : the cursor size has to be equal to the parent <row> or <math> height (that's why, it is important to correct MathML code when loads a file or paste, see below about these <row> add and remove corrections). In the following example, move from the left to the right, you can see that the caret height is good outside the formula and too small inside :

<p>Just a fraction example to test the Caret moves : 
   <math>
      <mrow>
         <mi>A</mi>
         <mo>=</mo>
         <mfrac>
            <mrow><mn>2</mn></mrow>
            <mrow><mn>8</mn></mrow>
         </mfrac>
         <mo>+</mo>
         <mfrac>
            <mrow>
               <mn>12</mn>
               <mo>-</mo>
               <mfrac>
                  <mrow><mn>567</mn></mrow>
                  <mrow><mn>17</mn></mrow>
               </mfrac>
            </mrow>
           <mrow><mn>37858727</mn></mrow>
         </mfrac>
      </mrow>
   </math>
. But you can do the calculus, if you want.
</p>

The caret height should be the same on the two pictures. Its height should be its height just after the word 'test'. That's the usual behavior. So, its size before '+' is wrong. May be a WebCore problem too so transfered to Issue#4

• no effect in <mi> token.
  
  • Deferred work. Understood. In <mi> with only one character, css attributes are not apply. It's a WebCore issue too, then transfered in my WebKit Repository Issue#3.
• Correct selection logic. Some parts of the formula disappear, Selection should be only one rectangle not many rectangles like on the picture :
  
  See the same formula with my application ScientificPad, only one rectangle selection :
  
  • Deferred work.
• Enhancement Mathematic edition needs to be improved (simply using keyboard)
  • making easiest the mathematic character insertions :
    • Done. Enable ∞, ∈, × , ⩽, α, ..., δ, ... ω, ..., Δ, ... simply using keyboard,
      
  • making easiest the mathematic formula insertions (simply using keyboard).
    • Firstly, I have to ask what is Igalia’s project about mathematic editing.
      • Done. No Igalia's project about that. According to their contracts, they concentrate their work on standard support rendering only. So I can go ahead.
    • In which part of the code, should I fix that ? In native WebCore, in native OpenJFX area or in Java OpenJFX area ?
      • Done. First in Java OpenJFX area.
    • Roadmap :
      • Confort my point of vue
        • Done. I had a short conversation with a member of mathonweb group and F. Wang about that point. That conforts my point of vue.
      • Write an application for testing.
        • Done. FXMath and FXMessages series (X1, X4).
      • Write the logic
        • Correct caret deplacements. Not a bug, but without other specification caret uses default the implementation. For example : in a fraction, when the caret is at the end of the numerator, if we press right arrow, the caret goes to the beginning of the denominator. But its right position should be rather immediately after the fraction.
          • Deferred work.
        • Use materials I developed in my own products and follow W3C recommendations about editing. Impove the minimal parser. Work In Progress.

          • Port the language register selection method : daily french and english, AsciiMath, OpenOffice Math and LaTex languages.

            • Work in progress. Presently, only daily French, German, Spanish, English languages are supported.

          • Port the Fluent Mathematic Writing logic.

            • Work In Progress.
              

Related to HTMLEditor and MathML w3c/mathml#118

[scientificware]

Resources

• http://cr.openjdk.java.net/~kcr/8201563/webrev/
• http://cr.openjdk.java.net/~kcr/8201563/webrev/raw_files/new/modules/javafx.web/src/main/native/Source/WebCore/bindings/java/dom3/

[scientificware]

New issue, new tool to debug.

• JavaFXMathML was used for JDK-8147476 but for HTMLEditor and MathML. Add more global attributes from HTML w3c/mathml#118, it's important to see DOM skeleton.

• So FXMath is borned !

  • relies on JavaFXMathML,
  • implements Fluent Mathematic Style Writing, FXMath adopts the common user point of vue who doesn't matter with MathML. He could be an elementary, a secondary student and of course a mathematician. So FXMath have to deal with these quite different language registers.
  • and includes DOM content displays.

[scientificware]

<mi>, <mo>, ... MathML token insert at the beginning or the end, yes. In this case, caret position is rigth, no more work just reached the first parent <mrow>.
Otherwise, no !

<mn> only digit and decimal separator enter, beginning and ending excepted.
<mfrac> if loads

<mrow>
   <mo> ( </mo>
   <mfrac>
      <mi> a </mi>
      <mi> b </mi>
   </mfrac>
   <mo> ) </mo>
</mrow>

Has to use :

<mrow>
   <mo> ( </mo>
   <mfrac>
      <mrow><mi> a </mi></mrow>
      <mrow><mi> b </mi></mrow>
   </mfrac>
   <mo> ) </mo>
</mrow>

Saves simplyfied content where <mrow> could be remove, if it contains only one element.

[scientificware]

Using Fluent Mathematic Style Writing, all mathematic editing uses only these keyboard keys.

• with IOS
  
• with Android
  
• Some progress, can mix MathML Token but not perfect as shown in the MathML content Window.
  

[scientificware]

User Input of Mathematics #1611

Website for the MathML Association.

GitLab Reprository for the MathML Association.

MathQuill

Xtext

• https://alain-bernard.developpez.com/tutoriels/eclipse/creation-grammaire-xtext/
• https://vimeo.com/138873446

[scientificware]

FXMessage is FXMath on mobile using Gluon Mobile

Writing a new application for mobile testers. The goal is obtain daily choice validations from selectionned testers. So have to use a common communication application like chat.

• In progress
  
  

[scientificware]

Show formula given in LaTeX or its MathML-conversion in javafx-label
<div contenteditable="true">
<p contenteditable="true">
...

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.stage.Stage;
import javafx.scene.layout.VBox;
import javafx.scene.web.WebView;

public class TestRM extends Application {

    private Stage window;
    private Scene scene;


    private WebView webView = new WebView();

    String javaversion = System.getProperty("java.runtime.version");
    String javafxversion = System.getProperty("javafx.runtime.version");
    String osinfo = System.getProperty("os.name");
    String osarch = System.getProperty("os.arch");
    String agent = webView.getEngine().getUserAgent();

    private String string2 = new String("""
        <html>
            <head>
                <title>TestRM</title>
            </head>
            <body contenteditable=\"true\">
                <p>
                    <math xmlns=\"http://www.w3.or/1998/Math/MathML\" display=\"block\">
                        <semantics>
                            <mfrac>
                                <mrow>
                                    <mo>-</mo>
                                    <mi>b</mi>
                                    <mo>&pm;</mo>
                                    <msqrt>
                                        <mrow>
                                            <msup>
                                                <mi>b</mi>
                                                <mn>2</mn>
                                            </msup>
                                            <mo>-</mo>
                                            <mn>4</mn>
                                            <mi>a</mi>
                                            <mi>c</mi>
                                        </mrow>
                                    </msqrt>
                                </mrow>
                                <mrow>
                                    <mn>2</mn>
                                    <mi>a</mi>
                                </mrow>
                            </mfrac>
                            <annotation encoding=\"SnuggleTeX\">$$ \\frac{-b \\pm \\sqrt{b^2-4ac}}{2a} $$</annotation>
                        </semantics>
                    </math>
                </p>
                <p>Version de Java . : %s</p>
                <p>Version de JavaFX : %s</p>
                <p>OS .............. : %s</p>
                <p>CPU ............. : %s</p>
                <p>User Agent ...... : %s</p>
            </body>
        </html>
        """.formatted(javaversion, javafxversion, osinfo, osarch, agent);

    VBox vbox = new VBox(webView);

    public static void main(String[] args) {
        launch(args);
    }

    public void start(Stage primaryStage) {
        webView.getEngine().loadContent(string2);
        window = primaryStage;
        scene = new Scene(vbox, 400, 300);
        window.setScene(scene);
        window.show();
    }
}

[scientificware]

Presently, the patch for JDK-8147476 solves MathML display issues in WebView. Consequently, it solves MathML display in HTMLEditor too.

The MathML rendering is pretty good and equivalent to MathML Rendering with Safari.
I think that JDK-8089878, which describe only rendering issues, is also solved.

Set HTMLEditor content with the following code displays the quadratic formula as expected in JDK-8089878.

<math display="block"> 
   <mrow> 
      <mi>x</mi> 
      <mo>=</mo> 
      <mfrac> 
         <mrow> 
            <mo>−</mo> 
            <mi>b </mi> 
            <mo>±</mo> 
            <msqrt> 
               <mrow> 
                  <msup> 
                     <mi>b</mi> 
                     <mn>2</mn> 
                  </msup> 
                  <mo>−</mo> 
                  <mn>4</mn> 
                  <mi>a</mi> 
                  <mi>c</mi> 
               </mrow> 
            </msqrt> 
         </mrow> 
         <mrow> 
            <mn>2</mn> 
            <mi>a</mi> 
         </mrow> 
      </mfrac> 
   </mrow>
</math>

The remaining problems are related to editing capabilities.

Bugs :
-Trying to modify a formula will occurate some trouble in your document. For example, try to modifiy the previous formula, by removing 2a from the fraction, this will break your formula.

• MathML content is not updated after an insertion. This only occurs when MathML Token are in a table like in the Mozilla MathML Torture Test.
• Cursor disappears in token when the identifier name has only one character. If the name of the identifier is longer than 1 character, the cursor appears normaly. This not occurs in and .
• In , cursor disappears too.
• Bad selection logic. Some parts of the formula disappear and selection should be only one rectangle not many rectangles.

Enhancements : A common user can not easily and directly use MathML to insert a mathematic formula in HTMLEditor.

• MathML language is too verbose, an interface like AsciiMath for editing would be appreciate.
• Correct caret deplacements. Without other specifications caret uses the default implementation. For example : in a fraction, when the caret is at the end of the numerator, if we press right arrow, the caret goes to the beginning of the denominator. But its right position should be rather immediately after the fraction.

Each test could be made with the previous MathML code.

[scientificware]

Cursor deplacements : EditorClientJava

[scientificware]

Tutoriel JavaFX Webview et Webengine

[scientificware]

https://css-tricks.com/

[scientificware]

See exchanges with ebraminio and khaledhosny.
SVG font: extract SVG per character #1192

harfbuzz/harfbuzz#2016 (comment)

[scientificware]

FXMessage is FXMath on mobile using WebView

Writing a new application for mobile testers. The goal is obtain daily choice validations from selectionned testers. So have to use a common communication application like chat.

• In progress
  • Navigation menu works
  • Math editing works
  • Message sending works

[scientificware]

Bug id ? - Height of cursor should be same as row-height, not height of text.

• WebKit source code :
  trac.webkit.org

• Webcore Rendering (WebKit Blog) :

  1. The Basics
  2. Blocks and Inlines
  3. Layout Basics
  4. Absolute/Fixed and Relative Positioning
  5. Floats

• WebKit bug database :
  bugs.webkit.org
  Bug 13820 - Height of cursor should be same as height of text, not line-height

• Rendering MathML (Folder):
  RenderMathMLBlock extends RenderBlock

  • RenderMathMLToken.cpp extends RenderMathMLBlock
    • RenderMathMLOperator extends RenderMathMLToken
      • RenderMathMLFencedOperator extends RenderMathMLOperator
  • RenderMathMLRow extends RenderMathMLBlock
    • RenderMathMLFenced extends RenderMathMLRow
    • RenderMathMLMath extends RenderMathMLRow
    • RenderMathMLMenclose extends RenderMathMLRow
    • RenderMathMLPadded extends RenderMathMLRow
    • RenderMathMLRoot extends RenderMathMLRow
  • RenderMathMLFraction extendsMathMLBlock
  • RenderMathMLScripts extendsMathMLBlock
    • RenderMathMLUnderOver extends RenderMathMLScripts
  • RenderMathMLSpace extendsMathMLBlock

• Rendering (Folder) :
  RenderBlock extends RenderBox localCaretRect
  RenderBox extends RenderBoxModelObject localCaretRect
  RenderText Base implementation localCaretRect
  RootInlineBox extends InlineFlowBox computeCaretRect
  InlineFlowBox Base implementation
  TextLayout.java getCaretShape
  RenderInline extends RenderBoxModelObject0 LocalCaretRect
  RenderBoxModelObject extends RenderLayerModelObject
  RenderLayerModelObject extends RenderElement
  RenderElement Base implementation
  RenderObject Base implementation localCaretRect
  Internals absoluteCaretBounds
  Editing.cpp localCaretRectInRendererForCaretPainting

• Tokens and elements :

Token	Element
annotation	MathMLAnnotationElement
annotation-xml	MathMLAnnotationElement
maction	MathMLSelectElement
math
mfrac	MathMLFractionElement
mfenced	MathMLRowElement
msubsup	MathMLScriptsElement
merror	MathMLRowElement
mpadded	MathMLPaddedElement
mphantom	MathMLRowElement
mrow	MathMLRowElement
mstyle	MathMLRowElement
mover	MathMLUnderOverElement
munder	MathMLUnderOverElement
munderover	MathMLUnderOverElement
msqrt	MathMLRootElement
mroot	MathMLRootElement
mi	MathMLTokenElement
mn	MathMLTokenElement
mo	MathMLOperatorElement
mtext	MathMLTokenElement
ms	MathMLTokenElement
mspace	MathMLSpaceElement
msub	MathMLScriptsElement
msup	MathMLScriptsElement
mtable	MathMLPresentationElement
mtr	MathMLPresentationElement
mtd	MathMLPresentationElement
mmultiscripts	MathMLScriptsElement
mprescripts	MathMLPresentationElement
menclose	MathMLMencloseElement
none	MathMLPresentationElement
semantics	MathMLSelectElement
maligngroup	MathMLPresentationElement
malignmark	MathMLPresentationElement
mglyph	MathMLPresentationElement
mlabeledtr	MathMLPresentationElement
mlongdiv	MathMLPresentationElement
mscarries	MathMLPresentationElement
mscarry	MathMLPresentationElement
msgroup	MathMLPresentationElement
msline	MathMLPresentationElement
msrow	MathMLPresentationElement

[scientificware]

Selection with the same document. Apple Touch Safari 9.

[scientificware]

<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<style type="text/css">

</style>
<body>
<h1>Fraction</h1>
<p>This is an empty fraction.
<math>
<mfrac>
<mrow></mrow>
<mrow></mrow>
</mfrac>
</math>
</p>
<h1>Fraction</h1>
<p>This is a normal fraction :
<math>
<mfrac>
<mrow><mn>7</mn></mrow>
<mrow><mn>3</mn></mrow>
</mfrac>
</math>
</p>
</body>

[scientificware]

<!DOCTYPE html>
<html>
   <head>
      <title>Page Title</title>
   </head>
   <body>
      <p>
         <math> 
            <mi>A</mi>
            <mo>=</mo>
            <mi>y</mi>
            <mo>&times</mo>
            <mi>sin</mi>
            <mfenced>
               <mrow>
                  <mi mathvariant="monospace">x</mi>
                  <mo>&times</mo>
                  <mi mathvariant="bold">&pi;</mi>
               </mrow>
             </mfenced>
         </math>
      </p>
      <h1>This is a Heading</h1>
      <p>This is a paragraph.</p>
   </body>
</html>

[scientificware]

Enchaînement des appels de méthodes :

RenderBlock extends RenderBox and defines :

• paint method (protected)

  • calls paintObject method

• paintObject method (protected)

  • paints background, borders, ... with RenderBox::paintBoxDecorations method (public)
  • paints contents with paintContents method
  • paints selection with paintSelection method
  • paints floats with paintFloats method
  • paints outline with paintOutline method
  • paints continuation outlines
  • paints caret with paintCaret method

• paintContents method (private)

  • paints child with paintInlineChildren or paintChildren methods.

According this render implementation none paint should be made after a call to the renderBlock paint method. MathML implementation doesn't respect this and paint or paintChildren method are overided and paint proceed direct painting.

L'écriture mathématique implique que certaines chaines soient en italique par exemple le nom d'une variable. Mais ce n'est pas le cas pour le nom d'une fonction. Pour simplifer, l'implémentation actuelle considère que les variables d'un seul caractère sont bien des variables et elles sont affichées en italique par exemple <mi>i</mi> sera bien affiché i. Mais <mi>in</mi> ne sera pas affiché en italique.

[scientificware]

https://github.com/w3c/mathml :
TeX2MathML converter implementation guidelines :

w3c/mathml-docs#39