User Input of Mathematics

[AdamSobieski]

I'd like to share some ideas from the Educational Exercises and Activities Community Group. The ideas pertain to the user input of mathematics and include mathematics recognition hints provided in metadata.

The ideas are available at: https://www.w3.org/community/exercises-and-activities/wiki/User_Input_of_Mathematics .

[johanneswilm]

Hey @AdamSobieski . This looks very interesting. If I understand your idea right, you are adding some information about what kinds of math input is to be expected, and any character recognition system or keyboard input system for math formulas will then have an easier time doing so. For example, if the hint makes it clear that only chemical formulas are to be expected, the input will be less likely to interpret a specific input as something else.

Whether or not it is viable has a lot to do with how much interest there is from browsers which again depends on the general population. Hitherto the attraction for general input has not been that great. So browsers have gone a decade or more in code for richtext input, while some very specific JS projects only have focused on input and they have been quite annoyed by not being able to get around some of the bugs in this space.

The idea with input events is therefore to transfer most of the work to JavaScript, where the specific code with domain knowledge can live. The various input types are just a long list of all the input types we have found in one of the four browsers. Most new input types will probably be entirely defined in JavaScript and therefore not be mentioned by the spec at all.

Previously, the interest for mathml and similar was also not that great among browser makers. Do you think that has changed and that they are willing to implement this and have staff work on this over the years fixing bugs? If not, the best we can do is probably to see if the input JS libraries for math have all the primitives they need in input events rather than have the browser directly work with formulas.

[AdamSobieski]

@johanneswilm , thank you. The hyperlinked content (https://www.w3.org/community/exercises-and-activities/wiki/User_Input_of_Mathematics) shows a means of providing contextual hints for mathematics user input recognition. Such recognition hints are poised to boost recognition accuracy and to help to provide “just works” scientific and mathematical input recognition.

Presently, there are four types of mathematics recognition hints. Firstly, there is the category or subject of input; this could be the variety of scientific notation expected, e.g. chemistry, or the variety of mathematics expected. Secondly, there are the symbols expected, e.g. “x” or “chi”, “w” or “omega”. Thirdly, entire expressions can be provided as hints. Fourthly, templates can be provided, expressions with blanks, e.g. for fill-in-the-blank user experiences.

From a perspective of facilitating educational exercises and activities in Web browsers, what we would like to see is user input for mathematics exercises beyond multiple choice forms (for instance, we can observe https://math.ly/ which utilizes multiple choice for algebra exercises). What we would like to see is students able to naturally write or speak multiple steps of mathematical reasoning for educational exercises and activities.

To your questions, we can observe hardware trends with respect to mobile computers, tablets and laptops shipping with stylus devices. Stylus-based computing and related Web user experiences appear to be a priority for major operating system vendors and major browser vendors. Microsoft Edge, for instance, supports pen-centric computing in the form of Web notes. Microsoft also provides a Math Input Control.

Speech technologies also appear to be popular among major browser vendors.

Input Events Level 2 seems scalable and ready for textual and mathematical handwriting and speech recognition user input. I noticed that the InputEvent interface includes use of DataTransfer. In addition to text, rich text, hypertext and mathematics, we can consider that recognition components or services could offer additional data, e.g. InkML data, through such interfaces.

Understanding that recognition hints now exist, a needed technology is a means of invoking recognition components or services with available recognition hints. This could vary per operating system, passing the data from provided recognition hints to components or services upon each invocation. Once these technical details are considered, hints-enabled browser-to-component or browser-to-service communication, a community of developers stands ready to provide third-party mathematics recognition components and services.

We can hope that awareness of and interest in mathematics, in MathML, is increasing among major browser vendors.

P.S.: Based upon your interest in document authoring, you may find interesting a technology which impressed me a few years ago, Lurch (https://www.youtube.com/watch?v=DVivAtsRRKM , http://nathancarter.github.io/weblurch/app/app.html).

[AdamSobieski]

https://discourse.wicg.io/t/proposal-user-input-of-mathematics/2998

[siusin]

Thanks all.

We're closing this issue on the W3C HTML specification because the W3C and WHATWG are now working together on HTML, and all issues are being discussed on the WHATWG repository.

If you filed this issue and you still think it is relevant, please open a new issue on the WHATWG repository and reference this issue (if there is useful information here). Before you open a new issue, please check for existing issues on the WHATWG repository to avoid duplication.

If you have questions about this, please open an issue on the W3C HTML WG repository or send an email to [email protected].