Add support for real-time text in "Type a message"

[mdrejhon]

Hello!

I'm a deaf software developer, who uses both transcription (for incoming text) and keyboard (for outgoing text).

I've been immensely enjoying the ability to use mimiuchi as a free alternative to Otter or other paid services which I also use. I like the fact that mimiuchi supports WebHook and WebSockets.

Thanks to "Type a message", mimiuchi conveniently can be programmed as a browser-based keyboard for another device (with a bit of programming glue). I also like using mimiuchi to turn a phone/iPad into a portable keyboard for a big screen.

Use case: Mimiuchi acts as a substitute for a bluetooth keyboard

Allows guests to bring their own device, to use as a keyboard for a main communications screen. When no spare keyboard is available.

Or that we're doing chatting on-the-go across the table (like a Ubiduo assistive device) privately between two offline devices (mimiuchi works offline once cached). Or multiple chatters (one at a time) chatting in real time to a central big screen (e.g. living room TV or conference room projector) in front of them.

Basically any iPhone/Android/iPad/laptop/etc can become a portable guest keyboard for another device such as a main big chat screen (on TV). Transcriptions still work but some people prefer to type. The guest typist sitting next to me, can use their device to send messages to the big screen (through mimiuchi) which works instead of a Bluetooth keyboard.

However, it's not as realtime-feeling as a bluetooth keyboard typing to a laptop connected directly to a big screen. It's nice giving people a choice to speak or type...

So as a workaround for this use case:

Is it possible to add a realtime mode to the "Type a Message" where keypresses are sent one by one in real time? Or even as batches every 0.25 second or 0.5 seconds? (A configurable time-based buffer to rate-limit webhook/websocket requests).

Basically an optional "Real time typing" mode where keypresses are sent individually (or almost).

[mdrejhon]

Some UX thoughts:

• A Setting that enables real time typing mode (Default off)
• Keep the UI unchanged, except that differentials are transmitted (updates to Type a Message textbox is monitored on a timer). Type a Message cleared upon Enter.
• Or modify the UI to instead type directly into the main text area.
• Other idea

One unresolved question is what to do with message edits if doing it via "Type a Message" box. Maybe use the same autocorrect protocol (text-replace). Or, if modified to type directly into the main text area, then only backspace would be supported.

Whichever is easier to adapt mimiuchi architecture to double as a realtime "web keyboard" client for a remote chatting screen or central chatting screen.

[naeruru]

Sure, I can work on this! I had some ideas for a separate, typing focused home screen, and I’ll mess with that more this week and next. I like your simple idea of just making the text window a text area of some sort

[mdrejhon]

Fantastic. Thanks, @naeruru!

When will this version be put up at www.mimiuchi.com? I'll give it a test.

This should produce some really interesting "use your smartphone as a web-based keyboard" and "guest keyboard" use cases.

• Situations that invites guest keyboards
• Ubi Duo style dual-keyboard assistive chat
• Collaborative typing onto a shared screen (assistive), whether splitscreen, texting-style or IRC-style
• WiFi keyboard for WiFi-only devices (e.g. @ddvk hack utility for BT-less Remarkable2 paper tablet), emulating proprietary TypeFolio

And the bonus, you can use either realtime dictation or realtime typing for all the above.

One person can type (in mimiuchi), another person can speak (in mimiuchi), straight into a 3rd app (just 100 lines of code) that is a splitscreen like old Unix Talk, BBS Talk or a Ubiduo

With this simple change, I suspect Mimiuchi just became probably the world's first general-purpose platform-independent "turn any device into a keyboard" client. Use line-keyboard, realtime-keyboard, or voice dictation, your choice. Very accessible!

And you can have multiple devices running multiple copies of mimiuchi to the same screen for collaborative use cases.

Your modification opens quite a lot of interesting assistive and non-assistive use cases.

[naeruru]

I just pushed it live now! Sorry for the wait. Yeah I think it is a good change. However, the documentation is a little sparse on these features and I'd like to focus on that next so people can understand the endpoints a little better. Also the UI on a smartphone for typing could be a little better. The current larger footer option is simply a stopgap at the moment. I'll work on it more soon

[mdrejhon]

Understood! mimiuchi is a great accessibility app for people who used WebCaptioner in the past...

But your app has other goodies that makes it double as a great "guest keyboard" or "accessible guest communication app" that has 3 modes (line input typing, realtime typing, and dictation).

I think it would be great to (better) publicize and optimize your app's newfound ability to be used as a super-flexible guest keyboard.

Related feature suggestion (brainstorm) for using mimiuchi as spontaneous guest keybords:

In theory I can add QR codes to my central chat screen that opens up mimiuchi to a specific IP address. The user can then broadcast their communications back to the central screen (splitscreen, IRC screen, groupchat text style, etc).

Don't know what the security implications are (analysis needed, good thought needed) but the ability to specify the REST/websocket as part of a URL argument would make this easier; at least when mimiuchi is used in a localhost situation.

Then anybody (on the same WiFi network) who scans a QR code on the central screen, can mimiuchi-chat to that central intranet chatting app screen (conference room screen, TV screen, classroom screen).

P.S. Ten years ago, I worked on XMPP XEP-0301 In-Band Real Time Text, though it's not all that widely deployed. But I had thought of a "scan QR code to join a central chat screen" system back then. Your app would probably only need minor modifications before mimiuchi was capable of being that guest client.