Design: Switching input mode

[compulim]

User story

Today, we mix typing and speech into the send box.

The end-user is not very indicative on whether the microphone is recording or not, especially without a waveform animation.

Please look at #1839 when working on this.

Alternatives

Today, we mix the input via keyboard and speech into a single input box.

Other implementations

Send box in Cortana and Siri can be either for keyboard or speech, but not hybrid.

Potential implementation

Tomorrow, we could use the microphone button to switch between 2+ types of input. And the end-user should has a very clear indication on what type of inputs they are using.

What this will fix?

• Synthesize all bot responses
  • Ability to synthesize all bot responses are very popular asks
• Synthesize proactive messages
  • When the user switch to speech mode, we can prime Web Audio API thru user gesture. This will enable synthesizing all proactive messages (Speech-enabled bot does not "speak activities" in certain cases #2474)
• Synthesize bot responses on file uploads
  • We also need to cater about file upload. If the user is in microphone input mode, they should still be able to upload files (e.g. mobile scenario). And the bot responses will be synthesized (Speech-enabled bot does not "speak activities" in certain cases #2474).
• Speech Synthesizer behavior change in V4 #2568

[Enhancement]

[compulim]

Punting this to R9.

[DesignPolice]

Thanks @compulim - I think we will need to do some testing on how best to do this, given the number of different settings that WebChat is used.

[Quirinevwm]

UX recommendations:
As discussed during our sync, this modality switching asks for a design pattern to support you. It's very hard to go for the hybrid interruption fix, let's first go for the chat versus speech fix.

(1) 1 modality
Only "chat" or "speech" modality, the bot is only accessible via 1 modality:

"Chat"
User types = (no > icon needed)
User shares/uploads = action button (clip or + icon) should be separate from modality, typically positioned left of insert field.

"Speech"
User talks = (microphone icon)

(>1) Switch between modalities, only send content via 1 modality to bot:
Hypothesis: when an agent supports both modalities, whenever the user starts typing "chat" in the "chat field" they can send over their message by clicking enter. If they want to "speak" to the agent, speech can be accessed by clicking on the speech icon.

The proposed pattern is that we stick to show the speak icon per default (on the right)/ upload icon (on the left) with supporting text in the entry field with 'Type a message', so whenever the user starts 'chat' (so typing) they can just click enter to send through their message, and “listening” when a user is recording.

Flow =
Whenever a user start typing and they decide halfway they do want to speak out their content, the user will need to erase their textual written chat, for the mic to be clickable and kick-off the conversation via voice by clicking on that mic icon. They can follow up by speaking again and clicking the microphone. Or start typing again in the "chat" field.

(See design link for exploration & design pattern)

(>1) move halfway from chat to speaking is out of scope for now.

[pinarkaymaz6]

Looking forward to this, is there any update?

[corinagum]

We're still in the design phase for this, but the team is also excited for this feature! We are hoping to get to coding phase soon!

[fraygosa]

update? I noticed the front-burner got removed. Would like to implement this on our bot.

[fraygosa]

update? I noticed the front-burner got removed. Would like to implement this on our bot.

The issue is because a user cannot determine whether they are typing or not. Users will have a difficult time thinking they have to speak. If the mic button can change to send button when there is typing it will go a long way to remove user frustration. This feature works on v3. Can this not be treated as a bug?

[harish-madugula]

Hey there!
any workaround or update on this?
any help is appreciated.
Thanks in advance.

[casperhuijsman]

Can you provide an update?

[casperhuijsman]

@cwhitten @compulim

Could you update on this?