iOS/Android: Text input error for screenreaders

[fabOnReact]

Issues facebook/react-native#30848 facebook/react-native#30859
PR facebook/react-native#33468
Branch facebook/react-native@main...fabriziobertoglio1987:text-input-errors
New Branch facebook/react-native@main...fabriziobertoglio1987:text-input-errors-build
Branch used for debugging with additional history facebook/react-native@main...fabriziobertoglio1987:text-input-errors-build-debug-statements-2
Docs PR facebook/react-native-website#3010
Related https://developer.android.com/reference/android/widget/TextView#setError(java.lang.CharSequence) https://developer.android.com/reference/java/lang/CharSequence https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#setError(java.lang.CharSequence) facebook/react-native#29070 facebook/react-native#29063 (comment) facebook/react-native#28952 #13

[fabOnReact]

fix issue with missing drawable

with drawable	without drawable

[fabOnReact]

Change of error state from onChangeText show/hides a TextInput error

Expected Result:
The Error shows next to the TextInput

Actual Result:
No Error Show next to the TextInput

Applying changes from with facebook/react-native#29070 fixes the issue.

click to open test

function ErrorExample(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState(null);
  const textinput = React.useRef(null);
  return (
    <>
      <Button title="create error" onPress={() => setError('button')} />
      <TextInput
        ref={textinput}
        errorMessage={error}
        onBlur={() => setError('onBlur')}
        onChangeText={newText => {
          setText(newText);
          setError(newText === 'error' ? 'this input is invalid' : null);
        }}
        value={text}
      />
    </>
  );
}

without PR 29070

without facebook/react-native#29070

without_pr_29070.mp4

with PR 29070

with facebook/react-native#29070

with_pr_29070.mp4

[fabOnReact]

Test Cases of the functionality (Android - Fabric)

error added onChangeText

2022-04-21.17-03-38.mp4

error added on onEndEditing

2022-04-21.17-04-45.mp4

[fabOnReact]

understanding onChangeText callback

https://github.com/fabriziobertoglio1987/react-native-notes/blob/3b7581cd763635aad605cc175bc3cb74182f4a85/ReactAndroid/src/main/java/com/facebook/react/views/textinput/ReactEditText.java#L564-L565

onChangeText is disabled in ReactAndroid by using replace instead of setText

https://github.com/fabriziobertoglio1987/react-native-notes/blob/c5762a77a9e720e0129175afc38c7dcf23bea7bd/ReactAndroid/src/main/java/com/facebook/react/views/textinput/ReactEditText.java#L567-L567

instead this callback controls onChangeText

https://github.com/fabriziobertoglio1987/react-native-notes/blob/c5762a77a9e720e0129175afc38c7dcf23bea7bd/Libraries/Components/TextInput/TextInput.js#L1114-L1131

solved with fabOnReact/react-native@d3d54e1

[fabOnReact]

AndroidTextInputProps and AndroidTextInputState.cpp

https://github.com/fabriziobertoglio1987/react-native/blob/a7a781ff4a13e744f4eb3007ef0657740b277a72/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputProps.cpp#L268

fabOnReact/react-native@5abe584

ReactEditText - need to add the errorMessageAndroid to AndroidTextInputState
currently errorMessageAndroid is part of the props but not of the state, while underlineColorAndroid is part of state.

I added errorMessageAndroid prop to

https://github.com/fabriziobertoglio1987/react-native/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputProps.cpp#L67-L70

but needs to be added also to AndroidTextInputState

https://github.com/fabriziobertoglio1987/react-native/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputState.cpp#L29

so the StateWrapper

https://github.com/fabriziobertoglio1987/react-native/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactAndroid/src/main/java/com/facebook/react/uimanager/StateWrapper.java#L4

https://github.com/fabriziobertoglio1987/react-native/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactAndroid/src/main/java/com/facebook/react/views/textinput/ReactTextInputManager.java#L1291

includes also the errorMessageAndroid

need to follow similar implementation of underlineColorAndroid

https://github.com/fabriziobertoglio1987/react-native/commits/681ed402de9d6f119cd30583ad5048e1cb75bab4/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputState.cpp

[fabOnReact]

passing mock values to AndroidTextInputShadowNode.cpp

to pass mock value of errorMessageAndroid to the constructor, I would need to generate a string of type jsi::String

https://github.com/fabriziobertoglio1987/react-native-notes/blob/67b466f083cd32066dc289a4eaf65b1151d97e10/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputShadowNode.cpp#L155-L166

The method createFromAscii could be used to generate a random string.

Relevant links:

• jsi.cpp
• using jsi::String::createFromAscii to generate jsi string
• AndroidTextInputState.cpp
• AndroidTextInputState.h

Compilation error messages

[fabOnReact]

adding errorMessageAndroid to c++ state

adding

https://github.com/fabriziobertoglio1987/react-native-notes/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputState.cpp#L74-L91

newState["errorMessageAndroid"] = "my error message";

adds the error message in the java ReactTextInputManager udpateState method

https://github.com/fabriziobertoglio1987/react-native-notes/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactAndroid/src/main/java/com/facebook/react/views/textinput/ReactTextInputManager.java#L1290-L1303

[fabOnReact]

see facebook/react-native#33468 (comment)

TextInputState paragraphAttributes and textAttributes

The above state is added here

https://github.com/fabriziobertoglio1987/react-native-notes/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactCommon/react/renderer/components/textinput/androidtextinput/react/renderer/components/androidtextinput/AndroidTextInputState.cpp#L85-L88

The functionality was initially introduced by adding the StateParagraph component commit 06ce568 to the state.

Updates the paragraph component to use State instead of Local Data, part of the path to a Fabric TextInput 💯

and then commit fabOnReact/react-native@13db6cb

This diff extends ParagraphState to expose not only the AttributedString associated to Text components, but also ParagraphAttributes that describes the visual high level props of the Paragraph

The AndroidTextInputState seems composed of two components:

• paragraph attributes like ellipsizemode textBreakStrategy maximumNumberOfLines
  This attribute control the styling
• attributesString in which we should add the errorMessageAndroid to update the state

https://github.com/fabriziobertoglio1987/react-native-notes/blob/b5786376f4efc590386860a574d29db90f3cddaf/ReactCommon/react/renderer/components/text/ParagraphState.h#L29-L38

Debug Statements fabOnReact/react-native@1e54f76

[fabOnReact]

setting the TextInput errorMessage state with setTextAndSelection Java API from JavaScript

Related links

• iOS/Android: Text input error for screenreaders #12 (comment)
• TextInput JavaScript call to setTextAndSelection
• ReactAndroid API to receive updates from JavaScript in ReactTextInputManager.java (setTextAndSelection)
• ReactAndroid API to receive updates from Fabric c++ state
• Fabric attributedstring
• Fabric AndroidTextInputState

Expected Result:
An error message is displayed on the TextInput when the user types error in the field

Actual Result:
No error message is displayed. The state update triggered in JavaScript is prior to the fabric state updates which over-rides the state and removes the message

click to open code used in the test

function ErrorExample(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState(null);
  return (
    <>
      <Button onPress={() => setError('button')} title="Press to set error" />
      <Text>
        Type error in the below TextInput to display an error message.
      </Text>
      <TextInput
        maximumNumberOfLines={2}
        accessibilityRole="button"
        hyphenationFrequency="normal"
        errorMessageAndroid={error}
        onBlur={() => setError('onBlur')}
        onEndEditing={() => setError('onEndEditing')}
        onChangeText={newText => {
          setText(newText);
          if (newText === 'error') {
            setError('this input is invalid');
          } else if (error !== 'onBlur') {
            setError(null);
          }
        }}
        value={text}
        style={[styles.default, {height: 50}]}
      />
    </>
  );
}

click to open recording of test case

2022-03-30.11-49-06.mp4

[fabOnReact]

Test Cases of the functionality (Paper) after removing changes to .cpp libs

2022-05-12.19-45-17.mp4

[fabOnReact]

xcode stuck when loading pods

My MacBook Pro has very limited storage (125GB model). I spend considerable amount of time saving storage and did the mistake of enabling iCloud. The files were stored on iCloud and xcode was having issues downloading them.
I'm buying new MacBook Pro this weekend 🥳 .

I also experienced this on my new MacBook Pro, I enabled iCloud by mistake.

Screen.Recording.2022-05-12.at.20.53.42.mov

[fabOnReact]

setMostRecentEventCount is called after invoking setScreeenreaderError

https://github.com/fabriziobertoglio1987/react-native/blob/4e8af270100916b8b67198020b402a02020de8d9/Libraries/Components/TextInput/TextInput.js#L1143-L1159

_onChange callback does the following:

1. saves the currentText from event.nativeEvent.text. This is the text received from native side (iOS or Android)
2. calls onChangeText(currentText).
   Here the onChangeText javascript callback may take the currentText from native and change it.
   The native and javascript version of currentText may differ here.
3. call setLastNativeText(currentText) which saves in state the last value of the text sent from the native side
4. increase the mostRecentEventCount

useLayoutEffect does the following:

1. if lastNativeText (from iOS/Android) differs from the props.value
2. we update the lastNativeText to props.value
3. we trigger setTextAndSelection with props.value to over-ride the native iOS/Android value of the text
   This callback runs only after onChangeText, as mostRecentEventCount is increased once onChangeText code is executed.

With the screenreaderError we don't receive a value from native (iOS or Android) that we want to over-ride.
The screenreaderError is sent from JavaScript as prop. The native value should allways be over-ridden from the JavaScript value.

I think the issue here is we use setTextAndSelection to over-ride the native value (iOS, Android) of the text, but this over-rides also something else.
The other solution was adding to the fabric TextInputState field ParagraphState the field screenreaderError.
When c++ and android/ios send each other updates about the changes in state, the change in the screenreaderError is detected (onChangeText) and the value is displayed, otherwise the update is not shown.
I decided to not implement this solution because as the problem is common between iOS and Android, and the update is triggered from onChangeText in JavaScript. Seems to me that the solution should be done in JavaScript.

I run a test with another prop textAlign. textAlign is will update if changed onChangeText callback and it is included ReactTextUpdate https://github.com/facebook/react-native/search?q=textAlign

[fabOnReact]

iOS - announcing error onChangeText and screenreader focus

HEAD fabOnReact/react-native@bb4597e

new.test.low.quality.mp4

[fabOnReact]

iOS - The screenreader announces the TextInput value after the errorMessage is cleared

source code for example

function ErrorExample(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState(null);
  const [accessibilityInvalid, setAccessibilityInvalid] = React.useState(false);
  return (
    <>
      <Button onPress={() => setError('button')} title="Press to set error" />
      <Text>
        Type error in the below TextInput to display an error message.
      </Text>
      <TextInput
        accessibilityErrorMessage={error}
        accessibilityInvalid={accessibilityInvalid}
        // onBlur={() => setError('onBlur')}
        // onEndEditing={() => setError('onEndEditing')}
        onChangeText={newText => {
          setText(newText);
          if (newText === 'Error') {
            setError('the newText is: ' + newText);
            setAccessibilityInvalid(true);
          } else if (newText === 'empty') {
            setError(newText);
            setAccessibilityInvalid(true);
          } else if (newText === 'null') {
            setError('');
            setAccessibilityInvalid(false);
          } else {
            setError(newText);
            setAccessibilityInvalid(false);
          }
        }}
        value={text}
        style={styles.default}
      />
    </>
  );
}

Video of the test

[fabOnReact]

iOS - Exception thrown while executing UI block: - [RCTTextView setOnAccessibiltyAction:]: unrecognized selector sent to instance (Paper) (main branch)

Issue experienced on main branch when opening the accessibility example. The second error displays Exception thrown while executing UI block: 'parentNode' is a required parameter if we remove the accessibilityActions and onAccessibilityAction from the AccessibilityExample.
Solution https://stackoverflow.com/a/61570223/7295772

[fabOnReact]

iOS - Exception thrown while executing UI block: - RCTUITextView setAccessibilityErrorMessage:]: unrecognized selector sent to instance (iOS - Paper on main branch)

[fabOnReact]

iOS - accessibilityErrorMessage is not announced on Paper

• The current solution does work on Paper
• The TextInput component is based on UITextInput
• The method updateProps is not called on Paper, but works on Fabric

Steps to solve this problem:

• read the implementation of accessibilityValueInternal
• Update the accessibilityValue using the accessibilityValueInternal

I was able to implement a solution in TextInput.js to trigger the accessibility announcement in JavaScript.
The announcement works correctly onChangeText (Fabric and Paper), but on Paper does not announce the error outside of the onChangeText callback.

• try to trigger the announcement manually in JS when the prop updates on Paper
• on Fabric this diffing logic triggers the error. Verify that similar diffing logic is triggered on Paper (check shadow node).

The implementation of accessibilityValue and accessibilityValueInternal should be enough for this functionality and I believe no native code needs to be added for iOS.

[fabOnReact]

Fabric and Paper use different implementations to handle diffing for props. Changes to native iOS implementation should be minimal, as the accessibilityValue prop is already implemented for iOS.
In Fabric the correct implementation is:

1. The accessibilityValue is first set with the accessibilityValue setter
2. accessibilityValue updates are done in RCTViewComponent. The previous solution (trigger manually the notification with UIAccessibilityPostNotification) could fix the issue of errors being announced two times. Currently this issue does not reproduce if we don't set the accessibilityValue and just trigger the announcement in javascript, but then screenreader does not announce the error onHover.

In Paper the accessibilityValue is set:

• RCTViewManager#accessibilityValueInternal
• UIView+React#accessibilityValueInternal
• accessibilityValue setter

I was able to trigger the announcement correctly onChangeText (no on hover) using only JavaScript:

1. adding useEffect callback in TextInput.js

  React.useEffect(() => {
    // triggers the error onChangeText
    if (props.accessibilityInvalid === true) {
      AccessibilityInfo.announceForAccessibilityWithOptions(
        props.accessibilityErrorMessage,
        {queue: true},
      );
    }
  }, [props.accessibilityInvalid, props.accessibilityErrorMessage]);

2. Not passing the accessibilityErrorMessage to iOS RCTTextInputView
   The above solution does not announce the error when the screenreader focus moves to the TextInput. The Screenreader needs to read the accessibilityValue, but setting this value may cause announcing the error two times.

[fabOnReact]

Implementation of TextInput Component on iOS

RCTBaseTextInputView is a wrapper View around the UITextField:
setAttributedText is part of RCTBaseTextInputView.m.

RCTBaseTextInputView does not implement backedTextInputView which needs to be over-ridden in RCTSinglelineTextInputView.

1. It modifies the TextInput component text and textAttributes using backedTextInputView
2. backedTextInputView (RCTUITextField) is a wrapper around UITextField

RCTUITextField is a iOS Component build on top of UITextField:
The setter setAccessibilityErrorMessage is part of Singleline/RCTUITextField, which inherits methods/functionalities from RCTBackedTextInputViewProtocol and UITextField

Solution:

1. save the prop accessibilityErrorMessage in an instance variable of Singleline/RCTUITextField
2. update the accessibilityValue when the text is set with RCTBaseTextInputView#setAttributedText
3. set the accessibilityValue to the text + self.backedTextInputView.accessibilityErrorMessage

Issues:

1. updating the instance variable self.errorMessage from RCTUITextField does not display any change in RCTBaseTextInputView#setAttributedText. The error is always null.

Todo:

• RCTUITextField#setAccessibilityErrorMessage does not update self.accessibilityErrorMessage
RCTUITextField#setAccessibilityErrorMessage can update self.accessibilityValue
  Compare the implementations of self.accessibilityValue and self.accessibilityErrorMessage
• self.backedTextInput.accessibilityErrorMessage = "error" works in setAttributedText. Try to log the values in setAttributedText to verify it.
• verify that iOS Fabric announces error outside of onChangeText
• fix tests cases from commit facebook/react-native@582b1e8 also on Fabric
• use same iOS Paper solution on Fabric
• verify iOS Paper solution works with non controlled TextInput
• comments in my own code review
• test multiline TextInput on iOS
• sometimes when the RN App builds, the screenreader announces random letter.. for example "o"
  but I don't think is a concern, because the changes are scoped for TextInput Component

[fabOnReact]

iOS - announce lastChar (not entire text) onChangeText and avoid multiple announcements (Fabric)

Commit fabOnReact/react-native@b53ed9d fixes the following issues on iOS Fabric (apply solution from Paper)

1)The error is announced, but not with the character onChangeText
2)The error is announced multiple times

Related #12 (comment) #12 (comment)

sourcecode

function AccessibilityErrorWithButtons(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState(null);
  const [accessibilityInvalid, setAccessibilityInvalid] = React.useState(false);
  return (
    <View>
      <RNTesterBlock>
        <TextInput
          accessibilityErrorMessage={error}
          accessibilityInvalid={accessibilityInvalid}
          onChangeText={newText => {
            if (newText === 'Error') {
              setError('this is the error');
              setAccessibilityInvalid(true);
            } else {
              setError('');
              setAccessibilityInvalid(false);
            }
            setText(newText);
          }}
          value={text}
          style={styles.default}
        />
        <Button
          onPress={() => {
            setError('This error is set with Button onPress callback');
          }}
          title="Press to set an accessibilityErrorMessage to a non-empty value"
        />
        <Button
          onPress={() => setAccessibilityInvalid(!accessibilityInvalid)}
          title={`Press to set accessibilityInvalid to ${
            accessibilityInvalid ? 'false' : 'true'
          }`}
        />
        <Button
          onPress={() => setError('')}
          title={`Press to set an empty error message`}
        />
      </RNTesterBlock>
    </View>
  );
}

low_quality.mp4

[fabOnReact]

iOS - announces or does not announce the accessibilityError through Button onPress (not onChangeText) (Fabric)

sourcecode

function AccessibilityErrorWithButtons(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState(null);
  const [accessibilityInvalid, setAccessibilityInvalid] = React.useState(false);
  return (
    <View>
      <RNTesterBlock>
        <TextInput
          accessibilityErrorMessage={error}
          accessibilityInvalid={accessibilityInvalid}
          onChangeText={newText => {
            if (newText === 'Error') {
              setError('this is the error');
              setAccessibilityInvalid(true);
            } else {
              setError('');
              setAccessibilityInvalid(false);
            }
            setText(newText);
          }}
          value={text}
          style={styles.default}
        />
        <Button
          onPress={() => {
            setError('This error is set with Button onPress callback');
          }}
          title="Press to set an accessibilityErrorMessage to a non-empty value"
        />
        <Button
          onPress={() => setAccessibilityInvalid(!accessibilityInvalid)}
          title={`Press to set accessibilityInvalid to ${
            accessibilityInvalid ? 'false' : 'true'
          }`}
        />
        <Button
          onPress={() => setError('')}
          title={`Press to set an empty error message`}
        />
      </RNTesterBlock>
    </View>
  );
}

first_low.mp4

[fabOnReact]

iOS - the error is announced with accessibilityInvalid true and does not clear after typing text (onChangeText) (Fabric)

sourcecode

function AccessibilityErrorDoesNotClear(): React.Node {
  const [text, setText] = React.useState('');
  const [error, setError] = React.useState('');
  const [accessibilityInvalid, setAccessibilityInvalid] = React.useState(false);
  return (
    <View>
      <RNTesterBlock title="TextView without label">
        <Text>Set an error, change textinput value and then announce it.</Text>
        <TextInput
          accessibilityErrorMessage={error}
          accessibilityInvalid={accessibilityInvalid}
          value={text}
          onChangeText={newText => setText(newText)}
          style={styles.default}
        />
        <Text>
          The value of the accessibilityErrorMessage prop is{' '}
          {error.length > 0 ? error : 'empty error'}
        </Text>
        <Button
          onPress={() => {
            setError('This is an error');
          }}
          title="sets accessibilityErrorMessage to 'This is an error'"
        />
        <Button
          onPress={() => setAccessibilityInvalid(!accessibilityInvalid)}
          title={`Press to set accessibilityInvalid to ${
            accessibilityInvalid ? 'false' : 'true'
          }`}
        />
      </RNTesterBlock>
    </View>
  );
}

second_low.mp4