feat: add new assertVisual command

[TheKohan]

Proposed changes

This PR is built on top of #1898 by @bartekpacia and implements a basic assertVisual flow which is much awaited in the community (#1222).

Given a simple React Native app

app.tsx

export default function App() {
  const [toggle, setToggle] = React.useState(false);
  return (
    <View style={styles.container}>
      <View style={[styles.dot, {backgroundColor: toggle ? 'red' : 'blue'}]}>
        <Button title="test" onPress={() => setToggle(!toggle)}></Button>
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    marginTop: 200,
  },
  dot: {
    backgroundColor: 'red',
    padding: 30,
    borderRadius: '50%',
  },
});

and a flow

test-flow.yml

appId: com.example.app
---
- launchApp
- tapOn: 'test'
- assertVisual: Login screen # compares the current screen against `.maestro/visual_regression/Login screen.png`

maestro will compare screenshots, producing result image with differences masking.

 ║  > Flow: test-flow                                             
 ║                                                                
 ║    ✅   Launch app "com.example.app"                                                              
 ║    ✅   Tap on "test"                                          
 ║    ✅   Tap on "test"                                          
 ║    ❌   Assert visual difference with baseline Login screen (threshold: 95%)                                                     
 ║   

Baseline	Current	Result

Result for now is being saved in .maestro/failed_visual_regression/ directory.

If there's no baseline, current will be saved as one.

What is missing:

• changes accepting flow
• cloud integration
• testing
• remove system-ui elements (like status-bar)
• component-scoped tests (partial screenshots for component of given test-id)

Testing

• in progress - test plan has not yet been laid out due to potentially big changes in the proposed flow.

Issues fixed

• [Feature Request] Visual Regression Testing #1222

[Fishbowler]

You've mentioned missing the "changes accepting flow". What have you got in mind for the workflow for a failure where the new "look" is as-intended?

[TheKohan]

You've mentioned missing the "changes accepting flow". What have you got in mind for the workflow for a failure where the new "look" is as-intended?

@Fishbowler - the only thing i could've think of as of right now is to keep baseline in repository and in case of expected diff fail CI with a message and let the user manually delete current baseline. #1222 - here are some takes on this problem also.

[jtdaugh]

You've mentioned missing the "changes accepting flow". What have you got in mind for the workflow for a failure where the new "look" is as-intended?

@Fishbowler - the only thing i could've think of as of right now is to keep baseline in repository and in case of expected diff fail CI with a message and let the user manually delete current baseline. #1222 - here are some takes on this problem also.

We would use this asap in our project and for us modifying the "baseline" image as part of the PR would be our expected workflow, and would actually be a great way for a design/product stakeholder to review any of the baseline screenshot diffs as part of a PR. In other words, no additional build here at all is a perfectly good "accept changes" workflow for our team.

[Fishbowler]

I'm not staff, but this LGTM. Awesome, in fact!

I've left a couple of comments, but neither is a hill I'll die on.

Would need an accompanying PR in the docs repo, which we'd probably want to label with "experimental" or something at this point.

[Fishbowler]

It's a nitpick, I know, but thinking of terminal in a vscode window on a MacBook... Could this be shorter?

Perhaps not 🤔

[bartekpacia]

ideas:

• visual difference -> visual diff maybe?
• display just filename of $baseline, not the whole path

[Fishbowler]

Should this be exposed as an environment variable (assuming you don't want to go as far as to expose it to workspace config at this point)?

[TheKohan]

@bartekpacia - would gladly accept review/guidance on this one. Also #2086 would be invaluable for this.

[rikur]

Excellent to see some new features coming into Maestro, thank you @TheKohan 🙏

Could mix very well with this: #1526

[bartekpacia]

Hey @TheKohan – this is some exciting work. I no longer work at @mobile-dev-inc, and am a bit sad I didn't have time to finish this:(

cc @amanjeetsingh150 – he's the right person to review this PR:)

[TheKohan]

@amanjeetsingh150 - can you review this PR ?

[amanjeetsingh150]

Can you add an integration test for this command for the different cases you expect?

[amanjeetsingh150]

Will this generated file highlight the diff between old and new?

[amanjeetsingh150]

I feel we should ignore the system status bar as scope of this PR since it would start to fail with difference in time or having notifications 🤔 ? WDYT?