Take screenshots of UI throughout functional tests. #7410
Conversation
cjcenizal
force-pushed
the
improvement/ui-screenshot-coverage
branch
2 times, most recently
from
42519de
to
ae9a309
Compare
|
Is this for visual regression testing? Should we also fail the tests for CI if a new screenshot is generated? That way we can ensure the modified screenshots are committed? |
|
@tylersmalley Right, exactly. I'm planning on using these in conjunction with our screenshot-comparison tool (#7395) to protect against visual regressions while I refactor our styles. I think it would be great to incorporate it into our automated testing, but I think we need to use the tool manually for awhile before we can use it to fail builds. For example, right now it can fail due to a resource not loading in time (e.g. a font), which results in a false positive, so we'd need to fix that issue first. |
|
That sounds great. Seeing how this is simply adding more screenshots, this LGTM. |
cjcenizal
force-pushed
the
improvement/ui-screenshot-coverage
branch
from
24ea8f1
to
50bd95a
Compare
cjcenizal
force-pushed
the
improvement/ui-screenshot-coverage
branch
3 times, most recently
from
ccec815
to
64a6786
Compare
|
I am getting a error when running test:visualRegression`: ./node_modules/.bin/grunt test:visualRegression
Compiling all doT templates...
Compiling visual_regression_gallery.dot to function
Running "test:visualRegression" task
(97.57%) screenshot-LineChart.png
(95.99%) screenshot-VerticalBarChart.png
(97.38%) screenshot-DataTable.png
(95.48%) screenshot-AreaChart.png
{ [Error: ENOENT: no such file or directory, open '/Users/tyler/Code/tylersmalley/kibana/test/screenshots/baseline/screenshot-LineChart.png']
cause:
{ [Error: ENOENT: no such file or directory, open '/Users/tyler/Code/tylersmalley/kibana/test/screenshots/baseline/screenshot-LineChart.png']
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/Users/tyler/Code/tylersmalley/kibana/test/screenshots/baseline/screenshot-LineChart.png' },
isOperational: true,
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/Users/tyler/Code/tylersmalley/kibana/test/screenshots/baseline/screenshot-LineChart.png' }
Warning: Task "test:visualRegression" failed. Use --force to continue.
Aborted due to warnings. |
|
Ah, that's because the task looks in your screenshots/session directory and tries to compare against files with the same name in the screenshots/baseline directory. I changed the way the screenshots are named, so you'll need to run the functional tests first (to re-generate all the session screenshots), and then run |
| common.debug('Start of test' + testSubName + 'Visualization'); | ||
| var vizName1 = 'Visualization ' + testSubName; | ||
|
|
||
| var vizName1 = 'Visualization MetricChart'; |
There was a problem hiding this comment.
This assignment is being used
|
This ran as expected after the clean:screenshots task was implemented. The gallery looks great, and I could definitely see us adding filtering down the road. |
cjcenizal
force-pushed
the
improvement/ui-screenshot-coverage
branch
from
46d4f97
to
cf6e7c1
Compare
- Add 'test:visualRegression' grunt task. - Run 'test:visualRegression' as part of npm script 'test'. - Add 'clean:screenshots task'. - Clean screenshots/session when funtional tests are run.
cjcenizal
force-pushed
the
improvement/ui-screenshot-coverage
branch
from
cf6e7c1
to
9e353fd
Compare
…hot-coverage Take screenshots of UI throughout functional tests. Former-commit-id: aa47599
No description provided.