Issues with skipped tests in summary and html report on V0.9.2 #755
Comments
|
sorry closing as you have not followed the instructions: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue |
|
I also said in https://stackoverflow.com/questions/55944353/karate-abort-script-failing-in-version-0-9-2 that it's expected to have a fail when abort() is used. |
ptrthomas
added a commit
that referenced
this issue
May 2, 2019
|
@MercierAdrien I just added a demo example for this, see commit above. I think the maven cucumber report has logic to mark the scenario as failed if there are skipped steps - let me know if my understanding is right. in Karate, we treat it as a pass, which is reflected in the maybe there is an opportunity to influence the 3rd party reports, but I remember we experimented with different statuses in #416 #464 and finally decided on |
|
looks like this has been requested here: damianszczepanik/cucumber-reporting#636 but looks unlikely to be fixed in the near future. I'm proposing we flip any skipped steps after an abort to "passed" - well anyway I keep saying that abort() is not the most elegant way to write tests - so here goes. reopening |
ptrthomas
added a commit
that referenced
this issue
Jul 17, 2019
…755 this is not ideal but because of the behavior of the 3rd party report - see issue details and thread for more
|
for the reference of those looking for more info the developer of the 3rd party report has changed the behavior, refer damianszczepanik/cucumber-reporting#862 (comment) |
|
@ptrthomas : It looks like that Cucumber-reporting fixed the issue: damianszczepanik/cucumber-reporting#862 |
|
@gerben86 yes. it requires some research into which version of that dependency fixes things (was not clear last I checked) and whether anything else breaks. this is one dependency that we have recommended the same old version for 2.5 years. if anyone does the research and comes up with a recommendation, sure :) |
|
@gerben86 how about this - we intro a |
|
@ptrthomas : I think that could work, because then we can decide how to treat the abort(). |
ptrthomas
added a commit
that referenced
this issue
Dec 15, 2019
|
implemented |
|
released 0.9.5 |
SalilGupta12
added a commit
to SalilGupta12/karate
that referenced
this issue
Jul 27, 2020
* webauto: this is big, we have friendly locators * webauto: move finder & mouse to interfaces so that the method signatures are easier to read for those viewing via readme / doc links * adjust maven profile for ui in hope that it works * Revert "Add afterFeature and beforeFeature hooks, RunnerBuilder" This reverts commit 88026a2 * Junit4 can run tests in parallel * improve zip based on feedback and doc * more doc updates based on feedback * implemented waitForText() short-cut * webauto: decided to rename waitForText() and waitForEnabled() * Revert "Merge pull request karatelabs#871 from pshrm/develop" This reverts commit cb905bd, reversing changes made to 0da4f41. * slight improve to timeline view tooltips * additional tweak for karatelabs#826 replaces karatelabs#863 * unified cli for karate core somewhat karatelabs#827 in the future, the netty picocli has to be merged and we also would have native html parallel reports but for now, we can start testing the vs code plugin for this parallel execution will now generate native html report and continuously update the results-json.txt which the plugin can read and update etc * default to proper karate runner system always * fix bugs with cli runner and switch netty fatjar to runner builder karatelabs#827 * big improvement to intellij ide support the test results will have timings and logs associated correctly also using the unified karate short-syntax for scenario / line number etc * webauto: decided to use 1 based indexing for friendly locators trust me it jst did not feel right with zero based indexing * thats what happens when you dont run the local tests * webauto: focus() breaks on fancy input fields, fixed * hooks expanded for all cases, beforeAll() afterAll() and even beforeStep() afterStep() * webauto: improve target api design * webauto: handle cookie failure in web-driver * fix for focus() js and chrome cookie error routine * some long pending call vs read doc * minor cosmetic readme tweak * added a couple more crazy examples * webauto: reduce intensity of friendly locator search * gracefully fail if browser / input field type does not support moving text cursor to end * webauto: implemented waitForResultCount() * better error message if relative path does not exist * draft version of vscode debug adapter protocol implementation * move debug server test to junit so that karate-apache is on cp * Add resources methods to the Builder * vscode debug server start option for both standalone jar cli and the runner, so this can be fired in a maven project also upgraded picocli to latest for the -d arity 0..1 feature * minor loggin changes to debug server * use of waitForResultCount * debug adapter decoder bad logic fixed * debug server now supports called features almost everything, step-in, step-out, multiple breakpoints * get build to work on windows and clean up this breaks karatelabs#751 but needs investigation / reopen * refactor / cleanup to prev commit ref karatelabs#751 * debug adapter protocol: implemented repl evaluate and step-back * debug server: hot reload of code works so after editing a feature in the editor, the restart button on the debug toolbar needs to be used to trigger the hot-reload limitation for now: only line edits, no line adds or deletes will be supported but still, this is going to be awesome * tweak to fileutils for absolute file paths which does come into play when doing vs code debug sessions * debug: another breakthrough - we can debug parallel threads and now the vscode launch can accept the whole command-line parallel-threads tags and all * cleanup for prev commit * standalone jar cli error stack trace trimmed also docs here and there * debug server proper optimization of netty decoder * improve prev commit even more * some license headers were missing, added * fix relative path resolution, no more weird /// * wip distributed testing capability abstraction of job server and executor nodes call back over http wip docker image that includes karate standalone which acts as an agent we can get all reports and logs back from remote executors - todo aggregate * jobdist: wip continues karate-chrome docker container now based on java+maven and with standalone jar improve supervisord for shutdown from within a docker container via bash command improved / cleanup of job-executor code * jobdist: reports aggregation working after scale-out able to convert cucumber json back to result java objects on server * distjob: wip ui test videos can be retrieved now just need to fiddle with paths and some env props in java todo background step cucumber-json conversion * distjob: ironed out most issues with workflow and the video file corruption problem is solved * distjob: wip proven to work on jenkins + kubernetes one thing pending is video for distributed mode * distjob: improvements, video embed * fix bad bugs in job / exec * fix nasty bug with nested call with driver * intro beforeStart and afterStop for driver config greatly improved docker image stability - for now not starting socat for remote debug todo * fix step out in debug server docker now can start ffmpeg if needed for docker target * improve / simplify fat-docker build * always wait for browser / driver port even if not starting * no report for empty features * fixed debug npe for background steps * screen recording for mobile automation added startRecordingScreen and stopRecordingScreen API implemented saveRecordingScreen with embed to html feature * distjob: bullet-proofing and some testing * improve timeline html tooltip * debug: log message for hot reload implemented * debug: improve user feedback * debug: improve log for prev commit * debug: try system exit for clean stop * added appium api to readme doc * review changes karatelabs#895 * distjob: lot of fixes and bulletproofing saving work * update release process after 0.9.5.RC2 * adding back the simple jobserver test it is needed * fixed bugs in debug server any step failure resulted in no more steps working, fixed variable string conversion failure would crash session * fix for debug and maven class paths * decided to do the right thing and change driver scripts() to scriptAll() also introduced the especially useful in debug repl highlightAll() * debug support should work for gradle now * adding local docker demo runner * driver is auto passed to called features and does not matter if shared or isolated scope here we have a bending of the rules, a driver is always global * lower-case-headers applies only to keys not values * one line change makes jdk12 compile work for 8 runtime * attempt windows fix for paths with spaces * attempt fix for windows single feature debug session * introducing scriptAll() that takes 3rd filter-predicate arg * renamed findAll() to locateAll() and intro locate() * added 3rd delay arg to driver input api * for sake of debugger, driver element to be stringify-able * classpath from jar was not working in some cases * minor doc updates before rc3 * big change: we have killed the karate ui this means everything builds on openjdk 8-12 and no more build and ci problems for developers the karate-ui is not needed anymore because we have the visual studio code extension for debugging and we really tried to use javafx but there were too many issues * doc updates since we killed the karate-ui * changes for karatelabs#903 * doc updates * one more doc tweak * ci can be normal openjdk now * ci mvn cache, and goodbye karate-ui * some doc edits, and to test ci * improve contributor guides * wip: working on distributed gatling support and realized we need a stand-alone gatling example just like we have jobserver so decided to finally open an examples folder that will hold multiple examples of which we have a few already, so they can now live in the karate monorepo * readme edits * distributed job server working for gatling, todo test docker * job executor now will ping server with a heartbeat every 15 seconds this is great to ensure all remote ends are healthy, in the future this will allow us to do the following a) slurp logs from remote executors as long running test is progressing, think gatling - so we can generate reports any-time b) abort a test - when the next heartbeat comes in we can respond with a special case abort message the heartbeat has to use a second http client else severe concurrecy issues happen so this second thread can now throw an exception which will cause the main executor loop to shutdown * ignore examples jar files * add hot reload functionnality for mock server * multipart/form-data endpoint success from REST client but fails in Karate tests karatelabs#797 * minor edit for driver.title readme * make sure docker container has deps pre-loaded for gatling improve logs for job server (more at info level) to give confidence that things are happening etc * rebrand to karate ui, and added link to readme * change file change handler to use Jave Watchservice instead of a sleep loop * implemented local address support for gatling refer https://stackoverflow.com/a/55458266/143475 * added missing highlight() to element / finder api also added friendly locator find() option to also use visible text, not just tag name * code cleanup after karatelabs#909 * set type so that it does not create work dir with name null * HACKTOBERFEST-match != fails with two integers * fire change event for js based select-box twiddling * added retry chained to finder / element * doc edits * MockSpringMvcServlet issue for @ControllerAdvice NoHandlerFoundException * code cleanup after karatelabs#931 * Update consumer-driven-contracts to spring boot 2 Update consumer-driven-contracts to spring boot 2 * making correction to karatelabs#932 * update supervisord command to be the correct path for google-chrome * Update Chrome.java * Update Chrome.java * Update Chrome.java * fix gatling value copy edge case ref karatelabs#936 * minor code cleanup * Fix formatting in pages site * Fix formatting in pages site * escape intellij magic log strings karatelabs#954 * escape intellij magic log strings karatelabs#954 * gatling url pattern bug pretty bad miss that causes tests that use the comma-delimited path keyword to lose the forward-slashes so a url like http://foo/bar/baz would become http://foo/barbaz * release tweaks for 0.9.5.RC4 * edge case syntax error should not hang tests #959 * safer impl for #959 * finally, solution for custom masking of http headers / payloads in logs karatelabs#699 also attempted along with this is more control over [report verbosity] possible from the execution-hook * doc edits minor * http log masking for jersey also karatelabs#699 * fix for karatelabs#970 and replace karatelabs#973 * some doc / typo edits * dynamic scenario outline pre-scenario vars deep-copy was losing js functions karatelabs#982 * error handling for karatelabs#967 * updating docs * dont auto-close driver in called scenarios karatelabs#969 * implemented tags support in karate-gatling karatelabs#968 * attempt to fix karatelabs#924 but doesnt seem to work * updated docs to use Runner.path() builder API and not the KarateOptions annotation for parallel test execution * edit release process cheatsheet * edge case for regex combined with array fuzzy karatelabs#988 * junit 5 should fail if no features found karatelabs#989 also decided to rename the static method / helper to run() * update doc for karatelabs#989 * edit readme for examples/jobserver * add getPrevResponse() to scenario-context for advanced hook use-cases * better xpath for wildcard with index karatelabs#993 * of all the times you commit without testing karatelabs#993 * doc edits and archetype sync for junit 5 * multipart streams will be re-readable for retry until karatelabs#999 * [break] implemented configure abortedStepsShouldPass karatelabs#755 * implemented driver poll for port configurable karatelabs#990 * implemented mock headerContains() karatelabs#997 * improve code for karatelabs#755 * Add support for optional body in Postman convert Also adding whitespace to the Postman converted output. * log collection for dynamic scenario outline karatelabs#1003 * input stream special handling in json embedded expressions karatelabs#1009 * doc edits and import cleanups * Add logic for Postman import on netty CLI * Include tests * fix build after karatelabs#1011 * wip - introducing [karate-robot] karate-robot is an attempt to build the following cross-platform capabilities into karate a) native mouse events b) native keyboard events c) desktop image capture d) image matching we have the basics in place for mac, linux and windows decided to use the javacpp presets for opencv https://github.com/bytedeco/javacpp-presets/tree/master/opencv also the [karate-chrome] docker image has been revamped - now includes an x-windows manager (fluxbox) and is compatible with [karate-robot] note that opencv on windows seems to be slow to startup / load the dll-s and may need investigation otherwise we have the foundation in place to be able to use images to locate areas on the screen and navigate todo: improve the robot api and add a configure option similar to karate driver (now branded as karate ui) which will inject a [robot] object and helpers similar to how [driver] works today for karate ui * [robot] keyword to init robot + config similar to [driver] see feature file in commit for example * [robot] working on image matching, introed region concept and chainable api-s to find, click - also highlight() is working, very useful to troubleshoot image stuff facing some challenges with more image matching, needs investigation may need to deep dive into the opencv routine and look at other options, finding a set of matches instead of one and also getting into thresholds, search types, image types (right now only grayscale) etc * [robot] simple flow works lesson learned - the way opencv image finding works currently requires the image to be the same size we probably need a way to match even if scale is different, todo * [robot] wip find by image even when scale is not 1:1 seems to be working and even came up with a threshold detection score based on trial and error robot will now retry if image not found based on the threshold score calc driver config now working to highlight match region if flag is true * some places in docs / examples to use enhanced scenario outline * support headless flag for firefox * Update documentation * Update documentation * [robot] wip relaxing the image find threshold slightly * synchronized getFreePort to avoid duplicate port assignments * skip unnecessary window close on quit * configure header / cookies now reflect in http-request-builder karatelabs#1025 * Always add 'alwaysMatch' block to capabilities * make some readme section more clear * improve readme * fixed bugs in ui demo scripts * cucumber tables will now appear in report json / html report karatelabs#1035 * Support acceptInsecureCerts in driver options * Updated README with acceptInsecureCerts * simplify w3c webdriver capabilties karatelabs#924 so the user can specify whatever is needed and we keep the karare side simple this makes sense as there are so many options, eg saas / remote drivers * improve upon prev commit karatelabs#924 * improve edit upon prev commit karatelabs#924 * update w3c capabilities handling karatelabs#924 also updated chrome-web-driver to be w3c compliant for latest chrome version some new problems with safari, will defer * always use w3c driver capabilities alwaysMatch karatelabs#924 * update ui web examples * upgrade apache httpclient version * karate can now use a remote webdriver instance added [webDriverUrl] key and some extra driver init logic and works fine against zalenium breaking change for some of the wip appium work see doc changes for all the details * minor doc edits * confirmed to work with aws device farm and selenium grid standalone also changes to what was discussed in karatelabs#924 so now it is [webDriverSession] and not [webDriverCapabilities] so user has full control over the webdriver POST to /session payload - which will take care of any remote / saas situation and quirks of implementations like selenium grid introduced [httpConfig] key and now you can configure the http client e.g. readTimeout which is needed for aws as it can take a long time for a device / browser desktop to be provisioned see readme edits for details * fixed doc typo update * refactored webdriver implementation especially the start life-cycle, better error detection and handling on session fail mobile / winappdriver breaking change for starting session, need to use [webDriverSession] * fixed gaps in winappdriver demo * improve stabilize ui infra docker-target will pull only if flagged, and will remove container instance at end added nice test to check docker container locally in demo/driver/core chrome native will always wait for http to be ready * ie driver wip * iedriver tested and working fine * added doc for iedriver * more logical refactoring of ui driver packages * added simple skeleton for testing / replicating karate ui issues * null in dynamic scenario outline cell causes npe karatelabs#1045 * updated docs to point to external example * fix link in prev commit * fix link in prev commit 2 * ok last try for the readme tweak * defensive coding for karatelabs#1047 * upgrade jersey version * update docs * [warn] karate.log() now pretty prints * karate-robot ready for (experimental) release now you can set a base-path from which images will be loaded added readme to get things going * prep doc for release wip * update doc release wip * doc edit wip * prep release 0.9.5 final * Bump netty.version from 4.1.37.Final to 4.1.50.Final in /karate-core Bumps `netty.version` from 4.1.37.Final to 4.1.50.Final. Updates `netty-handler` from 4.1.37.Final to 4.1.50.Final - [Release notes](https://github.com/netty/netty/releases) - [Commits](netty/netty@netty-4.1.37.Final...netty-4.1.50.Final) Updates `netty-codec-http` from 4.1.37.Final to 4.1.50.Final - [Release notes](https://github.com/netty/netty/releases) - [Commits](netty/netty@netty-4.1.37.Final...netty-4.1.50.Final) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Peter Thomas <[email protected]> Co-authored-by: Sharma Prashant <[email protected]> Co-authored-by: lukas.cardot <[email protected]> Co-authored-by: Peter Thomas <[email protected]> Co-authored-by: babusekaran <[email protected]> Co-authored-by: BenjamQC <[email protected]> Co-authored-by: Nishant-Sehgal <[email protected]> Co-authored-by: tbhasin <[email protected]> Co-authored-by: Manoj Y <[email protected]> Co-authored-by: BadgerOps <[email protected]> Co-authored-by: srangaraj1 <[email protected]> Co-authored-by: ghostwriternr <[email protected]> Co-authored-by: Thomas <[email protected]> Co-authored-by: Celeo <[email protected]> Co-authored-by: paaco <[email protected]> Co-authored-by: alexanderp <[email protected]> Co-authored-by: paaco <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is issue with respect to karate.abort() function where I am able to successfully skip the steps once my condition is met but where as I am able to see the status of the test as Failed in cucumber html reports.
The text was updated successfully, but these errors were encountered: