Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return test results in Yunit.Test #12

Open
aviaryan opened this issue Apr 21, 2016 · 5 comments
Open

Return test results in Yunit.Test #12

aviaryan opened this issue Apr 21, 2016 · 5 comments

Comments

@aviaryan
Copy link
Contributor

It would be better if Yunit.Test returns test results (instance.results variable in code) instead of void. That way one can programatically evaluate the results and do stuff. One possible use it to check the results and exit script with 1 in case a test fails, something which has been already discussed in #2 .

@infogulch
Copy link
Collaborator

Maybe, but isn't this the purpose of output Modules? Perhaps you could make a small module that just stores the results into a variable.

The main reasons it was designed to use output modules are:

  1. The output is live. Test suites can take a long time and you may not want to wait for a full suite to complete before seeing any output.
  2. It's easy to create your own module and simply plug it in, and use any number of output modules at once.

In this case (saving the results to a variable) you don't really care about point 1, but point 2 still stands.

@aviaryan
Copy link
Contributor Author

aviaryan commented Apr 22, 2016

  1. It's easy to create your own module and simply plug it in, and use any number of output modules at once.

I guess you are right. That is a way to do it and it will be more convenient for my needs but since there is only one format in which you are storing the test results and it is irrespective of the modules used, isn't it plausible to return the results in the method (Test). That way if someone wants, they can use the test results, they won't have to create a module for that.

results := tester.Test(suite1, suite2) ; doesn't it look good

I also have a feeling that returning the associative array instance.results as results will look ugly as it is not properly designed/formatted for end-user use.
So you may just return a boolean True/False value in tester.Test().

As you said, that test suites can be long so there maybe some users (or automated systems) who want to exit the tests immediately in case one of the test suite fails.

result = tester.Test(suite1)
if (!result){
    exitapp, 1
}
result := tester.Test(suite2)
...

That should be a fairly good use case. What are your thoughts ?

@infogulch
Copy link
Collaborator

If you want an early exit mechanism, you could make a ... drum roll ... module! (I'm sure a "There's a module for that." joke is hidden somewhere here.)

class EarlyExit ;untested
{
    Update(Category, Test, Result)
    {
        if IsObject(Result)
        {
            ExitApp 1
        }
    }
}

Then your test line would be: Yunit.Use(YunitStdout, EarlyExit).Test(suite1, suite2) and it would exit on the first failing test.

Having Test() return a simple value (like the number of failed tests) sounds like a good idea anyways. This is being discussed in #2.

Perhaps some of these convenience modules could be added to the library. Another module idea I can think of would wrap any other module and only pass failed tests on to the wrapped module. This would make those failing tests easier to find when using automated systems (e.g. git bisect).

@aviaryan
Copy link
Contributor Author

If you want an early exit mechanism, you could make a ... drum roll ... module!

I thought modules were more like "Output" modules and shouldn't use a strong command like ExitApp. BTW, my real use is to know if some test failed after executing all the tests and then ExitApp accordingly.
To use a module to do this, either I will have to use some global variable or a module class (static) variable. The later is obviously the better option but using a module for such a basic Test Framework feature is too much.

I like the returning no of tests failed in Test() idea you proposed in #2. Hope that gets implemented.

BTW, by automated systems I meant automated testing (CI). See https://github.com/aviaryan/Ahk-CI-Example/

@infogulch
Copy link
Collaborator

infogulch commented Apr 22, 2016

I thought modules were more like "Output" modules and shouldn't use a strong command like ExitApp.

I suppose. (In fact I think it would be a good idea to rename Yunit.Modules to OutputModules to make that more clear.)

The only thing I could see that could be a problem is that Update is called before the test suite's End method, which could result in things that aren't cleaned up. But this could be fixed very easily by just moving the call down a few lines.

The only other thing is the suite destructor, but iirc the __Delete method should be called for the test suite when the program exits.

Edit: I like your CI library. I'll definitely use it on my next project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants