Create tests

[tyrope]

The test program should do the following steps

• Run Willie.
• Execute core tests.
• Detect loaded modules.
• Find tests for loaded module and execute these.
• Generate human-readable test report to file. (Currently to stout)

[elad661]

My suggestions:

• Follow the convention of other FOSS projects and change the name from tester to tests
• Run its own willie instance so that we can easily detect commits that breaks the ability of the bot to start and connect.
  • to do that we will need to add a switch willie, something like --exit-on-error to make the bot exit with a predefined code as soon as a core exception is detected.
• Automatically test every new commit

[Hawxy]

Wouldn't something like: https://www.shiningpanda-ci.com/ or https://travis-ci.org/
Be useful?

[elad661]

A CI system is useless unless we have automated unit tests.

[elad661]

I'm confused, I thought we had tests. Why is this issue still open and why don't we have a CI in place already?

[tyrope]

This is to test core, not modules. @Venti9874's unit tests are for modules.

edit: closed/opened because wrong button...

[maxpowa]

There's some pretty decent core coverage (triggers/module/basic connection) at this point. I think having visibility to coverage on commits would help increase the push for testing. Can we allow Travis access to the org so we can run tests on commits?

[Exirel]

The Travis CI works fine so far! Sure thing the coverage is not the best yet, but it's improving - at least I'm working on it, slowly, but surely.

I don't know if we need an issue to follow the progress, or maybe a new issue to list the critical parts that require testing - or code refactoring to ease testing.

[dgw]

This is one of those issues so old I didn't get back far enough in the history when I took over the project last year and went around labeling/triaging stuff.

With multiple CI services checking every push to master and every opened pull request, I don't think we need this issue open any more. Sopel has tests. They might not cover everything, but they do exist.

@Exirel Since you're already involved in the testing initiative, I'll ask you: Should we continue amending this issue to track what still needs testing/refactoring, or close it and use a new one? Your opinion is important to me in this case, because I think you'll end up doing a lot of the relevant work. 😸

[Hawxy]

Man, I sure was confused when I got an email about this thread.

[Exirel]

@dgw as you said, multiple services are now in use to run tests and check quality of code - so for me the issue is fixed, and all the project need is to write test.

So let's close this one and we'll see later what we need. 👍