Test Suite

[JoshData]

We need tests. The tests should verify that every configuration change made by the Mail-in-a-Box setup, and any installed configuration files, are having the intended effect. For instance, SMTP TLS settings should be tested by an SMTP client that verifies that the TLS connection has the correct properties.

I could really use everyone's help here figuring out how to get a comprehensive test suite working. There are probably a few hundred things that need to have a test written.

[yodax]

Would you like to embed that in the status_checks? So normal status checks would be as is, but you could also run a diagnostics suite that would run all the tests? If we do it that way we can use it to debug issues on deployments.

[JoshData]

No, this should be separate. This is a tool for development, to make sure we are doing the right thing, not to check if the user has installed a box correctly.

[yodax]

Perhaps something like behave a python version of the cucumber language. That way we would be writing documentation and tests.

I have used SpecFlow (C# Cucumber port) before and find it a good tool to describe the desired situation clearly and easy to integrate into the back-end to run the actual test.

[TabTwo]

You wrote about testing tls, dns, smtp ....
Could we use something like Icinga/Nagios for this?
Like, if the external monitoring says everything is all right, then the miab V0.18 passes this unit-test.

So, we would need a ded. host/Vagrant box + provisioning + config of the monitoringsystem + a domain to test.

[JoshData]

I don't think any particular test framework is going to be very helpful. There are too many different sorts of things that need testing.

[TabTwo]

do you already have a list of things that need to be tested?

[JoshData]

No, and making that list seems like the first thing that needs to be done.

[TabTwo]

could you create a new repository "testsuite" (or whatever you prefer) so we can fork + collect ideas?

[JoshData]

You got it: https://github.com/mail-in-a-box/mailinabox-testsuite. I also made you an admin of the new repo. :)

[wsteitz]

do you want to enhance what is in the tests folder or create something completely new?

sovereign has a test suite that runs against a vagrant machine (https://github.com/sovereign/sovereign/blob/master/tests.py), I could imagine doing something similar for miab.

[JoshData]

Either way.

[brycefisher]

I realize this is an old thread, but it still seems relevant. To summarize what I'm seeing as the state of things:

• There is a desire from @JoshData to have tests in this project
• While there is a body of tests (in this repo in the "tests/" directory), there's no continuous integration system (or at least none integrated into this Github repo)
• There's a desire to eventually do in-depth functional testing of, for example, dovecot configurations, etc, etc

I think those are fantastic goals, but perhaps quite lofty. I propose we setup continuous integration to setup unit tests for the various bits of python so that any PR submitted to the project gets at least some minimal amount of automated testing done. Hopefully, this would reduce of some of the burden on @JoshData. I would start very small, with perhaps only 1 or 2 simple tests that can run consistently, and slowly working on building out a bigger test suite.

I think this is a small enough task someone could contribute a minimal py.test unit testing PR and walk JoshData through the necessary steps of integrating a free-for-open-source cloud CI.

Before diving into the details around what testing framework, what CI, and what specifically to test, does this sound like a good idea to you @JoshData? Would you be willing to accept a well-written PR to this repository that setup minimal, continuous testing?

[JoshData]

Thanks for reviving this old issue. It's an important one.

There's always an allure of setting up a new tool like CI but not doing the less interesting work of building the tests (and the CI part is the largest time-draw for me), so I'd like to see some more useful tests in place before we do CI for the existing tests.

Crucially, we're missing tests for installing the box from scratch and performing upgrades. We have a Vagrant file but nothing around it.

[brycefisher]

I'm often tempted by the shinynesss, and agree that can totally distract from more important things. One lesson I try to take from my day job and apply to my own open source projects is the power of incrementalism. I try to ask myself what one small change I could make to help myself keep going, or get closer to some larger milestone, especially if the smaller step is useful on its own. To me, having a minimal CI setup that does anything on every push (even just echo "It works!") can be a force multiplier by ensuring that whatever tests you have (or want to build ;-) ) are actually used. If that's not something you want, I'll leave it alone :-) Thanks for this project!! -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[JoshData]

I agree with incrementalism, but if the CI system can't install the box from scratch, it's not really testing anything --- it's just sending me emails about things that don't matter.

[brycefisher]

Fair point! What if I work on setting up a test for setting up MIAB from scratch that runs in CI on a fork? If I was going to do that, I'd probably start thinking through what kind of assertions I'd need to make. I saw some prewritten tests linked to here that interacted with MIAB over IMAP. Would something along those lines be valuable? -- Sent from my Android device with K-9 Mail. Please excuse my brevity.