Place module related features and contexts into a subdirectory of a modules dir

[Vinai]

This is a copy of a Skype discussion on the subject:

Marco Debo De Bortoli: I generally prefer to have features and tests separated from the code base. It's a personal preference of course but it's driven by the fact I "need" and easy way to deploy my source code without the tests suite. Tests are not required by application to work.
Features and tests are of course helpful and they clearly describe more or less what the module does... again not relevant outside of a development scenario...

Vinai Kopp: I can follow that absolutely.
The other approach only makes sense from a Magento module developers perspective
They have a Module centric view. Adding a module to a system should simply add all required components, including any tests. That way interoperability with other extensions by different authers can better be ensured
Since there is no "standard" directory setup outside of the Magento base dir, they like to put all files that belong to their module into that modules directory

Jakub Zalas: if module is a reusable thing, i'd opt for having tests in the module folder
[21.05.13 12:06:19] Marco Debo De Bortoli: Yes I can see that somehow and it's also the way Mage2 is taking making everything self-contained
since it can be taken out of the project and installed in another one

Vinai Kopp: Yes, it is true what you are saying in regards to Mage2

Marco Debo De Bortoli: @jakub: Modules are all reusable else where and there are tools like modman which helps in the process

_Vinai Kopp: So BehatMage using app/code/[codepool]/[NameSpace]/[Module]/features would perfectly suit that approach.
But it should only use modules that have been activated in app/etc/modules
For that reason it makes sense (and also follows the "magento" way of thinking), that every module that offers BehatMage features should add it's node to the etc/config.xml file
something like

<behat>
    <features>
        <NameSpace_Module/>
    </features>
</behat>

Marco Debo De Bortoli: Something to definitely look into and the same should go for specs and such

[jamescowie]

Digging up this idea as a potential feature to include in the next release. I think its a good idea to provide options to the user. So I see two scenarios:

1. Features are stored in the default location within the root of the project and behat can locate and run these.
2. Features are defined in a modules app/etc/modules/...xml file and MageBehat will parse these files grab and behat nodes and add those to the run list.

Do you have any more ideas as this issue was added some time ago.

[ajbonner]

+1 this is definitely a good feature.

One of the reasons I stuck with the MageTest library over EcomDev_Test was the fact I could easily keep my test/production code separate. And I know that equally, there are those that really want to include testcode directly with their module code and don't like it being separated.

[Vinai]

Thanks for resurrecting this issue, James.
At the moment I have to say that I personally use three approaches.
First, project tests go into a separate tests/ directory outside of the Magento root. For Behat that of course would be a features/ directory.

For modules I use two approaches, depending on how the module is distributed.
For new modules that use soley modman I use

Namespace_Module/src/... Magento directory structure ...
Namespace_Module/tests/... Tests ...

However, with older extensions, where the source is not within a src/ subdirectory, or if I want to use EcomDev_PHPUnit for DB fixtures support or to work with an existing infrastructure (judge for example), which relies on EcomDev_PHPUnit, I put the tests in the actual module within
app/code/community/Namespace/Module/Test.

That said, I'm not very happy with this inconsistent state of things.
My preference would be to have only two locations for testing related code.
A features/ (and/or tests/) directory in the project root for project tests, and a Namespace_Module/features directory in the module archive.
To put that into context within a project

/.modman/Namespace_Module/src/... [module code mirroring the Magento dir structure]
/.modman/Namespace_Module/features/... [module Behat features]
/.modman/Namespace_Module/tests/... [module PHPUnit tests]
/.modman/Namespace_Module/spec/... [module PHPSpec specs]
/features/...[project Behat features]
/tests/...[project PHPUnit tests]
/spec/...[project PHPSpec specs]
/htdocs/... document root containing Magento ...

Hmm, maybe it would make sense to group features, tests and specs together within one parent, "meta" or "qa" or something. OTOH, it doesn't hurt and probably is a lot easier this way.

[fooman]

This would be awesome to have. I am currently using

app/code/community/Namespace/Module/Test
app/code/community/Namespace/Module/Test/features

as anything at the root level does not combine well the moment you have two extensions. I currently have Behat and EcomDev Phpunit in the extension Test directory but anything that would allow us to place it directly under the module dir (features, spec as Vinai proposed) would be a big improvement.

In an ideal world all extensions would be distributed with their tests included and I am hopeful that with Magento 2 this might become more common place. Having tests and features contained within the module would allow for this to happen (it would still be easy enough to not distribute these folders if you do not want to do this). It is harder the other way around as currently there is no separation on the root level.

EcomDev looks into the extension's config.xml file, ie
app/code/community/Namespace/Module/etc/config.xml

and I think adding this

<behat>
    <features>
        <NameSpace_Module/>
    </features>
</behat>

in there as well makes sense.

Alternatively looking ahead at Magento 2 https://github.com/magento/magento2/tree/master/dev their approach seems to be dev/tests and then further breakdown by type and module.

[adam-paterson]

Has anybody made any progress with this? I think with @amacgregor's book release more and more people are going to be looking to separate their tests from the application. Especially module developers.