First integration tests for provisioning api

[DeepDiver1975]

@PVince81 @Raydiation @schiesbn @blizzz @karlitschek

So - before touching the provisioning api to implement V2 - I wanted to have integration tests to be sure nothing will be broken.

TODO:

• setup jenkins on this branch
• integrate this into the regular ci mechanism (AFTER merge)
• test all provisioning APIs
• get coverage results

[karlitschek]

nice 👍

[DeepDiver1975]

@MorrisJobke please have a look since you did use behat already as well - THX

[DeepDiver1975]

@cmonteroluque @rperezb just FYI - this is what I was talking about in the past regarding integration testing of the OCS API

[LukasReschke]

Indentation looks off compared with above.

[DeepDiver1975]

mimimi ... 🙈

[LukasReschke]

[LukasReschke]

Besides me doing mimimimimi, this is cool 🚀

[BernhardPosselt]

Indention :hides:

[DeepDiver1975]

@Raydiation are you by chance aware of a way to collect coverage data of php dev server?

[BernhardPosselt]

@DeepDiver1975 what I've found: Behat/Behat#92

[BernhardPosselt]

TL;DR: Not needed for integration tests. 100% code coverage is only usable for unit tests because it would take too long otherwise

[BernhardPosselt]

IMHO: write a simple parser for our API docs and run them against ownCloud :)

[BernhardPosselt]

As in provide an example API call and the result, run the example call against the API with test data and then compare it :)

That way you generate API docs that are tested and also help you integration test stuff :)

[oparoz]

Or use Codeception which has an API tester

http://codeception.com/docs/10-WebServices

<?php
$I = new ApiTester($scenario);
$I->wantTo('create a user via API');
$I->amHttpAuthenticated('service_user', '123456');
$I->haveHttpHeader('Content-Type', 'application/x-www-form-urlencoded');
$I->sendPOST('/users', ['name' => 'davert', 'email' => '[email protected]']);
$I->seeResponseCodeIs(200);
$I->seeResponseIsJson();
$I->seeResponseContains('{"result":"ok"}');
?>

[DeepDiver1975]

@DeepDiver1975 what I've found: Behat/Behat#92

I'm not talking about coverage on the behat process - but on the php -S localhost:8080

[DeepDiver1975]

Or use Codeception which has an API tester

Well - there are plenty of tools - we finally simply need to get started ...

[BernhardPosselt]

@DeepDiver1975 does unit test coverage fail because of the test server?

[BernhardPosselt]

http://xdebug.org/docs/code_coverage btw

[oparoz]

Well - there are plenty of tools - we finally simply need to get started ...

I have found that there were not that many, but I agree, tools should not slow down any effort to improve tests.

What I've gathered so far for Codeception

+

• Faster
• More readable tests
• Does Behavioural, functional and unit with one "language", not intermediary language needed
• Works better when lots of tests are needed

-

• New syntax to learn for die-hard PHPunit fans

[scrutinizer-notifier]

A new inspection was created.

[DeepDiver1975]

Jenkins is green 🎉

https://ci.owncloud.org/job/ocs-api-integration-tests/11/

[DeepDiver1975]

What I've gathered so far for Codeception

I was looking at codeception multiple times - I don't want to introduce another paradigm

[oparoz]

I was looking at codeception multiple times - I don't want to introduce another paradigm

OK. Just know that, although I haven't tested with core tests yet, my PHPUnit tests in other projects run without any changes. You only need to familiarise yourself with the new syntax and setup if your intend on writing new kind of tests (behavioural, API).

[MorrisJobke]

As in provide an example API call and the result, run the example call against the API with test data and then compare it :)

That way you generate API docs that are tested and also help you integration test stuff :)

@Raydiation I wrote exactly this some weeks back for another project ;) But it's not in the shape I want it

[DeepDiver1975]

So @oparoz wrote one test with codeception - https://github.com/owncloud/gallery/blob/b84508a0b687b91b623c6f81ea99e5fe40daa846/tests/api/CreateUserCept.php

Looks interesting as well .... which path to choose?

@MorrisJobke @PVince81 @icewind1991 @blizzz

[oparoz]

For the API stuff, Codeception is easy to use and coders can quickly get going. It's literally about typing a couple of commands and writing your first test, the same way you write a scenario. If you need to show the steps involved to a BA or a PO, then you can convert the script to English.

Example

$I->amHttpAuthenticated('admin', 'admin');
$I->haveHttpHeader('Content-Type', 'application/x-www-form-urlencoded');
$I->sendPOST($baseUrl . '/users', ['userid' => 'test', 'password' => 'test']);
$I->seeResponseCodeIs(200);
$I->seeResponseIsXml();
$I->seeXmlResponseIncludes(
    XmlUtils::toXml(
        ['status'   => 'ok']
));

becomes

* I am http authenticated "admin","admin"
* I have http header "Content-Type","application/x-www-form-urlencoded"
* I send post "/ocs/v1.php/cloud/users",{"userid":"test","password":"test"}
* I see response code is 200
* I see response is xml
* I see xml response includes "DOMDocument"

Of course, with Behat, once the commonly used steps are written in PHP, then things can go very fast as well from scenario to test and it might be more flexible.

Some potential problems

• it will create a bunch of folders inside the folder you want to use, so, for core, you have to choose wisely. Maybe build/integration. The structure looks like this: https://github.com/owncloud/gallery/tree/testing-with-codeception/tests
• if you want to use the codeception built-in server (add some config lines and you have a server running during the tests only), then you'll need to use composer to install all the dependencies and run codecept from vendor/bin. Otherwise, you can just use the phar to generate and run stuff. There may be a way around that. I haven't dug too deep.
• if your QA engineers are used to working with Gherkin and have a backlog of acceptance tests, then you may not want to have 3 tests systems: PHPUnit for unit/integration, Behat for acceptance, Codeception for api/integration. In my projects, I prefer to use one tool for all three and it works because Codeception can execute PHPUnit tests.

I've written some more (more!?! ;)) on my decision to use it for Gallery here: owncloud/gallery#264 (comment)

[LukasReschke]

FWIW I do prefer Gherkin (Behat) syntax.

[oparoz]

It might be worth getting some feedback via the mailing list?

[oparoz]

An older comparison of the two from 2 years ago. Things might have changed on both sides
http://codeception.com/05-06-2013/specification-testing-coparison.html

[MorrisJobke]

I also like the gherkin syntax more.

[MorrisJobke]

Tested and the code looks good. 👍

[DeepDiver1975]

I'm merging this now @SergioBertolinSG any future test case via pull request please - THX