Writing Tests for Extensions

Discussion forum for Extension Writers regarding Extension Development.
User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Writing Tests for Extensions

Post by HoLyCoW » Sun Apr 19, 2015 4:44 am

I know that Testing our extensions are not required, however it is encouraged. I would like to get started writing some Tests, but I'm brand spanking new at it. I had a few questions I wanted to ask...

1) I noticed that some of the extensions that are tested use PHPUnit and therefore have Unit Tests. Is Unit Testing the only kind of tests we can do since we're just writing extensions as opposed to whole web applications?

2) What about Acceptance or Functional tests? Should we be writing these too using Codeception or something of the like?

3) Lastly, since I've never written any kinds of tests, what exactly should I be testing for? Would I really just use it if my extension provides some kind of form (i.e. ACP settings)? Or do I need to write tests anytime my extension deals with data?

Thanks in advance.

User avatar
Wolfsblvt
Registered User
Posts: 634
Joined: Sun Oct 26, 2014 9:12 pm
Location: Solingen, Germany
Contact:

Re: Writing Tests for Extensions

Post by Wolfsblvt » Sun Apr 19, 2015 3:58 pm

1) This the only kind I know, yes.
There are specific classes to extend for unit testing, they already exist in phpBB.
What do you mean by only kind, what other tests can you think of that may are useful?

2) There are functional tests. Take a look at official boardrules extension: https://github.com/phpbb-extensions/boa ... functional
You can learn how they work and how do you do functional testing by reading that tests, I would guess (:

3) The more you test, the better it is :P
Goal should be 100% code coverage, if you want to be perfect.
Most of the time you should test different inputs for your method and check the output. So consider all cases that may happen.

And for learning how to do testing: The best is read test cases from extensions and learn from them (:
If you have a specific extension request and you are willing to pay for, you can write me a PM.
My extensions (Trending: @Mention SystemAdvanced PollsUser Online Time)

»Du kamst zu uns. Deine Stimme kam. Du zeigtest uns die Sterne. Sie funkelten. Wir konnten sehen.«

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Mon Apr 20, 2015 1:01 am

Wolfsblvt wrote:1) This the only kind I know, yes.
There are specific classes to extend for unit testing, they already exist in phpBB.
What do you mean by only kind, what other tests can you think of that may are useful?
Well the other kinds of tests that I was thinking of was functional and acceptance testing.
Wolfsblvt wrote:2) There are functional tests. Take a look at official boardrules extension: https://github.com/phpbb-extensions/boa ... functional
You can learn how they work and how do you do functional testing by reading that tests, I would guess (:
Oh, I forgot to check the repository. I downloaded the release of this extension, but I didn't see the tests directory. I forgot that they could remove it for the release.

[edit]
I see that they extend a class called \phpbb_functional_test_case. Where is this class located? I can't find it. Is this a custom testing framework?
Wolfsblvt wrote: 3) The more you test, the better it is :P
Goal should be 100% code coverage, if you want to be perfect.
Most of the time you should test different inputs for your method and check the output. So consider all cases that may happen.
What do you mean 100% code coverage? From what I've been reading, I thought that it wasn't possible to test every single scenario. How do I know how much "code coverage" i have? Will phpUnit tell me?
Last edited by HoLyCoW on Mon Apr 20, 2015 8:18 am, edited 1 time in total.

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Mon Apr 20, 2015 8:16 am

Oops. I hit on "reply" instead of edit. Sorry for the double-post

User avatar
Wolfsblvt
Registered User
Posts: 634
Joined: Sun Oct 26, 2014 9:12 pm
Location: Solingen, Germany
Contact:

Re: Writing Tests for Extensions

Post by Wolfsblvt » Mon Apr 20, 2015 8:23 am

Yeah, I would say you should start learning by reading the tests of this extension (:

For how to set up travis tests in your own extension there is a guide here inside this forum, I think you have found it already.

Code coverage isn't by PHPUnit, no. Scrutinizer has the possibility to do so, when you set it up correctly. This is done for some of the official extensions, sure for the newest.

What I meant with "code coverage" is not really to test every scenario, but more to test every line of code. So if you have a if/else block, you need at least two datasets, so that both blocks can be reached. For switch/case with 10 cases you need at least 10, etc.
Of course that's no real indicater if all of this code is running, but it is a start, and something that can be covered by automated testing services.

EDIT://
Yes, it is a custom testing framework wich extends PHPUnits framework.
For all the classes see here: https://github.com/phpbb/phpbb/tree/3.1 ... _framework
They are located under .\tests in the GitHub repository.
If you have a specific extension request and you are willing to pay for, you can write me a PM.
My extensions (Trending: @Mention SystemAdvanced PollsUser Online Time)

»Du kamst zu uns. Deine Stimme kam. Du zeigtest uns die Sterne. Sie funkelten. Wir konnten sehen.«

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Mon Apr 20, 2015 4:09 pm

Thanks very much Wolf

User avatar
VSE
Extensions Development Coordinator
Extensions Development Coordinator
Posts: 4883
Joined: Sat Jan 17, 2009 9:37 am
Location: Los Angeles, CA
Name: Matt Friedman
Contact:

Re: Writing Tests for Extensions

Post by VSE » Mon Apr 20, 2015 6:38 pm

Actually code coverage is done by PHPUnit. Scrutinizer is just a code analysis and metrics platform that can also display code coverage reports generated by your testing platform.

PHPUnit can generate test coverage reports in many ways, either directly outputting info into the command line window, or generating a really nice HTML document to view in a browser. http://tdd-deciphered.com/part/18/verif ... e_coverage

Also, functional tests are a minor part of testing and not part of code coverage. What you really want to be doing is unit testinng, via the phpbb_test_case and phpbb_database_test_case classes. This tests that the data you send to a function comes out the other end as expected. Functional testing is very slow and is just used to test how things work in the way a user experienced your web app through a browser (which at its simplest means loading URLs and crawling the HTML for expected text and stuff in the DOM).
Dictated but not read.
Official phpBB Extensions My Extensions & MODs
Please do not PM me for support.

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Fri Apr 24, 2015 6:55 am

Okay, so I was looking over the code for the tests and wanted to see what the output of the tests were for the boardrules. Here's what I did:
  1. mkdir ~/Sites/Sandbox
  2. cd ~/Sites/Sandbox
  3. git clone https://github.com/phpbb/phpbb.git phpbb_repos
  4. cd phpbb_repos/phpBB/ext
  5. mkdir phpbb
  6. cd phpbb
  7. git clone https://github.com/phpbb-extensions/boardrules.git
  8. cd boardrules
  9. phpunit
Here's what I get:

Code: Select all

PHP Warning:  require_once(phpBB/includes/startup.php): failed to open stream: No such file or directory in /Users/holycow/Sites/Sandbox/phpbb_repos/tests/bootstrap.php on line 17

Warning: require_once(phpBB/includes/startup.php): failed to open stream: No such file or directory in /Users/holycow/Sites/Sandbox/phpbb_repos/tests/bootstrap.php on line 17
PHP Fatal error:  require_once(): Failed opening required 'phpBB/includes/startup.php' (include_path='/Users/holycow/.composer/vendor/phpunit/php-text-template:/Users/holycow/.composer/vendor/phpunit/php-timer:/Users/holycow/.composer/vendor/phpunit/php-file-iterator:.:/usr/local/Cellar/php55/5.5.23/lib/php') in /Users/holycow/Sites/Sandbox/phpbb_repos/tests/bootstrap.php on line 17

Fatal error: require_once(): Failed opening required 'phpBB/includes/startup.php' (include_path='/Users/holycow/.composer/vendor/phpunit/php-text-template:/Users/holycow/.composer/vendor/phpunit/php-timer:/Users/holycow/.composer/vendor/phpunit/php-file-iterator:.:/usr/local/Cellar/php55/5.5.23/lib/php') in /Users/holycow/Sites/Sandbox/phpbb_repos/tests/bootstrap.php on line 17
Here's the contents of my global composer.json file:

Code: Select all

{
    "require-dev": {
        "phpunit/phpunit": "4.6.4",
        "phpunit/php-timer": "~1.0",
        "phpunit/php-text-template": "~1.2"
    }
}
I'm I doing something wrong? Should I be copying some of the testing framework files to the tests directory inside the boardrules folder in order to run the tests?

thanks in advance.

User avatar
Wolfsblvt
Registered User
Posts: 634
Joined: Sun Oct 26, 2014 9:12 pm
Location: Solingen, Germany
Contact:

Re: Writing Tests for Extensions

Post by Wolfsblvt » Fri Apr 24, 2015 3:35 pm

You must execute phpunit from phpBB root folder, where the tests are.

So the command would be: phpBB\vendor\bin\phpunit -c phpBB\ext\phpbb\boardrules\phpunit.xml.dist

You can read that also here: Simple way to test your ext on TravisCI
If you have a specific extension request and you are willing to pay for, you can write me a PM.
My extensions (Trending: @Mention SystemAdvanced PollsUser Online Time)

»Du kamst zu uns. Deine Stimme kam. Du zeigtest uns die Sterne. Sie funkelten. Wir konnten sehen.«

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Fri Apr 24, 2015 4:05 pm

Wolfsblvt wrote:You can read that also here: Simple way to test your ext on TravisCI
I looked at that, but I guess my mind just blocked it out. :mrgreen:


Thanks!

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Sun Apr 26, 2015 4:56 am

Okay, I tried re-reading all the documentation I could find before posting this time. But I don't see anything about the issue I'm getting.

I tried running the tests for boardrules, but all I get are a bunch of the same errors. So I tried with the acme/demo tests too, and I'm getting the same error a bunch of times.

I keep getting:

Code: Select all

RuntimeException: You have specified an invalid dbms driver:
I googled this and thought that maybe my config.php was setup incorrectly, but this is what I have which looks fine to me:

Code: Select all

$dbms = 'phpbb\\db\\driver\\mysqli';

$acm_type = 'phpbb\\cache\\driver\\file';

User avatar
Wolfsblvt
Registered User
Posts: 634
Joined: Sun Oct 26, 2014 9:12 pm
Location: Solingen, Germany
Contact:

Re: Writing Tests for Extensions

Post by Wolfsblvt » Sun Apr 26, 2015 11:23 am

Do you run the tests locally, or on travis?
Tests are not using your config.php, they will use a special config_test.php, where the setup for testing is stored.

So the exception tells you mysqli is invalid?
If you have a specific extension request and you are willing to pay for, you can write me a PM.
My extensions (Trending: @Mention SystemAdvanced PollsUser Online Time)

»Du kamst zu uns. Deine Stimme kam. Du zeigtest uns die Sterne. Sie funkelten. Wir konnten sehen.«

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Sun Apr 26, 2015 4:50 pm

I'm running them locally using mysql 5.6.24

With this command:
❯ phpBB/vendor/bin/phpunit -c phpBB/ext/acme/demo/phpunit.xml.dist

I get this error:

Code: Select all

PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Users/holycow/Sites/Sandbox/phpbb/phpBB/ext/acme/demo/phpunit.xml.dist

.EEE

Time: 110 ms, Memory: 6.50Mb

There were 3 errors:

1) acme\demo\tests\dbal\simple_test::test_column
RuntimeException: You have specified an invalid dbms driver:

/Users/holycow/Sites/Sandbox/phpbb/phpBB/phpbb/config_php_file.php:158
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_test_case_helpers.php:153
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_database_test_case.php:160
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_database_test_case.php:172
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_database_test_case.php:115

2) acme\demo\tests\functional\demo_test
RuntimeException: You have specified an invalid dbms driver:

/Users/holycow/Sites/Sandbox/phpbb/phpBB/phpbb/config_php_file.php:158
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_test_case_helpers.php:153
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_functional_test_case.php:47

3) acme\demo\tests\functional\demo_test
RuntimeException: You have specified an invalid dbms driver:

/Users/holycow/Sites/Sandbox/phpbb/phpBB/phpbb/config_php_file.php:158
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_test_case_helpers.php:153
/Users/holycow/Sites/Sandbox/phpbb/tests/test_framework/phpbb_functional_test_case.php:47

FAILURES!
Tests: 2, Assertions: 3, Errors: 3.

Also, I just noticed something interesting...
Above I used the command inside the phpBB/vendor directory, which according to the output is version 4.1.0. However, I also have phpunit installed at the system level (also 4.1.0). If I run the same command but with the system-wide phpunit command, I get a different error.

With this command:
❯ phpunit -c phpBB/ext/acme/demo/phpunit.xml.dist

I get this error:

Code: Select all

PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from /Users/holycow/Sites/Sandbox/phpbb/phpBB/ext/acme/demo/phpunit.xml.dist

.PHP Fatal error:  Call to undefined method acme\demo\tests\controller\main_test::hasExpectationOnOutput() in /Users/holycow/.composer/vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php on line 548

Fatal error: Call to undefined method acme\demo\tests\controller\main_test::hasExpectationOnOutput() in /Users/holycow/.composer/vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php on line 548

User avatar
Wolfsblvt
Registered User
Posts: 634
Joined: Sun Oct 26, 2014 9:12 pm
Location: Solingen, Germany
Contact:

Re: Writing Tests for Extensions

Post by Wolfsblvt » Sun Apr 26, 2015 10:45 pm

Have you set up the testing database locally?
It is needed for executing db testcases locally.

Seems this is missing there :P

You have to create a mysqli database named phpbb_tests somewhere. Best of course also locally.
Then you need a file where you specify the connection to this empty database.
Create a file named test_config.php in your .\tests folder and use this content:

Code: Select all

<?php
/**
*
* This file is part of the phpBB Forum Software package.
*
* @copyright (c) phpBB Limited <https://www.phpbb.com>
* @license GNU General Public License, version 2 (GPL-2.0)
*
* For full copyright and license information, please see
* the docs/CREDITS.txt file.
*
*/

$dbms = 'phpbb\\db\\driver\\mysqli';
$dbhost = 'localhost';
$dbport = '';
$dbname = 'phpbb_tests';
$dbuser = 'root';
$dbpasswd = '';
$phpbb_functional_url = 'http://localhost/phpBB/';
Replace $dbhost, $dbport, $dbuser and $dbpasswd with the connection to your testing database.
For functional tests you have to speficy an url where you can access your local forum so it can be crawled.

Try running the tests again then.
(Be careful, for me the execution replaces the normal config.php inside phpBB folder)

EDIT://
Use your global phpunit will not work. As you can see, the real exception methods are not displayed, cause the function to format them is missing.
If you have a specific extension request and you are willing to pay for, you can write me a PM.
My extensions (Trending: @Mention SystemAdvanced PollsUser Online Time)

»Du kamst zu uns. Deine Stimme kam. Du zeigtest uns die Sterne. Sie funkelten. Wir konnten sehen.«

User avatar
HoLyCoW
Registered User
Posts: 49
Joined: Tue Oct 21, 2014 5:51 am

Re: Writing Tests for Extensions

Post by HoLyCoW » Mon Apr 27, 2015 3:39 am

Thanks wolf. Got it working!

Post Reply

Return to “Extension Writers Discussion”